Honestly I've found a lot of things tend to be easier once you go for actual human foods in terms of treats, and the quality control is generally significantly better than with pet grade things. It can also be a lot cheaper.
In the US a great alternative to yogurt drops to avoid dairy entirely would be Beech Nut's fruit & veggie melties - do be careful not to get the yogurt melties they have though. The fruit & veggie ones are very simple in terms of ingredients - no added sugar of any kind(all sugar coming from the whole food ingredients themselves), no preservatives, the only non whole food type ingredient being fruit pectin which is very safe.
Beech-Nut Naturals(R) Fruit & Veggie Melties banana, blueberry & green beans Stage 3 Snack
Beech-Nut Naturals(R) Fruit & Veggie Melties apple & pumpkin Stage 3 Snack
Using plain puffed grains would honestly be my go to for replacing the Gerber baby puffs entirely. While those aren't the worst thing you could give your hamster especially compared to a lot of storebought treats, they're still not exactly the healthiest option.