Bedding depends on country,same with cage.However,mice are definitely much cheaper to have cages for and they don't need quite as much space unless you have lots of mice.Male mice have to be kept alone though,so keep that in mind before getting any mice.
Mice are much cheaper and easier to find toys for,as well as a lot of other supplies,just because they are considerably smaller than hamsters except robo hamsters.
Hamsters and mice aren't going to be the most tame pets,but both will tolerate handling.Don't place them on your shoulder though,with mice you could easily lose them and with hamsters they could have a serious fall and that could lead to severe injuries or even death.
How big of a cage can you have?This could be a very good indicator on what is best for you
For bedding,the best option in the UK and some other countries where it is available is Fitch.However in the US,Kaytee Clean and Cozy is preferred over Carefresh due to having nowhere near as much dust and being lower in cost.With hamsters,that cost is easily reduced by having more substrate that is considerably deeper as that means less wasted substrate/bedding and in turn saves you a lot of money(hence,larger cages are actually cheaper long term for hamsters).
Keep in mind that no matter what animal you get,you shouldn't expect them to be "cheap" or pocket money pets.Any animal can become ill and in that situation,vet bills can be extremely costly.That isn't considering the initial startup costs which are always going to be expensive no matter what kind of pet you're getting.