I've had both hamsters and mice.
Syrian hamsters HAVE to live alone - no pairs or groups or they'll fight to the death. Winter whites may live in single sex pairs, but it's not guaranteed. If you only have space/finances for one cage then I'd recommend only getting one winter white. Fancy mice girls should live in pairs/groups. Male fancy mice can fight as they mature. If neutered they can live with females.
As you say there are different mouse species, but the ones most usually seen as pets are the fancy mice. There are harvest mice, zebra mice, multimammate mice, spiny mice, African pygmy mice etc. but these are more exotic mice and can be less easy to find.
I find hamsters smell less than mice - male mice are particularly pungent often.
Mice can be prone to respiratory issues so should be kept on a less dusty substrate, like Carefresh, Fitch, cardboard squares, Finacard etc. All of these are suitable for hamsters also.
I can't really comment on cost as I had my mice along with a room full of other rodents so didn't separate out what each of them cost. They seemed similar in use of substrate/food to the hammies, although the naked mouse ate a lot more than her furry friends
I love hamsters and mice, though hamsters are my favourite! I don't currently have mice as a) I don't have space for them as well as the hamsters, and b) I lost quite a few in a short space of time and decided to have a mousey break.