Hi Jen
welcome to the forum, i hope you enjoy your time here
With baby hamsters the more you handle them the better really. Did you get from a pet shop or from a breeder? If you got him from a breeder then he should be already used to being handled, but if you got him from a pet shop then he probably wont be used to being handled at all - but dont worry, this isnt a problem, it might just take a little longer to tame him.
As you havent really started the taming or bonding process yet, then i am sure another few days wont matter - but as soon as you get back start again with the handling.
Just get him out as much as possible and get him used to you by letting him crawl over you and around you, try not to make any sudden movements which may frighten him, and reward him with treats every time you get him out of his cage.
I am sure you will soon have him very tame
the main thing is to just handle him a smuch as possible
Tammy x