[Thin Lizzy]

Have you tried the tissue trick where you stuff some toilet tissue up your sleeves and leave for a couple of hours and then put in his cage, he'll sniff it, this gets him familiar with your scent. Do this everyday for a week, add some tissue from your sleeve, speak to him as much as you can softly and quietly. Also praise him when he's out and about in his cage.
Hammies can sense when you're scared.
Offer him treats and praise him, tell him he's a good boy, they love getting praised, boosting their confidence.