I scared my new dwarf

[Dodo]

Hi everyone!

I brought back a 9 week old dwarf hamster home last week. His name is Stormi and he's a cutie. I've barely seen him since, as he was getting adjusted to his new set up. However this morning he was out. He's lovely, not shy, so me and my kids offered treats and watched him go by. I was arranging a flax spray and stupidly decided to stroke him on the back. He freaked out so I took my hand away, which made a banging noise against the cage's bars, which scared him more. He ran away, jumped on his wheel, gave it a few turns, then decided to go back to his little burrow. I'm feeling a bit mortified that I scared him out so soon when everything was going so well.
What should I do now? Leave him alone entirely for a couple of days? Spend time near his cage just talking? Any advice and reassurance, or anecdotes of hamsters not resenting our clumsiness would be much appreciated