Hello Frodo's Mum,
It will be ok, trust me. It can take hamster two weeks until it is used to its new environment. Going into a new home is the most stressful experience for every hamster baby, so you need to be patient. For me, it was really about two weeks until Yasmin started to go out of her house more. We were waiting for her, but she was just too scared to go out. Now, she is 3 months old and she is absolutely fine with us holding her.
My advice is:
1. Rub your clean hands on some tissue, Frodo will use it to build hid nest and it will smell like you. You can do that every day with bedding, so he gets to use on your smell.
2. Guve him food all the time. Yasmin is currently expecting me to always hold some food
as sunflower seeds, tiny bit of apple, pumpkin seed, or you can try normal food mix, if he os not bored with it and takes it, if you give him many treats. Food is the best technique to build hamster trust. So if you hear him active in his nest, but hidden, you can hold a treat near to the entrance of his hause and talk to him. He will associate your hand with food
3. Be patient. You did not ruin anything, but every scare like this can take progress a day or two back. It will be surely ok, it just takes time.