Thank you for your replies! I've decided to take it slow for now, and am staying up every other night to interact with him, and go to bed early every other night, so I don't lose my mind.
The last few nights he's seemed more confident in the cage. If I put my hand on the meshed lid he doesn't mind at all. He takes treats through the lid and investigates my scent with his little nose.
However, tonight I decided to take it to the next level and put my hand in the cage with a pumpkin seed. He was on his way in my direction but as soon as I put my hand in he froze and held his position for at least 5 minutes. Then he went back to his safe corner and stayed there while I kept my hand still for another 10 minutes. After that I just left the pumpkin seed where my hand was and walked away. But he still hasn't moved (it's been another 15 min), that's how freaked out he is from my hand being half a meter away
I will keep at it, although I wish I didn't scare him so easily.