Flipping onto his back is a sign he's completely terrified and is ready to attack. Take it nice and slow, start by just sitting by the cage and talking to him for as long as it takes for him to be comfortable with that (might be anything from minutes to weeks), then gently open and close the door while talking to him until he's comfortable with that (same). When you can open and close the door without him freaking out, try putting your hand in your cage as a closed fist (so there's less for him to bite) and letting him sniff around it. Then try putting your hand under him just as he's about to jump down from a toy or something, so he begins to associate hand with floor. When he's happy to jump onto your hand, try encouraging him to walk over it to get to his ball.... and go from there as feels natural
Try using these tips as well, but different things work for different hamsters-
Taming tips - Hamster Central WIKI