With regard to the toiletting, I've found they only use a potty litter tray if you put it in the place they've chosen to pee. eg a corner of the cage. They do tend to be quite clean normally so I'm wondering if your hammy has some kind of distress or something to get dirty like that. You could tray putting the litter tray where he has chosen to pee regularly - I put chinchilla sand in the litter tray. And you could put a small piece of pee'd on substrate on top of the sand to show it's still the toilet area.
If he is peeing in his house, then having a bigger house can help. With our Syrian,when we gave him a bigger house, he moved his pee corner inside the house, in a corner at the end opposite his nest - so I moved his litter tray into the house
and he only pees in his potty now. Although some hams pee in their wheels as well.
A good cheap option for a larger house is a shoebox. You can cut the bottom out of the box, and cut a hole in front for the door, and use the lid as a lift-off roof. Wait and see which end he builds his nest and put the litter tray in the other end. I have a hamster sized litter tray which seems to fit easily in most houses that are shoebox size. (Linked below).
If what Nancy says is the case then that is sad, and he may have had some trauma,so lots of tlc and patience with him. But I guess it's possible he is still young and they're not noticeable yet. How old is he?
They are usually very clean and wash a lot so him having a soiled bum area isn't really that normal I think. The other thing that might help is the type and depth of substrate. Paper based substrates like Fitch in the Uk, or wood pulp substrates like Carefresh are popular and also non allergenic (some hamsters get allergies to some wood beddings).
But if he is sitting in his pee area, maybe he has some pain or discomfort. And if he has been castrated, this may be causing physical discomfort as well as distress - it might be worth a vet visit to have a checkover and they can give some pain relief medication if they think he has been castrated - and advice as well. Do you have a pet carrier.
Poor little guy - if he is a guy. Maybe try and distract him by a bit of scatter feeding - they love foraging for food. I tend to put half the usual amount of hamster mix in the food bowl and scatter the other half round on the substrate. You could also "hide" a treat or two in different places in the cage for him to find.