I'm sorry to hear of your hamster becoming ill.
Sadly this is not uncommon in hamsters bred for pet stores as they are born in rodent mills and the parents will not have had the chance to be kept for a while to establish if the lines have any health issues.
Unfortunately this type of tumour can be very aggressive and it is unlikely to be operable if it is in one of the major organs. Personally I think palliative care is best and keeping the hammy comfortable and spoiling her for what time she has left always watching to see when you feel that life has become a burden. The price quoted by your vets sounds way over the top and if you decide you want to go down that route it may be best to call around and see how much other vets would charge.
I always feel that if a hamster only has a short time they won't want to spend it trying to recover from an operation.
I took on a little hamster of a similar age from a pet store who developed a similar problem and sadly she did not live long after diagnosis. This was a kidney tumour. Her story is here :
Sparkle's last chance
I hope you find an answer. Report it to the pet store / customer services as this is the only way it will get back to the suppliers in the rodent farms that they need to look at their lines.