I'm wondering if one of the reasons you're seeing less of Zylvan is partially down to the laziness that seems to affect male Chineseys?
I see a lot less of James and Matthew than I do of Jess and Vicky, who always seem to be busy doing something. Both of my boys are tame and enjoy human contact but sometimes I think that if I didn't wake them up I'd hardly see them at all.
Try not to be too hard on yourself over this - you'll get there with him.