Hi there,
I believe moving him back and forth can be quiet stressful, through noise, new smells etc. supply him with enough food and water for 2 days and he will be fine whilst your away, it's probably a lot better for him to stay at home where he is used to everything. Try it and see if he is any better !