Told to be local and at my house even though I’m hybrid

My boss told me today that I need to be at my house during the week when we're slow so I can be dragged out to visit work sites on the whim. My partner lives 2 hours away and I have been here working from home. Is this even legal?