Denying to do work outside your department that isn’t on your job description?

I’m just wondering the take here. My job description says during my “downtime” I’m to “assist other staff”. My boss (owner) is now trying to have me be given tasks by someone in a different department. Assisting isn’t doing someone’s job for them? Especially when it’s outside of my field, right? Or am I just an ass