“I guess you don’t like money”

I'm sure others have heard this or something similar before.

You work your 8 hours a day, 5 days a week. Yes, of course it's not nearly enough money as you want to get.

But the part that bugs me is if you're asked to stay late, or asked to work a 6th day, or a double shift, you are hesitant or just say no, and the response from managers, coworkers and/or just others is “I guess you don't like money.”

I'm like “No, I do like money, I just don't like overworking myself and driving myself crazy not even having down time.”

Sad part, despite a (verbal) agreement between me and my manager about my schedule going forward where I'm working specific hours, sounds like they'll want me working late shifts (ending around 11pm, where my commute has me not getting home until close to midnight).

I'm completely against this, but all my other coworkers of course are mad whenever I “get out of it” because they have to do it.