But if the pay isn't adequate, nothing else matters.
I'm sure people would be willing to move jobs and take a pay cut for a better office culture (or going remote), less hours, better perks, or a better location if it was a choice between two living wages.
So when some asshole hits you with “It really is all about the money to you, isn't it?” don't tell them they're right. Tell them you have your priorities in order, and that you'll give consideration to their other points once your basic needs are met.