This is the golden phrase I have memorized for any time I'm asked to work an irregular shift I don't want. It's well memorized, so that I don't have any moment of fluster or pause.
It gives them no details for them to debate. But it's a commitment, so there's no wiggle room. If they ask what exactly it is, I reply “It's private.” If they keep asking, I just keep repeating “It's private” in an increasingly firm tone. They get nowhere.
It works every time for me.