Stop telling your boss why you can’t come in on your day off.

You are not obligated to tell them why you can't come in. If you do, they will do everything they can to get you to still come in. The only reason they want to know is so they can use that information to get you to think work is more important than whatever you have going on. A simple “I'm sorry, I am not able to come in.” is all you need to say.