You know what you owe other people outside your family? Zero. Nothing. When invited, you can simply say sorry, I can't make it. When invited to bring your mom somewhere, have the words ready, "oh, sorry, I can't, I'm busy that day." You owe nothing more than that. Their reaction is their problem, not yours, and you don't have to explain how you feel...