I do the same thing. I go everywhere I am invited, I just make sure I eat before I go or bring something with me. I want to be included even if I can't partake in eating. You need to stay social and your QOL will stay positive. Make the situation about the people you are spending time with instead of the food you can't have.