I used to have the same problem but insert husband, work, friends, family (no kids - just parents). It took me a long time to realize that I was able to spend some time taking care of myself and then I could "be" with them and "care" for them even more than I did when I was spending that time with them previously.
Maybe start small? Carve out an hour a week? Or two half hour blocks a week? Start where you feel comfortable and build. I still have the feelings that "You should be doing this with 'insert person here' instead of exercising" but I remember that I've committed to myself and that means I'm a better person for them. I've also found that scheduling time with the people I love means that I can let that other unscheduled time be just for me a little more guilt free.
Best wishes to you!