Our bodies get used the whatever routine is in place - food, calories, exercise. So I agree with LOLEMA. Change is what you need.
Instead of swimming more, try doing something different once or twice a week. Go cycling, dancing, bowling, play tennis or racquetball. If you walk or swim for 30 or 45 minutes, try doing it for an hour and really push yourself. You could also break it into two shorter more intense daily sessions - weight lifting or using stretch bands in the morning, and aerobics like Zumba in the evening.
If you eat three meals and two snacks, change that up. Maybe right now you eat a big breakfast, lite lunch and medium dinner with just fruit or veg stix for snacks. Try instead having a lighter breakfast & yogurt for AM break, sandwich wrap and cup of soup for lunch; cheese & crackers for PM break; and big dinner salad.
Or vary your calories. I heard of a program where you basically cut your calories in half for three or four days a month then eat your normal range the other days.
Maybe you need less protein that you think, or you have cut your fat too much. You want about 3 servings including at least one heart healthy type - avocado, olive oil, fish oil. No less and no more.
Please let us know how that goes for you and what works best for you.