You have to learn to listen to your body. If you're having problems breathing, have fatigue, have body aches, a cough, etc. then you want to lay off any exercise until you feel better. How are you going to exercise if you can't breathe properly ? Which is why you need to rest.
If you have a slight sniffle, some times a little light cardiovascular exercise can help.
Take it easy a few days. You will not derail any of your efforts at good health or weight loss if you can't exercise for a few days because of a flu or cold. Part of living a healthy lifestyle is learning to take care of our bodies. That means getting some rest when we need it.