What is this either or? He says you should take a rest day, you take one, so who's right? You both are.
Are you saying you'd prefer to ditch the rest day for more exercise?
You should not 'challenge' yourself daily. Some exercise will be pushing your cardio/strength limits, other exercise sessions will be more about maintaining fitness. Some light cardio on a 'rest' day is normally fine.
Most people do benefit from at least one day a week of little to no exercise, and if you're training very hard then one week out of eight with little to no exercise as well.
You don't drive your car at maximum revs everywhere you go. Neither should you run your body like that.
Deb, in New Zealand