Your body does most of it's changing when you sleep and when you rest! Don't think you have to do hours of exercise 7 days a week. Sometimes several days of rest after weeks of vigorous exercise will actually help you make the gains you are after. (ie. speed, strength)
Plan for some down time every 8 weeks if you are a daily, vigorous exercise-type!