Yes it makes a significant difference.
Compare the average likely daily burn of an office worker with a waitress. Neither is 'exercising', but the waitress is going to, on average, burn more calories.
Non Exercise Active Thermogenesis
I watched a show on NEAT where they had a guy do various things like walk if he could get to the destination within 20 minutes, do ironing daily, walk his kids to school, hand-wash his dishes. He gained weight. All that extra activity made him hungry and he upped his intake.
On the other side, you might also have to watch it the other way. My other half has just moved from a physical daily job into a sit down office job. He's going to have to watch his intake - if he eats like he always has, he will probably gain weight. Not enough NEAT.
Deb, in New Zealand