The tricky thing about body weight is that it's not the best indicator of body fat, you can weigh very little and still have a high body fat percentage. It's called skinny fat and it sounds like you have this problem if you have "bits of fat" you're not happy with at your low weight.
At 5'3"/104 pounds I'd say you have no weight to lose. I'm 5'4"/122 pounds and not looking to lose any weight. I have had people (people I trust to tell me the truth), tell me not to lose any weight because then I would be too skinny. I must have more muscle mass than you though.
The best way to combat being skinny fat is to strength train. Lifting heavy weights will help you achieve the results you are looking for. If you continue to eat fairly healthy and lift weights (upper and lower) 2x a week, practice pilates 2x a week and run 1-2x a week with some sprint training you should start to see some great results. kbellqueen.blogspot.ca/2013/01/lets-talk-a
ETA: More is not always better for exercise. The body needs time to heal between workouts, so if you're really sore after a workout take it easy the next day with some easy walking.
Edited by: JUSTEATREALFOOD at: 5/24/2013 (08:05)