Is it ok to have fitness goals if you're not overweight?!

I weigh 186 lb and I'm 6'0". But I do have a bit of muscle. I don't know off hand what my body fat % is, I'm guessing around 15%. I don't feel I am unhealthy, but I'd like to get my body fat down to around 10%. In an ideal world I'd like to lose fat and gain more muscle simultaneously, leaving my weight unchanged, but I don't believe the body works like that so I'm willing to lose a little muscle in the process of losing fat, then increase calories and cut back on cardio to gain that muscle mass back.

Is this a good goal? In a world full of fatties who have legitimate reasons to feel self conscious about their body image, I feel a little stupid telling others about my goals when my overall build is skinny. I feel like they're sitting their thinking "oh if he's trying to lose some weight, he must think I'm a big huge tub of lard"