Everyone does not want to race you.
Ignorant drivers may scream "HEY LANCE ARMSTRONG" as you pedal along on your road bike, but this does not make it so, even if you're draped in LIVESTRONG. And on that same note, just because someone else is out on a racing bicycle in team gear does not mean they want to race. If it's a weekend, they might actually be returning from an actual race or a long training ride. Either way, beating them to the nearest lightpost/crosswalk/Starbucks isn't much of a challenge, if they don't know they're involved. Save the racing for sanctioned events - or at least mutually agreed-upon ones.