Think about buying one of those electric car dryers if you have bad luck with (or little patience for) towel drying. It's basically like a big hair dryer for your car. One nice bonus is you can blow water out of all those little crevices in your car -- like the ones around your side mirrors, trunk, or hatch. You can't do that with a towel.
Just drive off after you've washed your car without drying it. You'll get spots. If you're willing to go to all the work of washing your car by hand, dry it off so it looks nice. That'd be like getting the sweetest pair of kicks ever, and then wearing them to go off-road BMXing in the mud.