Jerry Seinfeld learned the definition of "bad naked" when he began dating a hot blonde nudist who did everything in the buff, from eating waffles to playing Scrabble. When viewed outside of a sexual context, nudity can look, well, unattractive.
While Hannah Horvath/Lena Dunham has received all sorts of attention (most of it negative) for her willingness to disrobe and put all of her imperfections on display, her boyfriend Adam (Adam Driver) spends the majority of season one in only his boxers. To be fair, the two have a very active sex life and Adam does spend a good bit of time pleasuring himself, so maybe it's just a case of easier access. But like you, women like it when you leave something to the imagination.