Fedoras worn with anything other than a suit just doesn't work unless you're say, Justin Timberlake. But even then, it only works because he's Justin Timberlake. The whole high/low look works for items like dressy trousers and sport coats, which can easily be worn with denim and more casual items, but it doesn't work for fedoras. To wear a fedora in public is to tell the world you're a card-carrying member of the "forever alone" club.