Simply put, the idea behind this custom is that you never wear the crest, logo, or combined colors of another school. Something that is more a hallmark of the “Ivy League” era of education, it would be significantly frowned upon if you were, for example, a Harvard student, and you were caught with a Yale sweater on.
This ideology lends itself to isolating the other, and embracing those who fit into your grouping. As Ivy League schools and other prestigious institutions of higher learning have historically remained racially isolated and built up figurative and literal walls with the aim of keeping diversity at bay, wearing one's school crest was not only displaying pride in the college, but also a reminder that one was wealthy, well-educated, and "belonged," whereas someone who was crestless was outside of this grouping.
Today, that kind of thinking really only applies to wearing rival school colors on a major game day. If you've applied to college in the last 20 years, you can recall picking up a T-shirt or two at the bookstores of the various schools you were considering as you went through the application process. In the early 20th century, when guys were known as “Harvard Men,” directly associated with their university, that policy of university exclusivity made more sense. Today, curating your wardrobe based on the colors of your university and its rivals seems not only excessive, but ridiculous.