Anyone with an interest in men's style has been told the mantra of "the less you wash your clothes, the better." This might have started around the time when everyone got into raw denim, but obviously your sweats are not that. So wash them.
Putting your sweats through the spin cycle after every two wears or so will keep them from sagging around the knees and in other spots. Higher-quality fabrics like Nike Tech Fleece are made from a different composite and don't give as easy. There are also sweats from brands such as John Elliott + Co. and Ronnie Fieg's Kith that have reinforced knees, which keeps the proportions consistent for longer.