Sometimes it feels great to wear a pair of sweats, especially to work, but don't get in the habit of it. The thing about wearing sweats to work is making sure that no one really pays them too much attention. If you put them on five days a week, though, it makes it a lot more known that you're clearly not the type of person who wears the pants.
Save the day that you're wearing sweats for the time you're breaking out a new pair of sneakers, have the perfect outfit, or for the times that you down too many drinks at an open bar the night before. Just don't look like the guy who always turned up the night before, and that's why he's always clad in sweats.