There are grey sweatpants, black sweatpants, navy sweatpants, and ones with paneled designs and prints. The latter should be the ones that are best reserved for wearing off-duty or not at all. Simple, mono-colored sweats are typically the best for the office space, and will certainly draw the least attention to what's on your lower half.
There are, of course, exceptions to this rule—so it's best to use your own discretion. But if you're not trying to get flack for wearing sweats: the plainer the pants, the better.