I've gone to tons of neutral games, and I just wear regular clothes. Sometimes, if I'm shopping before the game and find a home team shirt I really like, I will buy it and wear it, but usually I just wear whatever is comfortable. It's funny to see people trying to figure out which team you are supporting, so if I can I will find a shirt that vaguely represents the colors of both teams.
There is no way I would go out and buy an orange shirt just to attend a Clemson game. When will you ever wear it again? I went to the UGA-Kentucky game a couple years ago, and I threw on a red polo just to make my hosts happy.
I was at the "Kick 6" Iron Bowl and I wore a gray shirt with some maroon stripes, blue jeans, and a blue hat. I was sitting in an Alabama section and at halftime people were demanding to know which team I was supporting.
The only time I wore MSU gear to a neutral game was the first ever SEC Championship game in Birmingham. I figured it was fair game to represent any SEC school.