You likely won't be the only Clemson fans staying at your hotel. Ask them if they offer a shuttle service to and from the game. Also it is not "that" far from the stadium, I've walked from UofL to downtown a few times before, but I'm not recommending that.
Also 4th Street, main st, and market...