I've been inside The Palestra(Franklin Field is also awesome) just not for any games and yes, it's awesome. My folks saw UK play Princeton there in '77 NCAAT. I love all of the Philly gyms and such.
I don't think it's a fear of road games as much as it's a strategy to get ready for tournament play. I think he prefers these neutral sight games because they are more similar to what the team will experience in the tournament.
I understand that is his stated logic, but I have to be honest about it. Coach K says the same thing and I think it's b/c he wants to worm his way out of ever having to play true road games b/c he knows his chances of losing on the road are higher than losing at a neutral site. I think after Cal lost to IU in 2012, he decided that he's just not going to do it anymore unless he has to (i.e. against UL every other year and in the SEC)