What if someone said send you "back to the farm" or "Podunk Iowa".
I would probably not have a huge problem with it, personally. But I could see a scenario where it is not appropriate. If I was one of the only white kids in a school made up almost entirely of people that don't look like me and I was on a team full of black coaches and one of the coaches was yelling it to single me out and make me feel like ****, that would probably not be appropriate.
So why do you think it is OK for Doyle to tell kids he would send them back to the ghetto? Why is this something we need to fight to protect on our college football teams?