Even as a Gator fan I can tell you that the fraternity guys are absolutely the ones to stay away from. And that includes the ones that are recent alumns (say, the last five years) because they start tailgating the night before and work themselves into a frenzy.
The problem is much worse if you end up parking offsite, but within walking distance of the stadium, because then you walk by all these lots full of drunk obnoxious fans who look up in their hazey state of consciousness, see you in your UT regalia, and just start screaming profanities at you.
By the stadium, its fine, because you have crowds and police around and that tends to tame everyone quite a bit. There's usually at least one loud mouth drunk within about 7 or 8 rows of you, and they will be looking for a fight. I think I've seen at least one guy like that at every game I've been to for my entire life -- and that's not just at Gator football games. I think its in the Constitution or something.
At any rate, my advice to minimize chances of anything other than a little good nature ribbing: 1) either pay for the parking pass right next to the stadium or park out at the mall and take a shuttle bus. Just avoid walking by fraternities or tailgating parties as much as you possibly can; 2) If some guy in the stands starts yelling at you from 30 seats away, assume he's a drunk and just don't even acknolwedge him. For God's sake don't yell back, because that's what he want so he can show off for his friends. If you ignore him, he'll call you a few choice names for 30 seconds but then he'll give up and look for someone else to taunt so he can be a bigshot; 3) if UT happens to pull the upset, be happy and joyous and what not, but remember where you are and save the singing "It's great" yada yada yada for the drive home.