He was an old, pompous, cantankerous man who had worn the last nerve of those who employed him but he was in no way a racist. He made a very inappropriate comment when he called Alvin Garrett a "little monkey" on MNF, the only problem was that it was a term he'd used many times before in reference to white players. Not to mention what he called his own grandson.
All anyone has to do is look at the way he interacted with Ali, at a time when the initial reaction for most in the media to a loud and opinionated black athlete was not a favorable one, to know that Cosell was far from a racist.