You could argue him going back to his alma mater (if that is what he ends up doing, who knows) is a form of loyalty in and of itself.
Though I see your point.
I for one, as a Duck fan, hope he leaves. I don't want to have to play Harbaugh coached Stanford teams. He's still the only coach...