I can certiainly make the opposite argument. In fact, you could argue that someone deciding to get into politics cares about their family so much, that they are willing to endure public scrutiny in order to help create a better community for their family. It is hard to say this about any politician given the cynical lense we view them with, but the argument is credible.
As a father, I have no problem being scrutinized for my beliefs and decisions. In fact, I welcome it. My ideas and principles either stand up or they don't, I want my kids to see which they result in.
But getting back to the matter...how again does it not matter?