It's kind of a weird question to ask what is the hardest because ultimately you should be judged vs. your peers who play the same position. Even though it's hard to defend a PK, other goalies aren't stopping them either and you're graded accordingly.
Sure, Qb is a complicated position, requiring special physical and mental attributes ...
... but I opine the most difficult position to play well is cornerback ... gotta cover well, and gotta play the run wide, and quickly judge shifting priorities ... and when a play goes badly, the whole world is pointing at you ... but gotta shake it off and be ready for the next play.
You have to snap it perfectly 70 times a game while snapping your head up only to see a 330 lb DT/NT trying to run you over. On top of that...oh just point out things all over the field and generally be the brain of the offense.