Rowing is definitely the hardest, but you can only understand that once you've actually raced in a shell.

8 people with perfect balance despite height and weight differences, and wind possibly tipping the boat to one side. Equal power on both sides to keep the boat moving relatively straight with the help of a coxswain. Perform the same motion approximately 220 times in a row, in perfect unison, despite fluxuating rate of the team. So you basically can't drift mentally, you have to stay focused on the motion of the guys in front of you. Extreme fatigue, as it is, not only, a full body (legs, back, arms) exertion, but is approximately 6 minutes in length. Runners must only worry about their legs in a mile race while rowers have to handle their entire muscle structure coming into use.