You can be both athletic and unable to carry weight during long uphill walks.

E.g. upper body dominant sports, or activities not focused on endurance would not be as advantageous here.

Only epakai has mentioned altitude so far. I’m athletic and once took a cable car up to 5000m. At the top I started walking on the flat trail and was out of breath in a minute or two and had to stop.