It's a mystery at least 3,000 years old. Physicists attributed it to refraction while the “angle-of-regard” hypothesis said it was due to the angle that our eyes (or head) make relative to the horizon. The more your eyes are angled upward, the thinking goes, the smaller something looks, due to the physiology of our visual system.
And then some psychologists today say it is a conflict between the conscious and subconscious mind, because, as Kepler said, “Perception does not belong to optics but to the study of the wonderful.”
Brian Gallagher at Nautilus details the history of the mystery. H/T Real Clear Science