Understanding vision – whether from a neurobiological, psychological or philosophical perspective – represents a daunting challenge that has been pursued for millennia. Here, the authors consider the evidence that, with respect to the perception of geometry, the human visual system solves problems by incorporating past human experience of what retinal images have typically corresponded to in the real world. This empirical strategy, documented by extensive analyses of scene geometry, explains many otherwise puzzling aspects of what we see.