Distinct representation of locomotive action affordances in human behavior, brains and deep neural networks

To decide how to move around the world, we must determine which locomotive actions (e.g., walking, swimming, or climbing) are afforded by the immediate visual environment. The neural basis of our ability to recognize locomotive affordances is unknown. Here, we compare human behavioral annotations, functional magnetic resonance imaging (fMRI) measurements, and deep neural network (DNN) activations to both indoor and outdoor real-world images to demonstrate that human visual cortex distinctly represents different locomotive action affordances in complex visual scenes. Hierarchical clustering of behavioral annotations of six possible locomotive actions show that humans group environments into distinct affordance clusters using at least three separate dimensions. Representational similarity analysis of multi-voxel fMRI responses in scene-selective visual cortex regions shows that perceived locomotive affordances are represented independent from other scene properties such as objects, surface materials, scene category or global properties, and independent of the task performed in the scanner. Visual feature activations from DNNs trained on object or scene classification as well as a range of other visual understanding tasks fail to fully account for behavioral and neural representations of locomotive action affordances. Model predictions of human-perceived locomotive affordances can be increased by training DNNs directly on affordance labels or using affordance-centered language embeddings, but the best human-model alignment is obtained through probing a multi-modal large-language model (GPT-4). This suggests that pairing visual features with specific linguistic representations may be needed to capture affordance perception in humans. These results uncover a new type of representation in the human brain that reflect locomotive action affordances.