The Pseudoscientific Structure of “Perceptual Decision-Making” and other “Signal Detection Theory"-based Research (or How Science Works When It Works)

The assumptions and formulas of “Signal Detection Theory” (SDT) dominate psychophysics and neuroscience, and serve as the basis of visual neuroscience under the rubric of “perceptual decision-making.” Here, I discuss how the overly simple, ad hoc assumptions of SDT served to rationalize the chronic failure of traditional psychophysics to achieve reliable results; how the constraints on outcomes imposed by the traditional methods combined with SDT to artificially immunize core assumptions from empirical challenge; and how consequently, research activity has been reduced to a seemingly uncomplicated - yet still non-replicable - matter of mere measurement and correlation. I contrast the structure of this ever-barren approach to the structure of research that respects reality and expands our knowledge of the natural world.

Deep Learning Networks and Visual Perception

Deep learning is an approach to artificial intelligence (AI) centered on the training of deep artificial neural networks to perform complex tasks. Since the early 21st century, this approach has led to record-breaking advances in AI, allowing computers to solve complex board games, video games, natural language-processing tasks, and vision problems. Neuroscientists and psychologists have also utilized these networks as models of biological information processing to understand language, motor control, cognition, audition, and—most commonly—vision. Specifically, early feedforward network architectures were inspired by visual neuroscience and are used to model neural activity and human behavior. They also provide useful representations of the perceptual space of images. The extent to which these models match data, however, depends on the methods used to characterize and compare them. The limitations of these feedforward neural networks to account for, for example, simple visual reasoning tasks, suggests that feedback mechanisms may be necessary to solve visual recognition tasks beyond image categorization.

Topology-preserving smoothing of retinotopic maps

Retinotopic mapping, i.e., the mapping between visual inputs on the retina and neuronal activations in cortical visual areas, is one of the central topics in visual neuroscience. For human observers, the mapping is obtained by analyzing functional magnetic resonance imaging (fMRI) signals of cortical responses to slowly moving visual stimuli on the retina. Although it is well known from neurophysiology that the mapping is topological (i.e., the topology of neighborhood connectivity is preserved) within each visual area, retinotopic maps derived from the state-of-the-art methods are often not topological because of the low signal-to-noise ratio and spatial resolution of fMRI. The violation of topological condition is most severe in cortical regions corresponding to the neighborhood of the fovea (e.g., < 1 degree eccentricity in the Human Connectome Project (HCP) dataset), significantly impeding accurate analysis of retinotopic maps. This study aims to directly model the topological condition and generate topology-preserving and smooth retinotopic maps. Specifically, we adopted the Beltrami coefficient, a metric of quasiconformal mapping, to define the topological condition, developed a mathematical model to quantify topological smoothing as a constrained optimization problem, and elaborated an efficient numerical method to solve the problem. The method was then applied to V1, V2, and V3 simultaneously in the HCP dataset. Experiments with both simulated and real retinotopy data demonstrated that the proposed method could generate topological and smooth retinotopic maps.

Spatial structure, phase, and the contrast of natural images

The sensitivity of the human visual system is thought to be shaped by environmental statistics. A major endeavour in visual neuroscience, therefore, is to uncover the image statistics that predict perceptual and cognitive function. When searching for targets in natural images, for example, it has recently been proposed that target detection is inversely related to the spatial similarity of the target to its local background. We tested this hypothesis by measuring observers' sensitivity to targets that were blended with natural image backgrounds. Importantly, targets were designed to have a spatial structure that was either similar or dissimilar to the background. Contrary to masking from similarity, however, we found that observers were most sensitive to targets that were most similar to their backgrounds. We hypothesised that a coincidence of phase-alignment between target and background results in a local contrast signal that facilitates detection when target-background similarity is high. We confirmed this prediction in a second experiment. Indeed, we show that, by solely manipulating the phase of a target relative to its background, the target can be rendered easily visible or completely undetectable. Our study thus reveals a set of image statistics that predict how well people can perform the ubiquitous task of detecting an object in clutter.

Diffeomorphic Registration for Retinotopic Maps of Multiple Visual Regions

Retinotopic map, the mapping between visual inputs on the retina and neuronal responses on cortical surface, is one of the central topics in vision science. Typically, human retinotopic maps are constructed by analyzing functional magnetic resonance responses to designed visual stimuli on cortical surface. Although it is widely used in visual neuroscience, retinotopic maps are limited by measurement noise and resolution. One promising approach to improve the quality of retinotopic maps is to register individual subject’s retinotopic maps to a retinotopic template or atlas. However, none of the existing retinotopic registration methods has explicitly quantified the diffeomorphic condition, that is, retinotopic maps can be aligned by stretching/compressing but without tearing up. Here, we developed Diffeomorphic Registration for Retinotopic Maps (DRRM) to simultaneously align retinotopic maps in multiple visual regions under the diffeomorphic condition. Specifically, we used the Beltrami coefficient to model the diffeomorphic condition and performed surface registration based on retinotopic coordinates. The overall framework is simple and elegant and preserves topological condition defined in the atlas. We further developed a unique performance evaluation protocol and compared the performance of the new method with several existing image intensity-based registration methods on both synthetic and real datasets. The results showed that DRRM is superior to the existing methods in achieving diffeomorphic mappings in synthetic and empirical data from 3T and 7T magnets. DRRM may improve the interpretation of low-quality retinotopic maps and facilitate adoption of retinotopic maps in clinical settings.

Using deep neural networks to evaluate object vision tasks in rats

In the last two decades rodents have been on the rise as a dominant model for visual neuroscience. This is particularly true for earlier levels of information processing, but a number of studies have suggested that also higher levels of processing such as invariant object recognition occur in rodents. Here we provide a quantitative and comprehensive assessment of this claim by comparing a wide range of rodent behavioral and neural data with convolutional deep neural networks. These networks have been shown to capture hallmark properties of information processing in primates through a succession of convolutional and fully connected layers. We find that performance rodent object vision tasks can be captured using low to mid-level convolutional layers only, without any convincing evidence for the need of higher layers known to simulate complex object recognition in primates. Our approach also reveals surprising insights on assumptions made before, for example, that the best performing animals would be the ones using the most abstract representations–which we show to likely be incorrect. Our findings suggest a road ahead for further studies aiming at quantifying and establishing the richness of representations underlying information processing in animal models at large.

An ecologically motivated image dataset for deep learning yields better models of human vision

Deep neural networks provide the current best models of visual information processing in the primate brain. Drawing on work from computer vision, the most commonly used networks are pretrained on data from the ImageNet Large Scale Visual Recognition Challenge. This dataset comprises images from 1,000 categories, selected to provide a challenging testbed for automated visual object recognition systems. Moving beyond this common practice, we here introduce ecoset, a collection of >1.5 million images from 565 basic-level categories selected to better capture the distribution of objects relevant to humans. Ecoset categories were chosen to be both frequent in linguistic usage and concrete, thereby mirroring important physical objects in the world. We test the effects of training on this ecologically more valid dataset using multiple instances of two neural network architectures: AlexNet and vNet, a novel architecture designed to mimic the progressive increase in receptive field sizes along the human ventral stream. We show that training on ecoset leads to significant improvements in predicting representations in human higher-level visual cortex and perceptual judgments, surpassing the previous state of the art. Significant and highly consistent benefits are demonstrated for both architectures on two separate functional magnetic resonance imaging (fMRI) datasets and behavioral data, jointly covering responses to 1,292 visual stimuli from a wide variety of object categories. These results suggest that computational visual neuroscience may take better advantage of the deep learning framework by using image sets that reflect the human perceptual and cognitive experience. Ecoset and trained network models are openly available to the research community.