Encoding-specificity instead of online integration of real-world spatial regularities for objects in working memory

In daily life scenarios, most objects are not independent of each other; rather, they show a high degree of spatial regularity (e.g., beach umbrellas appear above beach chairs, not under them). Previous studies have shown a benefit of spatial regularities in visual working memory (VWM) performance of real-world objects, termed the spatial regularity effect. However, the mechanisms underlying this effect remain unclear. The spatial regularity effect can be explained by an “encoding-specificity” hypothesis or a “perception-alike” hypothesis. The former suggests that spatial regularity will enhance the visual encoding process but will not operate in information integration during VWM maintenance, while the latter suggests that spatial regularity will play a role in both the visual encoding and VWM maintenance processes. We tested these two hypotheses by investigating whether VWM integrates sequentially presented real-world objects by focusing on the existence of the spatial regularity effect. In Experiment 1, we manipulated the presentation (simultaneous vs. sequential) and regularity (with vs. without regularity) of memory arrays among pairs of real-world objects. The spatial regularity of memory objects improved the VWM performance in simultaneous presentation trials, but not in sequential presentation trials. In Experiment 2, we examined whether overburdened memory load hindered the spatial regularity effect in sequential presentation trials. We again found an absence of the spatial regularity effect, regardless of the memory load. These results suggest that participants were unable to integrate real-world objects into pairs based on spatial regularity during the VWM maintenance process. Therefore, the present results support the “encoding-specificity” hypothesis, implying that although the spatial regularity of real-world objects can enhance the efficiency of the encoding process in VWM, VWM cannot exploit spatial regularity to help organize sampled sequential information into meaningful groups.

Developmental Changes in Gaze Behavior and the Effects of Auditory Emotion Word Priming in Emotional Face Categorization

Social interactions often require the simultaneous processing of emotions from facial expressions and speech. However, the development of the gaze behavior used for emotion recognition, and the effects of speech perception on the visual encoding of facial expressions is less understood. We therefore conducted a word-primed face categorization experiment, where participants from multiple age groups (six-year-olds, 12-year-olds, and adults) categorized target facial expressions as positive or negative after priming with valence-congruent or -incongruent auditory emotion words, or no words at all. We recorded our participants’ gaze behavior during this task using an eye-tracker, and analyzed the data with respect to the fixation time toward the eyes and mouth regions of faces, as well as the time until participants made the first fixation within those regions (time to first fixation, TTFF). We found that the six-year-olds showed significantly higher accuracy in categorizing congruently primed faces compared to the other conditions. The six-year-olds also showed faster response times, shorter total fixation durations, and faster TTFF measures in all primed trials, regardless of congruency, as compared to unprimed trials. We also found that while adults looked first, and longer, at the eyes as compared to the mouth regions of target faces, children did not exhibit this gaze behavior. Our results thus indicate that young children are more sensitive than adults or older children to auditory emotion word primes during the perception of emotional faces, and that the distribution of gaze across the regions of the face changes significantly from childhood to adulthood.

The visual encoding of graspable unfamiliar objects

We explored by eye-tracking the visual encoding modalities of participants (N = 20) involved in a free-observation task in which three repetitions of ten unfamiliar graspable objects were administered. Then, we analysed the temporal allocation (t = 1500ms) of visual-spatial attention to objects’ manipulation (i.e., the part aimed at grasping the object) and functional (i.e., the part aimed at recognizing the function and identity of the object) areas. We found a reversed quadratic trend in the way participants visually explored the objects. Within the first 750ms, participants tended to shift their gaze on the functional areas while decreasing their attention on the manipulation areas. Then, participants reversed this trend, decreasing their visual-spatial attention to the functional areas while relatively increasing fixations to the manipulation areas. Crucially, the global amount of visual-spatial attention for objects’ functional areas significantly decreased as an effect of stimuli repetition while remaining stable for the manipulation areas, thus indicating stimulus familiarity effects. These findings support the action reappraisal theoretical approach, which considers object processing and tool use as abilities emerging from the integration among semantic, technical/mechanical, and sensorimotor knowledge.

A Visual Encoding Model Based on Contrastive Self-Supervised Learning for Human Brain Activity along the Ventral Visual Stream

Visual encoding models are important computational models for understanding how information is processed along the visual stream. Many improved visual encoding models have been developed from the perspective of the model architecture and the learning objective, but these are limited to the supervised learning method. From the view of unsupervised learning mechanisms, this paper utilized a pre-trained neural network to construct a visual encoding model based on contrastive self-supervised learning for the ventral visual stream measured by functional magnetic resonance imaging (fMRI). We first extracted features using the ResNet50 model pre-trained in contrastive self-supervised learning (ResNet50-CSL model), trained a linear regression model for each voxel, and finally calculated the prediction accuracy of different voxels. Compared with the ResNet50 model pre-trained in a supervised classification task, the ResNet50-CSL model achieved an equal or even relatively better encoding performance in multiple visual cortical areas. Moreover, the ResNet50-CSL model performs hierarchical representation of input visual stimuli, which is similar to the human visual cortex in its hierarchical information processing. Our experimental results suggest that the encoding model based on contrastive self-supervised learning is a strong computational model to compete with supervised models, and contrastive self-supervised learning proves an effective learning method to extract human brain-like representations.

Individuals with autism spectrum disorder have altered visual encoding capacity

Perceptual anomalies in individuals with autism spectrum disorder (ASD) have been attributed to an imbalance in weighting incoming sensory evidence with prior knowledge when interpreting sensory information. Here, we show that sensory encoding and how it adapts to changing stimulus statistics during feedback also characteristically differs between neurotypical and ASD groups. In a visual orientation estimation task, we extracted the accuracy of sensory encoding from psychophysical data by using an information theoretic measure. Initially, sensory representations in both groups reflected the statistics of visual orientations in natural scenes, but encoding capacity was overall lower in the ASD group. Exposure to an artificial (i.e., uniform) distribution of visual orientations coupled with performance feedback altered the sensory representations of the neurotypical group toward the novel experimental statistics, while also increasing their total encoding capacity. In contrast, neither total encoding capacity nor its allocation significantly changed in the ASD group. Across both groups, the degree of adaptation was correlated with participants’ initial encoding capacity. These findings highlight substantial deficits in sensory encoding—independent from and potentially in addition to deficits in decoding—in individuals with ASD.

DENSE-GWP: AN IMPROVED PRIMARY VISUAL ENCODING MODEL BASED ON DENSE GABOR FEATURES

Brain visual encoding models based on functional magnetic resonance imaging are growing increasingly popular. The Gabor wavelet pyramid model (GWP) is a classic example, exhibiting a good prediction performance for the primary visual cortex (V1, V2, and V3). However, the local variations in the visual stimulation are quite convoluted in terms of spatial frequency, orientation, and position, posing a challenge for visual encoding models. Whether the GWP model can thoroughly extract informative and effective features from visual stimulus remains unclear. To this end, this paper proposes a dense GWP visual encoding model by ameliorating the composition of the Gabor wavelet basis from three aspects: spatial frequency, orientation, and position. The improved model named Dense-GWP model could extract denser features from the image stimulus. A regularization optimization algorithm was used to select informative and effective features, which were crucial for predicting voxel activity in the region of interest. Extensive experimental results showed that the Dense-GWP model exhibits an improved prediction performance and can therefore help further understand the human visual perception mechanism.