People with autism spectrum disorder have difficulty interpreting facial expressions.
Using a neural network model that reproduces the brain on a computer, a group of researchers based at Tohoku University have unraveled how this comes to be.
“Humans recognize different emotions, such as sadness and anger by looking at facial expressions. Yet little is known about how we come to recognize different emotions based on the visual information of facial expressions,” said paper coauthor, Yuta Takahashi.
The research group employed predictive processing theory to help understand more. According to this theory, the brain constantly predicts the next sensory stimulus and adapts when its prediction is wrong. Sensory information, such as facial expressions, helps reduce prediction error.
After this, the clusters of emotions were self-organized into the neural network model’s higher level neuron space – without the model knowing which emotion the facial expression in the video corresponds to.
The model could generalize unknown facial expressions not given in the training, reproducing facial part movements and minimizing prediction errors.
Following this, the researchers conducted experiments and induced abnormalities in the neurons’ activities to investigate the effects on learning development and cognitive characteristics. In the model where heterogeneity of activity in neural population was reduced, the generalization ability also decreased; thus, the formation of emotional clusters in higher-level neurons was inhibited.
This led to a tendency to fail in identifying the emotion of unknown facial expressions, a similar symptom of autism spectrum disorder.
According to Takahashi, the study clarified that predictive processing theory can explain emotion recognition from facial expressions using a neural network model.
“We hope to further our understanding of the process by which humans learn to recognize emotions and the cognitive characteristics of people with autism spectrum disorder,” added Takahashi. “The study will help advance developing appropriate intervention methods for people who find it difficult to identify emotions.”
Individuals with autism spectrum disorder (ASD) often have difficulty interpreting and regulating their own emotions, understanding the emotions expressed by others, and labeling emotions based on viewing the faces of others (Harms et al., 2010; Uljarevic and Hamilton, 2013; Sheppard et al., 2016).
These differences can contribute to social self-isolation by those with ASD either when others respond negatively if the person with ASD lacks a typical, socially expected response, or if the person with ASD chooses to socially isolate themselves to avoid possibly stressful interactions if they realize they struggle to recognize and respond appropriately to expressions of emotion by others (Jaswal and Akhtar, 2019).
Research investigating facial emotion recognition (FER) in ASD has primarily utilized static images composed of posed facial expressions (Pelphrey et al., 2007; Monk et al., 2010); however, more recent research has begun exploring the use of dynamic video with actors making posed facial expressions (Golan et al., 2015; Fridenson-Hayo et al., 2016; Simões et al., 2018). Few studies have utilized face stimuli of humans expressing genuine, spontaneous expressions of emotion, whether static or dynamic (Cassidy et al., 2014).
This distinction is important because research has shown that the human brain, and artificial intelligence (AI) systems, process posed facial expressions differently compared to how spontaneous expressions of emotion are processed (Hess et al., 1989; Schmidt et al., 2006; Wang et al., 2015; Park et al., 2020).
Results have been mixed with most studies indicating that posed expressions of emotion being easier to recognize than those that are spontaneous (Naab and Russell, 2007); however, accuracy for FER may also depend on the specific emotion being evaluated (Faso et al., 2014; Sauter and Fischer, 2018). This may be due to the prototypical nature of posed expressions (e.g., most people show fewer teeth when they smile for posed pictures; Van Der Geld et al., 2008), whereas there is much more variability in genuine expressions of some feelings such as sadness (Krumhuber et al., 2019).
Therefore, it has been proposed that the traditional use of posed facial expression stimuli in research may have artificially inflated behavioral measures of accuracy during emotion recognition tasks (Sauter and Fischer, 2018). Therefore, the historically prevalent use of posed facial expression stimuli in ASD research investigating FER may contribute to the mixed results seen in this research area.
How might these dissimilarities in posed vs. spontaneous facial expression stimuli be perceived differently by those with ASD? This review further argues that posed vs. genuine emotion is a critical factor that deserves more consideration when studying FER in ASD. We will first review the rich literature on the perception of posed facial expressions of emotion, highlighting the differences between ASD and control groups, though inconclusively.
We will then discuss some recent research investigating how individuals with ASD differ from controls when asked to produce posed facial expressions of emotion and review the latest advances in the field of posed vs. spontaneous/genuine facial expressions and implications into autism research in terms of both perception and production of genuine facial expressions. Finally, based on these findings, we propose a method of teaching FER for individuals with ASD.
Differences in FER in ASD
Autism studies investigating differences in understanding how others think or feel date back to as early as the 1970s (Langdell, 1978; Mesibov, 1984; Weeks and Hobson, 1987; Hobson et al., 1988; Ozonoff et al., 1990). In Langdell (1978) they found that adolescents with autism could identify schematically drawn happy and sad faces, but they demonstrated varying capability when sorting the faces just using the eye area.
Another study (Hobson, 1986) provided further convincing evidence about the differences in the appraisal of facial expressions of emotion by children with autism suggesting that their failure to understand the emotional states of others might be related to their difficulty in recognizing the difference between particular emotions. However, due to different experimental designs (e.g., sorting, matching, and cross-modal), the interpretation of these early results is often debatable (Celani et al., 1999).
A more systematic study about the nature of early differences in social cognition in autism was conducted in Dawson et al. (2004) using high-density event-related potentials (ERPs). It was found that children with ASD, as young as 3 years of age, showed a disordered pattern of neural responses to emotional stimuli such as fearful vs. neutral facial expressions. More specifically, typically developing children demonstrated a larger early negative component and a negative slow wave to the fear than to the neutral, while children with autism did not show significant differences in both experiments.
In contrast, the faster speed of early processing of the fear face among children with autism was associated with better performance on tasks assessing social attention such as social orienting, joint attention, and attention to distress. These findings have served as direct evidence for atypical psychological components involving emotion recognition among children with autism at a young age (3–4 years old).
To probe into the pathology of the underlying processes related to dysfunction in emotional and social cognition, it has been shown that amygdala dysfunction in ASD might contribute to a different ability to process social information (Adolphs et al., 2001). Varying face perception or emotion recognition in ASD might result from atypical fixations onto faces, which may, in turn, arise from amygdala dysfunction (Breiter et al., 1996; Baron-Cohen et al., 2000). This hypothesis is directly supported by evidence from both single-neuron recordings in the human amygdala (Rutishauser et al., 2013) and neuroimaging studies (Dalton et al., 2005; Kliemann et al., 2012).
Given the critical role of the amygdala in emotion processing (Adolphs, 2008), more systematic studies will be needed to reveal whether the amygdala has a different response for posed vs. genuine emotions. Further studies using visual scanning/eye-tracking (Pelphrey et al., 2002), or functional neuroimaging (Dalton et al., 2005; Pelphrey et al., 2005), have shown abnormal activity in patients with ASD.
Even with the enhanced emotional salience of facial stimuli, a positron emission tomography (PET) study showed that adults with ASD demonstrated lower activity in the fusiform cortex than typically developing (TD) controls and differed from the TD group within other brain regions (Hall et al., 2003). This line of research was further extended into the identification of differences in key components of human face processing systems that might contribute to the differences in processing facial expressions of emotion (Pelphrey and Carter, 2008).
Unlike previous studies employing more simplistic stimuli (e.g., the face stimulus as an exemplar of a given emotion, “100% expression”), subtle differences in FER were considered (Law Smith et al., 2010; Black et al., 2020). Using stimuli that incrementally morphed the expression between a neutral face and the posed expression, they found that adolescents and young adults with ASD were less accurate at identifying basic emotional expressions of disgust, anger, and surprise. In a follow-up study (Kennedy and Adolphs, 2013), adults with ASD were found to give ratings that were significantly less sensitive to a given emotion and less reliable across repeated testing.
Therefore, an overall decreased specificity in emotion perception suggests a subtle but specific pattern of differences in facial emotion perception among those with ASD. Along this line of research, significant differences were found between males and females with ASD for emotion recognition but not for self-reported empathy recognition (Sucksmith et al., 2013).
Most recently, a gender-biased study showed that differences in FER in females with autism might not be attributed to ASD but instead to their co-occurring alexithymia (difficulty describing one’s own emotions and those of others; Ola and Gullon-Scott, 2020). Thus, consideration for future FER studies is to recruit significant numbers of male and female participants with ASD and consider sex as a factor in the analysis.
We note that there have been several excellent review articles about research findings of FER in ASD (Harms et al., 2010; Bons et al., 2011; Nuske et al., 2013; Uljarevic and Hamilton, 2013). In Harms et al. (2010), demographic and experiment-related factors are addressed to account for inconsistent findings in behavioral studies of FER in ASD. Future studies of FER in ASD suggested by Harms et al. (2010) include the incorporation of longitudinal designs to examine the developmental trajectory of FER and behavioral and brain imaging paradigms that include young children. In Uljarevic and Hamilton (2013), a formal meta-analytic study has shown that recognition of happiness was only marginally modified in ASD, but recognition of fear was marginally worse than recognition of happiness.
In Nuske et al. (2013), it was found that (1) emotion-processing differences might not be universal to all individuals with ASD and are not specific to ASD; and (2) the specific pattern of emotion-processing strengths and weaknesses observed in ASD, involving difficulties with processing social vs. nonsocial, and complex versus simple emotional information, appears to be unique to ASD (Tang et al., 2019). It is also worth noting the “double empathy problem” described (Milton, 2012).
It was found that just like people with ASD have difficulty interpreting the facial emotions of TDs, TD people have just as much difficulty understanding people with autism. Such a “double” perspective has profound implications for ASD service providers because differences in neurology could lead to differences in sociality. A more recent study (Milton and Sims, 2016) has demonstrated a need for less focus on remediation for patients with autism. Instead, it advocated for focusing on limiting social isolation as a more constructive solution. The most recent study (Crompton et al., 2020) has shown that peer-to-peer information transfer concerning autism is more effective than information transfer between persons with and without autism.
Given the finding that FER differences are not strictly applicable to those with ASD (Nuske et al., 2013), several studies have been conducted to compare differences in FER in ASD with other neurological disorders. In Wong et al. (2012), emotion recognition abilities are examined for three groups of children aged 7–13 years: high functioning autism (HFA), social phobia (SP), and TD. Although no evidence was found for negative interpretation biases in children with HFA or SP, children with HFA were found to detect mild affective expressions less accurately than TD peers suggesting subtle changes in emotion expression are more difficult for those with ASD.
In Sachse et al. (2014), a similar study was conducted with adolescents and adults with HFA, schizophrenia (SZ), and TD to identify convergent and divergent mechanisms between ASD and SZ. It was found that individuals with SZ were comparable to TD in all emotion recognition measures, but the basic visuoperceptual abilities of the SZ individuals were reduced. By contrast, the HFA group was more affected in recognizing basic and complex emotions when compared to both SZ and TD.
As reported in Sachse et al. (2014), group differences between SZ and ASD remained but only for recognizing complex emotions after taking facial identity recognition into account. Such experimental results suggest that (1) there is an SZ subgroup with predominantly paranoid symptoms that do not show problems in FER but visuoperceptual differences only; and (2) no shared FER difference was found for paranoid SZ and ASD, implying differential cognitive underpinnings of ASD and SZ about FER.
A study by Lundqvist (2015) directly links sensory abnormality with social dysfunction of ASD – for example, hyper-responsiveness to touch mediated social dysfunction in ASD, and the tactile sensory system is foundational for social functioning in ASD. There is also evidence that social functioning in those with ASD is impacted by se
nsory dysregulation in multiple sensory modalities that arise early in the progression of the disorder (Thye et al., 2018). This meta-analysis suggests an early intervention that targets sensory abnormalities and social differences, considering the critical role ASD sensory processing differences play in social interactions. In another systematic review and meta-analysis (Zhou et al., 2018), quantitative comparisons of sensory temporal acuity were made between healthy controls and two clinical groups (ASD and SZ).
They revealed a consistent difference in multisensory temporal integration in ASD and SZ, which may be associated with differences in social communication. Finally, studying differential patterns of visual sensory alternation using neuroimaging (Martínez et al., 2019) has shown that SZ and ASD participants demonstrated similar FER and motion sensitivity differences, but significantly different visual processing contributed to FER group differences. This data would suggest that FER differences are not unique to ASD.
reference link : https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8300960/
Original Research: Open access.
“Neural network modeling of altered facial expression recognition in autism spectrum disorders based on predictive processing framework” by Yuta Takahashi, Shingo Murata, Hayato Idei, Hiroaki Tomita & Yuichi Yamashita. Scientific Reports
