Interacting with other people is almost always a game of reading cues and volleying back.
We think a smile conveys happiness, so we offer a smile in return.
We think a frown shows sadness, and maybe we attempt to cheer that person up.
Some businesses are even working on technology to determine customer satisfaction through facial expressions.
But facial expressions might not be reliable indicators of emotion, research indicates. In fact, it might be more accurate to say we should never trust a person’s face, new research suggests.
“The question we really asked is: ‘Can we truly detect emotion from facial articulations?‘” said Aleix Martinez, a professor of electrical and computer engineering at The Ohio State University.
“And the basic conclusion is, no, you can’t.”
Martinez, whose work has focused on building computer algorithms that analyze facial expressions, and his colleagues presented their findings today (Feb. 16, 2020) at the annual meeting of the American Association for the Advancement of Science in Seattle.
The researchers analyzed the kinetics of muscle movement in the human face and compared those muscle movements with a person’s emotions.
They found that attempts to detect or define emotions based on a person’s facial expressions were almost always wrong.
“Everyone makes different facial expressions based on context and cultural background,” Martinez said.
“And it’s important to realize that not everyone who smiles is happy.
Not everyone who is happy smiles.
I would even go to the extreme of saying most people who do not smile are not necessarily unhappy.
And if you are happy for a whole day, you don’t go walking down the street with a smile on your face.
You’re just happy.”
It is also true, Martinez said, that sometimes, people smile out of an obligation to the social norms. This would not inherently be a problem, he said — people are certainly entitled to put on a smile for the rest of the world — but some companies have begun developing technology to recognize facial muscle movements and assign emotion or intent to those movements.
The research group that presented at AAAS analyzed some of those technologies and, Martinez said, largely found them lacking.
“Some claim they can detect whether someone is guilty of a crime or not, or whether a student is paying attention in class, or whether a customer is satisfied after a purchase,” he said.
“What our research showed is that those claims are complete baloney.
There’s no way you can determine those things. And worse, it can be dangerous.”
The danger, Martinez said, lies in the possibility of missing the real emotion or intent in another person, and then making decisions about that person’s future or abilities.
For example, consider a classroom environment, and a teacher who assumes that a student is not paying attention because of the expression on the student’s face.
The teacher might expect the student to smile and nod along if the student is paying attention. But maybe that student, for reasons the teacher doesn’t understand — cultural reasons, perhaps, or contextual ones — is listening intently, but not smiling at all.
It would be, Martinez argues, wrong for the teacher to dismiss that student because of the student’s facial expressions.
After analyzing data about facial expressions and emotion, the research team — which included scientists from Northeastern University, the California Institute of Technology and the University of Wisconsin — concluded that it takes more than expressions to correctly detect emotion.
Facial color, for example, can help provide clues.
“What we showed is that when you experience emotion, your brain releases peptides — mostly hormones — that change the blood flow and blood composition, and because the face is inundated with these peptides, it changes color,” Martinez said.
The human body offers other hints, too, he said: body posture, for example. And context plays a crucial role as well.
In one experiment, Martinez showed study participants a picture cropped to display just a man’s face. The man’s mouth is open in an apparent scream; his face is bright red.
“When people looked at it, they would think, wow, this guy is super annoyed, or really mad at something, that he’s angry and shouting,” Martinez said.
“But when participants saw the whole image, they saw that it was a soccer player who was celebrating a goal.”
In context, it’s clear the man is very happy. But isolate his face, Martinez said, and he appears almost dangerous.
Cultural biases play a role, too.
Research is showing that, when it comes to reading people’s emotions, you can’t simply trust facial expressions.
“In the U.S., we tend to smile a lot,” Martinez said. “We are just being friendly. But in other cultures, that means different things — in some cultures, if you walked around the supermarket smiling at everyone, you might get smacked.”
Martinez said the research group’s findings could indicate that people — from hiring managers to professors to criminal justice experts — should consider more than just a facial expression when they evaluate another person.
And while Martinez said he is “a big believer” in developing computer algorithms that try to understand social cues and the intent of a person, he added that two things are important to know about that technology.
“One is you are never going to get 100 percent accuracy,” he said. “And the second is that deciphering a person’s intent goes beyond their facial expression, and it’s important that people — and the computer algorithms they create — understand that.”
Neutral facial expressions can be interpreted and responded to in different ways, due to lacking or potentially ambiguous information on the emotional state of the person or her relationship towards the interaction partner.
The ability to detect and differentiate neutral from emotional facial expressions develops rather late in childhood (Durand, Gallay, Seigneuric, Robichon, & Baudouin, 2007). Young children tend to wrongly attribute sadness or joy to neutral facial expressions.
During childhood, neutral facial expressions can be perceived as threatening and lead to increased negative affect, decreased positive affect, and physiological stress reactions (Mesman, van IJzendoorn, & Bakermans-Kranenburg, 2009).
Individuals with autism spectrum disorders (Eack, Mazefsky, & Minshew, 2015), depression (Bourke, Douglas, & Porter, 2010), schizophrenia (Kohler et al., 2003), and borderline personality disorder (BPD) (Daros, Zakzanis, & Ruocco, 2013; Mitchell, Dickens, & Picchioni, 2014) show abnormal amygdala responses to neutral facial expressions (Donegan et al., 2003; Harms, Martin, & Wallace, 2010; Sheline et al., 2001) and tend to interpret them as negative.
Studies on facial expressions in PTSD have, to date, focused on deficits in recognition of positive and negative facial expressions (Passardi et al., 2018; Poljac, Montagne, & de Haan, 2011) and on theory of mind (Mazza et al., 2012; Nazarov et al., 2014; Nietlisbach, Maercker, Rösler, & Haker, 2010; Schmidt & Zachariae, 2009); a construct related to but not identical with emotion recognition.
They found deficits in the recognition of positive (Passardi et al., 2018) and negative (fearful and sad) emotions (Poljac et al., 2011) and inconsistent results regarding theory of mind, ranging from generalized deficits (Mazza et al., 2012; Schmidt & Zachariae, 2009) to no abnormalities (Nazarov et al., 2014; Nietlisbach et al., 2010).
Only one study has analysed recognition of neutral facial expressions in adults with PTSD and found no differences compared to healthy controls (Nazarov et al., 2014). However, this study used static images, only depicting the ocular region of the face and has assessed the recognition of complex (positively valenced, negatively valenced and neutral) mental states as part of a theory of mind task. While no study has assessed interpretation of neutral (whole) facial expressions in adults with PTSD, as seen in everyday situations, there is evidence that individuals with PTSD – on a cortical level – fail to differentiate between angry and neutral facial expressions (when asked to watch the expressions passively). The authors interpreted this as difficulty to distinguish between threat and non-threat stimuli (Felmingham, Bryant, & Gordon, 2003).
There is evidence in PTSD-related populations for abnormalities in the interpretation of neutral facial expressions. For example, individuals with BPD – who show a high prevalence of trauma and maltreatment during childhood (Battle et al., 2004; Wingenfeld et al., 2011) – interpret neutral facial expressions as more negative than healthy controls and tend to interpret them as anger (Daros et al., 2013; Mitchell et al., 2014) or sadness (Meehan et al., 2017).
Likewise, individuals with dissociative identity disorder show high rates of childhood traumatization. When in a trauma-related emotional state, they perceive neutral faces as highly threatening (Schlumpf et al., 2013). Aligning with that, individuals with PTSD show high rates of child maltreatment, and child maltreatment is linked to negative interpretations of neutral facial expressions in children (Pollak, Cicchetti, Hornung, & Reed, 2000) and to impaired recognition of neutral facial expressions in adults (Wagner & Linehan, 1999).
In individuals with child maltreatment, neutral expressions might trigger memories of neglect or abuse, potentially contributing to the above-mentioned findings (Daros et al., 2013; Meehan et al., 2017; Mitchell et al., 2014; Pollak et al., 2000; Schlumpf et al., 2013; Wagner & Linehan, 1999).
Other factors that might influence the interpretation of neutral facial expression are alexithymia, the number of experienced trauma types (NOET), or dissociation.
Individuals with PTSD show a heightened prevalence of alexithymia (difficulties describing and identifying feelings (Sifneos, 1973)), which is related to emotion recognition deficits in healthy and clinical populations [for meta-analysis, see Frewen, Dozois, Neufeld, and Lanius (2008)].
In particular, there is some evidence that alexithymia might be associated with deficits in the recognition of neutral facial expressions in clinical and healthy populations, although results are inconclusive [for review see Grynberg et al. (2012)].
In previous analyses of the same sample that was assessed in the present manuscript, we had found that higher NOET and higher state dissociation were linked to deficits in the recognition of positive facial expressions (Passardi et al., 2018).
The present study aimed at assessing whether individuals with PTSD interpret neutral facial expressions more negatively than controls by using stimuli close to facial expressions found in everyday life. We used the same data set for which results on recognition of positive and negative emotions in individuals with PTSD have already been reported (Passardi et al., 2018).
Based on previous studies in individuals with child maltreatment (Pollak et al., 2000; Wagner & Linehan, 1999) and individuals with BPD (Daros et al., 2013; Meehan et al., 2017; Mitchell et al., 2014), we hypothesized that individuals with PTSD would show poorer recognition of neutral expressions and would rate neutral expressions more often as negative (i.e. as expressions of anger, contempt, fear, sadness, disgust, or embarrassment) compared to a healthy traumatized (TC) and a healthy non-traumatized control group (HC).
In addition, we aimed at exploring whether possible abnormities in the interpretation of neutral facial expressions are more closely associated with maltreatment during childhood than to the diagnosis of PTSD.
Assuming that biases seen in BPD or dissociative identity disorder (Daros et al., 2013; Meehan et al., 2017; Mitchell et al., 2014; Schlumpf et al., 2013) might be linked to child maltreatment and associated learning processes (Guitart-Masip et al., 2009; Pollak et al., 2000; Wagner & Linehan, 1999), we predicted that individuals with more traumatic childhood experiences would interpret facial expressions more often as negative than individuals with fewer traumatic childhood experiences.
To preclude that interpreting neutral expressions as more negative was due to a general tendency to misinterpret facial expressions, we also analysed confusing an emotional expression with a neutral expression and expected no differences between individuals with PTSD and controls. Since NOET, alexithymia, and state dissociation are related to emotion recognition deficits in PTSD (Passardi et al., 2018; Passardi, Peyk, Rufer, Wingenbach, & Pfaltz, 2019), we included these variables in the analysis as possible influencing factors.
We furthermore included interpersonal trauma as an additional variable to differentiate between the impact of childhood trauma versus interpersonal trauma.
We used reaction times (RTs) as additional, exploratory outcome measure for all analyses.
The present study assessed recognition and interpretations of neutral facial expressions in individuals with PTSD compared to traumatized and non-traumatized healthy controls. Against our hypothesis, PTSD participants did not perform differently from controls in the recognition of neutral expressions.
They also did not show different confusion patterns than healthy controls. Instead, individuals with high levels of childhood sexual abuse performed worse in the recognition of neutral facial expressions than individuals with low levels of sexual abuse. Furthermore, individuals with high versus low levels of childhood experiences of sexual abuse, physical neglect and emotional abuse interpreted neutral expressions more frequently as contempt and (for sexual abuse and physical neglect) more frequently as anger.
Compared to all other models based on different variables (e.g. alexithymia, NOET) or based on diagnostic groups, the model based on childhood sexual abuse had the best model fit, which suggests that misinterpretations of neutral expressions are best explained by experiences of childhood sexual abuse. Groups did not differ in RTs of correctly identified neutral expressions.
Negative interpretation of neutral expressions and child maltreatment
Our results suggest that the experience of child maltreatment, particularly childhood sexual abuse, but not PTSD is associated with difficulties in the recognition of neutral expressions and with negative interpretations of neutral facial expressions.
Learning effects on negative emotional expressions over the course of the emotion recognition paradigm do not seem to explain negative interpretations of neutral expressions in individuals with high levels of child maltreatment, as these variables were unrelated.
Furthermore, there were no group differences in interpreting emotional expressions as neutral. Thus, interpretations of neutral expressions as expressions of negative emotion do not seem to be a function of a general tendency to misinterpret facial expressions.
To date, only one published study has assessed the interpretation of neutral facial expressions in individuals with PTSD (Nazarov et al., 2014). This study found no differences between individuals with PTSD and healthy controls.
On the one hand, our findings align with this study and confirm its results in a different sample and by means of whole face stimuli that are close to real life, rather than showing the eye region of the face only.
On the other hand, our null finding is surprising, given that other clinical populations with a high prevalence of child maltreatment tend to interpret neutral facial expressions as negative (Battle et al., 2004; Daros et al., 2013; Meehan et al., 2017; Mitchell et al., 2014; Schlumpf et al., 2013; Wingenfeld et al., 2011).
According to our own and other (Catalan et al., 2018) findings, child maltreatment might be a more important predictor of this negativity bias than the presence of mental disorders like PTSD or borderline personality disorder.
It is interesting that interpretations of neutral expressions as emotional expressions in individuals with child maltreatment were restricted to negative facial expressions (anger and contempt) but were not found for positive facial expressions (joy or pride).
This is in line with Catalan et al. (2018), who found that child maltreatment in adults was associated with the attribution of negative emotions (anger and fear) to neutral facial expressions and aligns further with research on individuals with BPD who are hypervigilant towards anger-related cues, have difficulties differentiating angry from neutral faces, and tend to interpret neutral expressions as expressions of anger (Daros et al., 2013; Domes et al., 2008; Donegan et al., 2003; Mitchell et al., 2014).
There is some evidence that neutral facial expressions can trigger traumatic memories in individuals with dissociative identity disorder – a condition associated with severe child maltreatment (Schlumpf et al., 2013).
In an fMRI study in individuals with dissociative identity disorder, masked neutral faces activated brain processes related to arousal/vigilance and specific brain areas that have been linked to re-experiencing of traumatic events; this response was even more pronounced than responses of individuals with dissociative identity disorder to masked angry faces (Schlumpf et al., 2013). As suggested by anecdotal evidence, traumatic childhood experiences like sexual or physical abuse can be preceded, accompanied or followed by neutral facial expressions by the perpetrator.
Individuals with these types of adverse childhood experiences may thus have learned to not trust the seemingly calmness of neutral expressions, due to expectations that neutral situations are followed by aversive experiences (Schlumpf et al., 2013).
Due to hypervigilance towards threat-related cues, the threshold to detect anger (on a continuum from positive to negative facial expressions) might furthermore be lower in individuals with child maltreatment.
Indeed, maltreated children showed attentional preference towards angry facial expressions and a higher sensitivity towards detecting anger (Gibb, Schofield, & Coles, 2009).
Different types of child maltreatment might shape emotion recognition abilities in different ways (Pollak, 2003). In our study, it was childhood sexual abuse which best explained misinterpretations of neutral facial expressions and which was associated with difficulties in the recognition of neutral expressions. Several authors describe the detrimental impact of childhood sexual abuse on psychological, neurocognitive and neurobiological functioning [e.g. De Bellis, Spratt, and Hooper (2011)].
A recent representative epidemiologic study in Germany found that the highest prevalence of complex PTSD, which is characterized by severe impairments in affect dysregulation, negative beliefs about oneself and problems in relationships in addition to the typical symptoms of PTSD (World Health Organization, 2015), is found in individuals with childhood sexual abuse and rape (Maercker, Hecker, Augsburger, & Kliem, 2018).
Together with the association between sexual trauma and poorer performance on social cognition found in individuals with BPD (Preißler, Dziobek, Ritter, Heekeren, & Roepke, 2010), this suggests that childhood sexual abuse may have a particularly pronounced, detrimental impact not only on affected individuals’ cognitive and emotional development but also on social skills, including the ability to correctly identify facial expressions.
In sum, negative interpretations of neutral facial expressions in individuals with a history of child maltreatment may reflect learning processes taking place in neglectful and/or abusive environments.
While these learning processes may be adaptive in the initial environment, e.g. by alerting children to potentially harmful situations, they may have negative consequences on the building and maintenance of interpersonal relationships in adulthood.
Alexithymia, number of trauma types, interpersonal trauma, and dissociation
NOET, interpersonal trauma, alexithymia, and state dissociation seem to play a less relevant (and non-significant) role than childhood sexual abuse for the recognition of neutral facial expressions.
This is in contrast to the recognition of emotionalfacial expressions, for which, according to our previous research, NOET, state dissociation, and alexithymia appeared to play a more important role than PTSD diagnosis itself (Passardi, 2018; Passardi et al., 2019).
However, our results align with neurobiological studies demonstrating that facial emotion recognition and recognition of neutral facial expressions involve different neuronal structures (Fusar-Poli et al., 2009; Jehna et al., 2011) and with our finding that accuracy for neutral facial expressions was neither correlated with accuracy for positive nor for negative facial expressions.
Only few previous studies have analysed the association between alexithymia and recognition of neutral expressions. The results of these studies are inconclusive. In line with the present and our previous (Frewen et al., 2008) findings, evidence more strongly suggests that internal representations of emotional rather than neutral expressions are impaired in alexithymic individuals (Grynberg et al., 2012).
Similarly, state dissociation and higher numbers of experienced trauma types seem to disturb the processing of emotional facial expressions (Passardi et al., 2018) but not of neutral expressions.
Finally, our results suggest that it is not the ‘interpersonal-trauma-aspect’ of child malreatment but specific types of child maltreatment, which are associated with misinterpretations of neutral expressions.
Ohio State University