“When I see equations, I see the letters in colors.
I don’t know why,” wrote Nobel Prize-winning physicist Richard Feynman.
“I see vague pictures of Bessel functions with light-tan j’s, slightly violet-bluish n’s, and dark brown x’s flying around.”
Feynman was describing his grapheme-colour (GC) synesthesia – a condition in which individuals sense colours associated with letters and numbers.
Synesthesia is a family of conditions where individuals perceive stimulation through more than one sense.
GC synesthesia is just one form.
People with grapheme-colour synaesthesia experience reliable colour sensations whenever they see letters and/or numbers (Hubbard & Ramachandran 2005; Ward & Mattingley 2006), and sometimes when they hear speech (Baron-Cohen et al. 1993; Paulesu et al. 1995) or think about letters or numbers (Dixon et al. 2000). Ramachandran & Hubbard (2001a) reported an influential experiment to demonstrate the authenticity of grapheme-colour synaesthesia, termed the ‘embedded shapes task’.
They studied two synaesthetes who were shown arrays of achromatic graphemes for a brief period (1 s).
Some of the graphemes were arranged into one of four shapes (diamond, square, rectangle or triangle).
For example, there might be a triangle made up of Hs against a random background of Ps and Fs.
The two synaesthetes did significantly better than the control group (81% correct in synaesthetes versus 59% correct in controls), suggesting that they may have seen the achromatic graphemes as coloured, thus enabling them to see the embedded shape.
One reason why this result was considered a convincing demonstration for the authenticity of synaesthesia is that superior performance on a perceptual task is hard to fake.
This finding was replicated by Hubbard et al. (2005); five of their six synaesthetes performed significantly better than controls. However, Rothen & Meier (2009) failed to replicate the result in a group of 13 synaesthetes.
Other studies have used visual search paradigms in which a single target (e.g. 2), rather than an embedded shape, must be detected among an array of distractor graphemes (e.g. 5s) and response times are measured.
As in the embedded shapes task, the stimuli are physically achromatic but assumed to generate synaesthetic colours, thus facilitating their detection.
Studies using this and related paradigms have yielded mixed results. Some show no benefit at all (n = 23 participants in the following studies combined: Edquist et al. 2006; Sagiv et al. 2006; Gheri et al. 2008), although some single case studies do show a benefit (Smilek et al. 2001, 2003; Palmeri et al. 2002; Laeng et al. 2004).
There are several issues at stake here beyond the replicability of Ramachandran & Hubbard (2001a). First of all, the embedded shapes test has been widely publicized as offering strong proof of the authenticity of synaesthesia and its perceptual nature.
These fundamental claims have been cast into doubt by some researchers (Gheri et al. 2008). Second, the results of Ramachandran & Hubbard (2001a) pose important questions for theories of perception and attention outside the domain of synaesthesia.
For people without synaesthesia, searching for a shape or other target is enhanced if the colour of the target differs from the surrounding distractors (e.g. Treisman & Gelade 1980).
The standard explanation for this is that the colour information is processed automatically (pre-attentively) and in parallel across all the items in the display, so the target shape appears to ‘pop out’.
In situations in which colour does not discriminate between targets and distractors (e.g. all are achromatic, or some distractors are the same colour as the target) then participants are assumed to engage in a more time-consuming strategy in which the focus of attention moves from location to location until the target shape is found.
This is termed ‘serial search’.
Thus, better performance by synaesthetes is often interpreted as a greater reliance on faster ‘pop-out’ and less reliance on slower serial search (Ramachandran & Hubbard 2001b, 2003b).
However, this theory as applied to non-synaesthetic visual search assumes that colour and shape are processed independently.
This assumption does not hold for synaesthesia given that some amount of grapheme processing must be required for the colour to be induced.
As such it is unrealistic to expect synaesthetic colours to behave ‘just like real colours’ on these tasks.
There are at least two possible ways that the mixed findings could be resolved.
The first assumes that attention and serial search is required in synaesthesia, just as it is in visual search for feature conjunctions of colour and shape in non-synaesthetes.
In such situations, synaesthetes may experience a small proportion of graphemes as being coloured (i.e. those graphemes within the window of attention) and this could offer them a modest advantage in the absence of pop-out.
It may enable local grouping on the basis of colour (e.g. detecting one edge of a triangle), or may facilitate rejection of distractors.
It is to be noted that previous studies have not assessed what synaesthetes actually claim to see in these tasks.
By definition, all grapheme-colour synaesthetes claim to see colours under free viewing conditions, but this may not hold true for large arrays of graphemes presented with brief exposure.
The second way of resolving these mixed results is to assume that there are individual differences between synaesthetes.
One noted difference is between synaesthetes who experience colours subjectively bound to the observed grapheme (so-called ‘projectors’), versus those who experience the colour in their mind’s eye (so-called ‘associators’, for whom the colours are often bound to a ‘copy’ of the seen letter on some ‘inner screen’; Dixon et al. 2004; Ward et al. 2007).
Many of the demonstrations of superior performance in embedded shapes/visual search have come from projectors (but see Smilek et al. 2001, 2003; Palmeri et al. 2002; Edquist et al. 2006), leading to the suggestion that projectors experience synaesthetic colours pre-attentively but the more common associators experience them post-attentively (e.g. Dixon & Smilek 2005). Ward et al. (2007) offer a different interpretation of this distinction.
They suggest that both types of synaesthesia require attention for accurate binding of colour to grapheme, but that projectors are more likely to be aware of synaesthetic colours (in brief presentation) because, for these individuals, their synaesthetic percepts are in the same spatial location as the attended stimulus itself.
Other types of grapheme-colour synaesthesia require a shifting/dividing of attention between the location of the stimulus and the location of the colour, and this comes at a cost (slower identification of synaesthetic colours, less awareness of synaesthetic colours when attention is directed elsewhere).
Our present experiment is based closely on the experiments of Ramachandran & Hubbard (2001a) and Hubbard et al. (2005).
As in the preliminary study by Ramachandran & Hubbard (2001a), we used stylized 5s and 2s that are the mirror image of each other.1
These stimuli have been extensively reproduced elsewhere to demonstrate the phenomenon of synaesthesia (e.g. Ramachandran & Hubbard 2001b, 2003a), because low-level visual features cannot be used to disambiguate the graphemes (both consist of two vertical and three horizontal lines).
In addition, we asked synaesthetes to report what they saw on a trial-by-trial basis (e.g. what percentage of graphemes appeared coloured?) and we considered individual differences in the perceived location of synaesthetic colours (projectors versus associators).
In others, musical notes evoke colours; words have associated tastes; sequences of numbers are sensed as points in space; numbers suggest people, like an elderly man or a baby girl.
Many synesthetes have expressed how their condition has enhanced their lives.
When designing a set for a ballet or opera, the American artist David Hockney uses the colours he “sees” in the musical score.
Grammy award-winner Pharrell Williams has said that without his music-to-colour synesthesia, “I’d be lost.”
Now, an experiment led by University of Toronto psychologists has shown for the first time that grapheme-colour synesthesia provides a clear advantage in statistical learning – an ability to discern patterns – which is a critical aspect of learning language.
The result provides insight into how we learn, and how children and adults may learn differently.
According to psychology graduate student Tess Forest, “Our result shows that when people experience the same patterns with more than one sense – for example, aurally and visually – they are better able to learn the patterns.”
Forest is lead author of the paper describing the research, published online recently in the journal Cognition. She is a member of Assistant Professor Amy Finn’s Learning and Neural Development Lab in the department of psychology in the Faculty of Arts & Science.
Finn is senior author on the paper and co-authors include researchers from the department of psychology at the University of California, Berkeley.
The team obtained its result with an experiment in which subjects listened to an artificial “language” comprising nonsense words – for example, “muh-keh” and “beh-od.”
They were then asked to listen to a second set of words that included the artificial words, plus new artificial words they had not previously heard.
The new artificial words contained combinations of syllables not included in the original artificial language; in other words, the new artificial words represented a “foreign” language to the participants.
Participants were then asked to guess whether a word was part of the original artificial language or not.
The result?
GC synesthetes scored higher at distinguishing between the two “languages” than other participants in the experiments.
“You can think about it this way:
The GC synesthetes have twice the number of senses providing the same information,” says Forest.
Not only are they sensing the patterns of syllables by listening, they are also sensing those patterns using synesthesia-evoked colours.
“This result is important for thinking about how we learn,” says Forest, “because real-world learning outside the lab uses multiple senses.
Our result sheds light on the learning mechanisms in the brain that might be supporting statistical learning.”
One focus of Finn’s overall research is understanding how children and adults learn differently.
“This result has clear implications for understanding how children and adults might learn differently,” says Finn.
“This study shows that you can learn more when you have redundant information, that is, information perceived through more than one sense.
“This could be truer in infants if they are synesthetes – though I don’t think the evidence is very clear on that question yet.
But, it’s something that could be more true in kids because their developing attentional systems make them less able to focus on only one thing at a time.
When this information is redundant, this broader focus, could be quite useful for learning.”
More information: Tess Allegra Forest et al. Superior learning in synesthetes: Consistent grapheme-color associations facilitate statistical learning, Cognition (2019). DOI: 10.1016/j.cognition.2019.02.003
Journal information: Cognition
Provided by University of Toronto
