Contextual information contributes to the distortion of perception in short-term memory

We learned it as children: to cross the street in exemplary fashion, we must first look to the left, then to the right, and finally once more to the left.

If we see a car and a cyclist approaching when we first look to the left, this information is stored in our short-term memory.

During the second glance to the left, our short-term memory reports: bicycle and car were there before, they are the same ones, they are still far enough away. We cross the street safely.

This is, however, not at all true. Our short-term memory deceives us. When looking to the left the second time, our eyes see something completely different: the bicycle and the car do not have the same colour anymore because they are just now passing through the shadow of a tree, they are no longer in the same location, and the car is perhaps moving more slowly.

The fact that we nonetheless immediately recognise the bicycle and the car is due to the fact that the memory of the first leftward look biases the second one.

Scientists at Goethe University, led by psychologist Christoph Bledowski and doctoral student Cora Fischer reconstructed the traffic situation – very abstractly – in the laboratory: student participants were told to remember the motion direction of green or red dots moving across a monitor.

During each trial, the test person saw two moving dot fields in short succession and had to subsequently report the motion direction of one of these dot fields.

In additional tests, both dot fields were shown simultaneously next to each other. The test persons all completed numerous successive trials.

The Frankfurt scientists were very interested in the mistakes made by the test persons and how these mistakes were systematically connected in successive trials. If for example the observed dots moved in the direction of 10 degrees and in the following trial in the direction of 20 degrees, most people reported 16 to 18 degrees for the second trial.

However, if 0 degrees were correct for the following trial, they reported 2 to 4 degrees for the second trial. The direction of the previous trial therefore distorted the perception of the following one – “not very much, but systematically,” says Christoph Bledowski. He and his team extended previous studies by investigating the influence of contextual information of the dot fields like colour, spatial position (right or left) and sequence (shown first or second).

“In this way we more closely approximate real situations, in which we acquire different types of visual information from objects,” Bledowski explains. This contextual information, especially space and sequence, contribute significantly to the distortion of successive perception in short-term memory. First author Cora Fischer says: “The contextual information helps us to differentiate among different objects and consequently to integrate information of the same object through time.”

What does this mean for our traffic situation? “Initially, it doesn’t sound good if our short-term memory reflects something different from what we physically see,” says Bledowski. “But if our short-term memory were unable to do this, we would see a completely new traffic situation when we looked to the left a second time.

That would be quite confusing, because a different car and a different cyclist would have suddenly appeared out of nowhere. The slight ‘blurring’ of our perception by memory ultimately leads us to perceive our environment, whose appearance is constantly changing due to motion and light changes, as stable. In this process, the current perception of the car, for example, is only affected by the previous perception of the car, but not by the perception of the cyclist.”

---

Memory’s most compelling illusion is that it represents the past. However, it is clear from an ecological perspective that memory is all about the future. The purpose of memory is to learn about the environment to anticipate future demands – not just putting back the pieces of the past for recollection (remembering experience) but deriving possibilities based on the past to guide future adaptive behavior (premembering experience).

This notion has old roots (Helmholtz, 1867), but, surprisingly, the fundamental operations through which memories guide adaptive behavior lack an established theoretical framework, and their mechanisms have yet to capture the in-depth and systematic investigation they deserve.

We propose that memory, over a broad hierarchy of timescales, supplies the essential elements for selective attention to guide perception and performance flexibly and adaptively. We introduce the term “premembering” to capture this prospective and dynamic role of memory.

Our construct complements proposals for how memories can be used to inform other cognitive functions, such as adaptive control, decision-making, and imagining future situations (Box 1).

BOX 1

Prospective Memories in Different Cognitive Domains

We focus on how memories guide selective attention. The role memory plays in guiding behavior has also been considered within other cognitive domains.

• “Cognitive control” refers to the collection of mechanisms that set and adjust our goals and that monitor and regulate performance according to competing demands. Thus, the selective attention mechanisms we review are subordinate to cognitive control, prioritizing and selecting putative targets and overcoming irrelevant distraction within a particular goal setting. Memories are increasingly recognized to play an important role in influencing the degree of top-down control exerted on a given trial. They include both short-term traces between successive trials as well as intermediate memory traces that develop over task performance (Chiu and Egner, 2019).
• “Decision-making” refers to the process of choosing one of a set of alternatives to produce a beneficial outcome. Choices are made based on expectation of rewards developed through previous experience. Current models of decision-making mostly rely on reinforcement learning. Recent computational studies suggest that different types of memory traces may work together to optimize reinforcement learning, including those resulting from slow incremental implicit learning over trials as well as episodic traces uniquely linked to individual experiences (Botvinick et al., 2019).
• “Decision-making” is closely related to selective attention, which can be involved in prioritizing information for guiding choice behavior. Expectations play an essential role in both sets of processes. However, interestingly, they each stress a different consequence of prior knowledge. In decision-making, predictions based on priors are mainly used to attenuate the processing of what can be anticipated (Friston, 2010). In selective attention, predictions are mainly used to enhance the processing of anticipated task-relevant information. These two phenomena nicely illustrate the flexibility with which memory-related traces can be used to guide adaptive performance. The specific consequence of prior knowledge will be heavily dependent on the purpose of the task (Nobre, 2018).
• “Episodic future thinking” involves drawing on previous experiences to imagine oneself in future situations (Atance and O’Neill, 2001, Schacter et al., 2008). The construct is useful in different types of situations, such as navigation, planning to implement intentions, understanding others’ mental states, and simulating future events. Neuropsychological, developmental, and brain imaging studies have revealed substantial overlap between the neural system supporting episodic future thinking and episodic recollection, suggesting that the LTM traces available for recollection can also be used prospectively and flexibly to build novel plausible scenarios and run simulations (Schacter et al., 2008). Episodic future thinking differs from our construct of premembering in being a specifically deliberative process based on LTM traces available to awareness to inform behavior in the future. Premembering is a broader construct, considering the influences of memories of different types and timescales on ongoing or imminent behavior.

Selective attention (hereafter called “attention”) refers to the set of functions that prioritize and select information to guide adaptive behavior (Nobre, 2018). These functions modulate incoming sensory signals and influence their processing at multiple stages to inform awareness, decisions, actions, and subsequent memories. It has long been appreciated that short-term memory, or “working memory” (WM), plays an important role in forming attentional templates (e.g., Desimone and Duncan, 1995). Here we suggest that WM is part of a much larger family of heterogenous attention-guiding memory traces that span multiple timescales.

The premembering perspective has important implications for understanding the content and formatting of memory. Rather than slavishly storing and using veridical traces, the brain flexibly selects and even distorts memory content to enhance its utility in guiding attention. Moreover, task-relevant informational content may also be stored in a neural format that is optimized for the anticipated utilization of the memory to guide performance.

Multiple Timescales of Memories Guide Attention

The essence of memory is the traces left by passing experience. These range from transient perturbations to engrams that last a lifetime, capturing modality-specific fragments to relational and integrated wholes and supporting unconscious states to recollective phenomena.

Such traces can provide essential informational content required for prospectively prioritizing and selecting what is important. Working together, memory content and attention functions shape how the brain transforms incoming signals to guide perception, choice, action, and the formation of new memories to serve adaptive behavior in the future (Figure 1).

It can be argued that these prospective properties of memory are what define its fundamental ecological purpose: to collect relevant aspects of experience to anticipate future demands and guide behavior.

Short-Term Traces
Drawing a clear line defining when present becomes past may be impossible. Nevertheless, it is clear that past traces affect perception even from their earliest moments. The very stitching of visual perception across eye movements into an apparent cohesive flow may rely on short-term memories bridging the anchoring and landing fixation contents (Irwin and Gordon, 1998).

Furthermore, transient salient visual stimuli intrinsically capture attention and leave a brief excitatory trail, temporarily enhancing processing of stimuli that follow in their immediate wake (Posner, 1980; Figure 2A).

At slightly longer timeframes, WM provides a limited set of more durable traces that are independent of continuous sensory stimulation and resistant to interference and that act to guide adaptive behavior (Baddeley, 2003). The fundamental role WM plays in guiding attention is widely recognized and has been studied extensively (Desimone and Duncan, 1995). Information in WM has been considered the major source of top-down proactive attention.

Even before the target stimuli appear, these memory traces influence the pattern of brain activity in a proactive fashion to facilitate the processing of signals associated with likely relevant items (Chelazzi et al., 1993, Kastner et al., 1999, Stokes et al., 2009). These top-down anticipatory states based on WM templates are often associated with willful, volitional orienting of attention (Helmholtz, 1867, Posner, 1980). However, it is important to note that WM content can also influence incoming information processing in an involuntary fashion (Soto et al., 2008; see below).

Moreover, beyond the scope of classic WM, the relation between successive items can also influence information processing. For example, when searching for an odd-one-out target, its identification is facilitated when the current target shares a defining feature value (Maljkovic and Nakayama, 1994) or location (Maljkovic and Nakayama, 1996) with the target in the preceding trial.

Traces from preceding stimuli also alter the quality of perception. Reports of the spatial frequency and orientation of a visual item are systematically distorted by that of preceding items (e.g., Fischer and Whitney, 2014).

Short-term memory traces can be used to enhance or suppress information processing, depending on the situation. For example, after initial facilitation of stimuli in the wake of non-informative salient stimuli, decrements in performance occur at the same location; the “inhibition of return” has been proposed to encourage sampling and exploration (Posner and Cohen, 1984).

Pre-exposure to irrelevant or distracting stimuli can facilitate performance in visual search tasks by helping overcome distractor interference (Olivers et al., 2006). However, when the current target shares the identity or features of a distractor in the previous trial, performance decrements occur (negative priming; Fanini et al., 2006, Tipper, 1985). Reward associations can further magnify both facilitatory and inhibitory effects between successive stimuli (Della Libera and Chelazzi, 2006, Kristjánsson et al., 2010).

Intermediate Traces
Although most research has focused on the role of short-term traces in anticipatory attention, the memory traces that guide our perception stretch further back in time. Most of the knowledge that guides the interpretation of sensory signals and sets up expectations about the occurrence and context of relevant events is stored in our long-term memory (LTM; Nobre and Mesulam, 2014).

However, somewhere between short-term traces that guide performance between successive events and remote long-term memories that guide performance between distant episodes lives a rich variety of memory traces reflecting regularities learned and utilized within a task setting.

An elegant study by Carlisle et al. (2011) showed the gradual transition from WM to a longer-term trace when search targets were held constant for a number of trials (Figure 2B). For early trials, they showed a robust event-related potential associated with the location and number of items in the WM template (contralateral delay activity [CDA]; Vogel and Machizawa, 2004). As trials progressed, the CDA progressively diminished, implying a gradual handover to a longer-term, intermediate memory store.

Increasing numbers of experimental approaches are beginning to explore how different types of intermediate traces guide attention within task settings. “Probability cueing” refers to performance benefits resulting from the higher prevalence of a task-relevant item within a given location (Shaw and Shaw, 1977).

The frequent appearance of a target at a given location facilitates its identification, whereas the frequent appearance of an irrelevant stimulus at a location attenuates its distracting effect (Goschy et al., 2014, Wang and Theeuwes, 2018, Ferrante et al., 2018, Noonan et al., 2016).

Memory traces linked to probability cueing are long lasting and largely acquired implicitly (Jiang 2018). A related but different phenomenon is “selection history” (e.g., Kyllingsbaek et al., 2001), which refers to facilitation in identifying stimuli previously selected as targets independent of their probability.

“Reward history,” referring to the likelihood or value of a reward typically associated with given target stimuli within a task, has strong and persistent effects on performance (Anderson et al., 2011, Chelazzi et al., 2014, Krebs et al., 2011, Raymond and O’Brien, 2009). Reward manipulations often come bundled with manipulations of other sources of attention modulation, such as probability cueing or selection history.

In complex search environments, reward history combines with other factors to guide effective performance (e.g., Navalpakkam et al., 2010). In tasks that carefully control for other sources of influence, stimuli with features that have high reward associations act as potent distractors, even when they are otherwise inconspicuous and task irrelevant (Failing and Theeuwes, 2015, Hickey and Peelen, 2015).

Learning and utilization of more complex associations also occur within task settings. “Contextual cueing” shows that pick-up of repeated spatial configurations of stimuli within visual search arrays facilitates target identification (Chun and Jiang, 1998). This type of memory guides search behavior, resulting in fewer fixations and faster search times (Harris and Remington, 2017).

Neurophysiological markers of target detection are also enhanced (Johnson et al., 2007, MacLean and Giesbrecht, 2015, Schankin and Schubö, 2009). Learning of statistical regularities over time similarly leads to improvements in behavioral performance (Schapiro et al., 2016).

Interestingly, the presence of statistical regularities within stimulus streams spontaneously captures attention, leading to facilitation of target identification in structured relative to unstructured streams (Zhao et al., 2013). Both contextual cueing and statistical learning are thought to involve hippocampus-related memory systems, even though they are learned largely implicitly (Chun and Phelps, 1999, Schapiro et al., 2016).

A recent neurophysiological study of sequence learning in a serial response task has also revealed proactive anticipation of upcoming stimuli and associated responses based on learned spatiotemporal expectations (Heideman et al., 2018).

Long-Term Traces
Although the role of LTM in guiding attention has been less well explored than that of WM, its fundamental role in defining perception, choosing actions, imagining, and forming new memories has long been recognized (Bartlett, 1932, Helmholtz, 1867, Moscovitch et al., 2016; Box 1). We propose that the role memory plays in guiding attention may contribute to its fundamental influence over our mental experience and behavior.

In everyday situations, the bank of remote memory traces from previous settings and episodes arguably provides the richest source of information for guiding attention and perception. When waiting to meet a friend on a busy street, we rely on LTM to search them out based on their appearance and likely route.

LTM provides knowledge about whether and how relevant events occur in particular contexts and about their likely reward outcome values. Thus, they afford rich content for flexible, proactive, and dynamic biases to guide the prioritization and selection of relevant information (Aly and Turk-Browne, 2017, Bar, 2004, Nobre and Mesulam, 2014).

An increasing variety of experimental approaches is being used to explore how LTM content is used proactively to guide adaptive behavior. Although differing in detail, the approaches share the notion that the brain uses LTM information constantly, proactively, and predictively.

In visual search tasks using simple stimulus arrays, semantic knowledge of features commonly associated with objects facilitates target identification, overcoming usual costs associated with feature binding (Rappaport et al., 2013). On the other hand, distractors with associative links to the target disrupt the visual search by capturing attention (Moores et al., 2003). Furthermore, newly learned associations continue to affect performance on subsequent tasks, even when completely irrelevant (Fan and Turk-Browne, 2016).

In tasks using complex scenes or environments, “contextual priming” studies show that previous experiences within particular contexts guide the identification and interpretation of objects within the same contexts (Bar, 2004).

When searching for a specific item within a scene, LTM works alongside lower-level statistics of visual features to guide performance (Navalpakkam and Itti, 2006). Multiple types of LTM are at play (Wolfe et al., 2011, Le-Hoa Võ and Wolfe, 2015). Search performance benefits from schematic probabilistic knowledge of encountering the item within a particular context (Võ and Henderson, 2009) as well as from its likely location relative to the layout of the scene (Torralba et al., 2006) and to other objects within a scene (Bar, 2004).

In addition, specific knowledge about the location of an item within a particular scene also contributes—episodic guidance (Brockmole and Henderson, 2006). Interestingly, incidental memories formed through active search and selection of items within a scene lead to stronger performance benefits and subsequent memories than familiarity or explicit attempts to memorize items within scenes (Draschkow et al., 2014), suggesting that effective memories fall out of natural behavioral interactions.

Memory-guided attention tasks show how learning of object locations through repeated visual search in complex scenes or environments facilitates performance in subsequent tasks.

Performance measures reveal significant improvements in detecting and discriminating target items as a result of previously learned specific contextual association between the particular item and a scene or three-dimensional environment (Becker and Rasmussen, 2008, Draschkow and Võ, 2016, Kit et al., 2014, Summerfield et al., 2006).

“Memory-guided orienting” tasks provide a good platform to investigate anticipatory biases based on long-term memories (Summerfield et al., 2006; Figure 2C). Participants form object-scene associations to a high and stable level during a learning session; the consequences of these memory traces are then tested in a separate attention-orienting session.

The separation helps disentangle effects of early learning from those of subsequent memory utilization. In our own research, we have used learned scenes presented in isolation as memory cues, followed by the appearance of a target stimulus to be identified or discriminated.

The critical behavioral comparison is between performance depending on whether the target location is correctly predicted by previous memory experience (valid memory cue) versus incorrectly predicted (invalid memory cue) or unpredicted (neutral memory cue). Strong and reliable effects occur for both perceptual sensitivity (Patai et al., 2012) and response times (Summerfield et al., 2006).

By presenting memory cues and targets individually, it is possible to measure memory-related brain activity that biases excitability according to anticipation of the target location. fMRI reveals that memory cues engage the hippocampus as well as the dorsal frontoparietal network associated with the control of spatial attention (Summerfield et al., 2006, Stokes et al., 2012).

In addition, activity levels in visual cortical regions become spatially biased in anticipation of the target location (Stokes et al., 2012; Figure 2C). Electrophysiological recordings also reveal memory-related spatial anticipatory biases in the form of lateralization of alpha-band power (Stokes et al., 2012, Summerfield et al., 2011). Anticipatory biases in the cue-target period are followed by modulation of event-related potentials related to visual processing (Summerfield et al., 2011) and spatial selection (Patai et al., 2012, Doallo et al., 2013) of target items. Reward associations further potentiate the effects of memory-based orienting. Even a single association with a modest monetary gain or avoidance of loss during the last learning block leads to performance improvements in the subsequent attention-orienting session, even though reward is irrelevant to the orienting task. Target stimuli during the orienting task elicit larger visual P1 potentials when previously associated with reward or punishment avoidance (Doallo et al., 2013, Suárez-Suárez et al., 2019).

Rosen et al. (2014) have followed a similar approach involving learning and utilizing spatial contextual associations in a change-detection task. By comparing memory-guided and visually guided attention conditions, they have highlighted a set of cortical regions and differences in hemispheric lateralization associated with LTM-based orienting (Rosen et al., 2016).

Thus, findings with memory-guided orienting tasks clearly illustrate the ability of the brain to use LTM content proactively to guide prioritization and suggest the involvement of limbic, memory-related circuits. In monkeys, recent studies recording activity from neurons in the hippocampus and entorhinal cortex during free viewing provide converging evidence that medial temporal areas, traditionally associated with memory, are closely related to attention.

Neurons in the entorhinal cortex show strong visuospatial coding properties, signaling gaze location using multiple frames of reference (Meister and Buffalo, 2018). Entorhinal neurons further show a grid cell-like arrangement of viewed spatial locations, using only covert attention in the absence of any eye or other physical movement (Wilming et al., 2018).

In addition to grid cells, saccade direction cells are also present, with largely independent populations encoding the direction of previous versus future saccades (Killian and Buffalo, 2018). Together, such findings point to mechanisms that could support spatial attention based on LTM. Recordings in the hippocampus showed that slow-wave ripples, thought to promote plasticity, occurred during a search for object changes within scenes and that their probability increased with scene repetition and near remembered targets (Leonard and Hoffman, 2017). Such a mechanism would be a good candidate for supporting memory-guided attention during search.

Dynamic Prospective Memories
In addition to carrying information about the location and identifying attributes of anticipated events, memory traces also carry information about their timing. Memory-based temporal expectations can therefore enable the brain to prepare for events in a dynamic, temporally structured, and efficient way (Nobre and van Ede, 2018).

Temporal expectations rely on learning temporal regularities that occur over multiple timescales. Neural markers of proactive temporal anticipation have been observed when the timing of stimuli follows a regular temporal rhythm (Cravo et al., 2013), probabilistic conditional probabilities (Cravo et al., 2011), or sequences (Heideman et al., 2018).

Across episodes, long-term memories can also guide temporally structured anticipatory attention. Using a temporal variant of the memory-guided orienting task, Cravo et al., (2017) showed improved perceptual sensitivity and response times to detect and discriminate visual targets occurring at the learned temporal interval (valid memory cue) relative to the other interval (invalid memory cue). Event-related potentials elicited by the memory cue revealed clear modulation of target anticipation according to temporal expectation. Behavioral and neural markers of validity effects correlated with one another and with the quality of learning of the temporal association.

Plurality across Types and Timescales
Recognizing that memory traces of different types and timescales proactively guide perception invites us to reconsider dominant concepts and dichotomies in the attention literature.

The simple separation between “bottom-up” sources of prioritization based on physical salience and “top-down” sources based on current goals is clearly insufficient. To explore the natural kinds of memory-based attention, it is important to separate two fundamental factors: their volitional character and their source.

Subjectively, attention can be voluntary or involuntary, and its source can be external, based on physical salience (exogenous), or internal, based on brain states linked to current goals or previous experience (endogenous).

Often these two factors are conflated, with exogenous sources assumed to guide involuntary attention and endogenous sources assumed to guide voluntary attention. Instead, they should be viewed as theoretically separable. Although physically salient stimuli capture attention involuntarily, endogenous sources can guide attention both voluntarily and involuntarily.

In addition to their use in goal-based voluntary attention, both short-term as well as LTM sources can also have automatic, involuntary consequences for perception. For example, short-term memory carrying goal-related information to perform one task also inadvertently influences performance in another task, sometimes causing significant interference (Soto et al., 2008).

Memory traces acquired over a task setting—reflected in probability cuing, selection history, reward history, contextual cueing, or sequence learning—are often acquired incidentally and utilized involuntarily and without explicit knowledge (Chun, 2000, Jiang, 2018).

Remote, long-term memories for the presence and location of items within rich, complex scenes are more often accompanied by conscious recollection (Brockmole and Henderson, 2006, Jiang, 2018). Although it is likely that these associations are used to direct attention voluntarily, they may also orient attention involuntarily.

Under most conditions, benefits in LTM-based orienting correlate with measures of explicit memory (Cravo et al., 2017, Salvato et al., 2016), suggesting a viable source for voluntary attention. However, dissociations have also been noted. Although explicit retrieval of object-scene memories is significantly compromised in older individuals compared with younger counterparts, the benefits of memory-based attention are spared (Salvato et al., 2016). The relationship and the degree of interaction between these two modes of memory-based attention are important directions for future research (Nobre and Mesulam, 2014).

In recognizing the role that memory associations within task settings play in attention, some contemporary models of attention have proposed adding a third, “history” source to the traditional bottom-up and top-down sources in attention (e.g., Awh et al., 2012).

The inclusion is an important first step toward acknowledging the fundamental role played by memory, but there are also inherent limitations with treating memory as a single and separate additional source of attention control. On one hand, it is not possible to extricate memory from the two traditional sources of attention.

Brief memory traces contribute to effects of bottom-up attention, prioritizing stimuli that trail physical singletons, and WM traces are essential for goal-based top-down attention. On the other hand, amalgamating memory sources into one common history factor works against evidence suggesting the plurality of memory mechanisms at play.

The internal, memory-based sources of attention are many. Memories of various durations and types contribute, from traces that stitch the continuity of perception to those that guide the recognition of a long-lost friend. Most likely, multiple memory systems and mechanisms are involved.

For example, both contextual cueing and memory-based orienting involve LTM for the location of a target object within a given configuration in an array or scene, and in both cases, the medial-temporal areas have been implicated. However, brain imaging shows increased hippocampal activity during memory-guided orienting in scenes (Hannula and Ranganath, 2009, Summerfield et al., 2006) and decreased activity by repeated versus novel arrays in contextual cueing tasks (Greene et al., 2007, Giesbrecht et al., 2013).

Electrophysiological recordings also show different patterns of modulation of potentials associated with target selection (N2pc), with attenuation by memory-guided orienting (Doallo et al., 2013, Patai et al., 2012) and enhancement by contextual cueing (Kasper et al., 2015, Schankin and Schubö, 2009). Striking differences occur even when comparing different types of memory traces within a single task. Goldfarb et al., (2016) contrasted the effects of contextual cueing (associations between specific array configurations and target locations) versus probability cueing (probabilistic learning of most likely target locations and response choices) in guiding visual search in a common task. Although contextual cueing modulated activity in the hippocampus, probability cueing modulated activity in the striatum.

The effects were selectively predictive of attention benefits across successive trials and correlated with performance measures. Numerous functional dissociations have also been observed in behavioral effects of attention based on different types of memory within a task context.

Although often entangled in natural contexts, probability cueing effects can be separated from single-trial priming effects (Jiang 2018), from selection history (Ferrante et al., 2018, Wang and Theeuwes, 2018), and from goal-based, short-term-memory cueing (Goschy et al., 2014, Jiang, 2018). Likewise, reward history can be separated from selection history (Anderson et al., 2017).

Thus, the literature suggests a rich diversity of mnemonic influences on attention, with at least some functional dissociations. There is much empirical work ahead to reveal the natural kinds and neural mechanisms of memory-based attention to derive its principles of organization and develop useful theoretical models.

Traces of different kinds and durations could act separately and largely independently; they could compete, they could become integrated within a unified predictive model (e.g., in a Bayesian framework; Friston, 2010), they could converge and operate through a common priority map to modulate sensory processing (Bisley and Goldberg, 2010), or they could combine in mutually supportive ways to enhance the quality and flexibility of biasing signals (Botvinick et al., 2019). At this stage, it is premature to clump together memories into a single source of attentional control, and doing so could discourage or misguide much needed investigation in this important area.

Prospective and Adaptive Memories
Premembering has implications for understanding the nature of memory itself. Consonant with a role in serving future behavior by grounding attentional selection, the very nature of memory traces is forward looking. What is encoded, maintained, and selected for retrieval is strongly influenced by what is likely to be important for future behavior.

Future-Relevant Content of Short-Term Memory
At the shortest timescales, the contents of memories maintained across eye fixations are strongly influenced by the location of the upcoming saccade. Even before a saccade is made, the information corresponding to its upcoming landing zone is remembered better than information at other, equidistant locations (Irwin, 1992).

Within WM research, findings increasingly highlight the prospective qualities of these traces. The contents of WM are strongly influenced by current task goals. They are selective and can even be distorted to facilitate performance based on anticipated task demands.

The selectivity of WM was elegantly illustrated by a neurophysiological study in monkeys showing that neuronal activity in the prefrontal cortex (PFC) was largely dominated by the single relevant item of a multi-item array during a selective delayed-match-to-sample task (Rainer et al., 1998).

Later, responses in prefrontal neurons were shown to be most sensitive to stimulus dimensions that were relevant for making categorical discriminations imposed by the task (Freedman et al., 2001). Although prefrontal neurons differentiated these discriminant features, they were less sensitive to other equally available, non-discriminant physical features until these became task relevant. Similar effects were reported for the parietal cortex when monkeys performed categorical discrimination tasks based on arbitrary boundaries along continuous feature dimensions (Freedman and Assad, 2006).

Selectivity in WM is also well documented in the human brain. The CDA marker of WM derived from electroencephalograms (EEGs) shows that encoding can be spatially and item selective (e.g., Vogel and Machizawa, 2004).

In addition, selectivity can also occur for relevant features within memory items. Using multivariate pattern analysis (MVPA) to derive population-response properties from fMRI data, Serences et al. (2009) found that decoding during a WM delay depended on the expected demands during recall. Participants performed a delayed matching task on colored gratings and were cued that either color or orientation was relevant to the discrimination. Patterns of activity in the visual cortex selectively maintained the task-relevant feature, consistent with a prospective memory code for guiding future behavior (Figure 3A).

Studies in human participants have also demonstrated that selective contents in WM can be updated flexibly as predictions change about the items or features that will be relevant for task performance (Griffin and Nobre, 2003, Landman et al., 2003). Cues presented during the delay that retrospectively inform participants about the item most likely to be probed (retro-cues) result in significantly better memory performance.

fMRI studies show selective maintenance of task-relevant features after informative retro-cues that predict or instruct which stimulus or attributes will be probed (Kuo et al., 2014, LaRocque et al., 2017, Lepsien and Nobre, 2007, Sligte et al., 2009). Human neurophysiological studies also show flexible updating of selective content in WM during maintenance with retro-cues that indicate which item in memory is most likely to be probed (Kuo et al., 2012, Poch et al., 2014, Wallis et al., 2015; Figure 3B).

Such updating can also occur in the absence of external cues, when the passage of time is associated with the likely item to be probed, illustrating the flexibility of updating selective WM contents and the role of temporal expectations (van Ede et al., 2017).

From a functional perspective, it is not always optimal simply to maintain a veridical representation of previous input. When making some types of fine-grained delayed discriminations, it can be more advantageous to focus on neural information tuned away from the behaviorally relevant feature to maximize separability from distractor items (Navalpakkam and Itti, 2007).

In such tasks, behavioral measures confirm that WM templates are adaptive distortions of the experienced information (Navalpakkam and Itti, 2007, Geng et al., 2017). Application of multivariate methods to derive tuning functions of fMRI voxels confirmed that WM templates in such tasks reflect distortions of the perceived stimulus, thus being adaptive rather than veridical (Scolari et al., 2012).

Interestingly, in addition to being shifted, the relevant feature information maintained in the target template can also be sharpened to increase separability from distractor stimuli in the task (Geng et al., 2017, Yu and Geng, 2019; Figure 3C).

These studies clearly reveal the prospective face of WM: content is optimized to the expected demands of future processing. In some cases, the prospective WM representation triggered by a stimulus bears no physical resemblance to it. Rainer et al. (1999) trained monkeys on a paired-associate task that systematically manipulated the relationship between memory items and expected probe stimuli.

This revealed a subset of neurons coding for the expected stimulus during the delay period even though it was not actually presented on that trial. This also demonstrates the interaction between WM and LTM.

WM representations also go beyond the experienced stimulus to construct motor representations when specific actions can be anticipated. Most WM tasks tend to probe the perceptual content of the memoranda and typically isolate perceptual qualities of stimuli from particular responses.

In everyday life, however, the perceptual content of WM often also carries motor affordances, such as when reaching out and grasping the cup of coffee outside your field of view or repeating a telephone number to a friend. In a recent visual WM task that linked the reporting of particular stimulus orientations to a specific hand, it became clear that motor representations are derived from visually encoded stimuli and that motor information is in a similar state of readiness as visual content when participants are probed (van Ede et al., 2019; Figure 3D).

Future-Relevant Content of LTM
The prospective attributes of LTM are much more difficult to study. Except under highly constrained experimental conditions, it is difficult to determine what aspects of experience will prove useful to guide future behavior at distant time points. Nevertheless, the prospective nature of LTM can be gleaned from a few observations. The content of LTM is selective. We better remember items that were relevant during the encoding context (Aly and Turk-Browne, 2017).

We retain aspects of stimuli that are useful for adaptive behavior. Although our memories of the details in the appearance of a penny are shockingly poor (Nickerson and Adams, 1979), we readily remember the attributes that distinguish a penny from other coins. Similarly, our well-established difficulty with individuating and remembering faces from other cultures and races may, in part, reflect our focus on discriminant features that are informative within our given cultural context (Rhodes et al., 1989).

---

Source:
Goethe University Frankfurt