Artificial brains need sleep to maintain their stability

1
246

No one can say whether androids will dream of electric sheep, but they will almost certainly need periods of rest that offer benefits similar to those that sleep provides to living brains, according to new research from Los Alamos National Laboratory.

“We study spiking neural networks, which are systems that learn much as living brains do,” said Los Alamos National Laboratory computer scientist Yijing Watkins.

“We were fascinated by the prospect of training a neuromorphic processor in a manner analogous to how humans and other biological systems learn from their environment during childhood development.”

Watkins and her research team found that the network simulations became unstable after continuous periods of unsupervised learning. When they exposed the networks to states that are analogous to the waves that living brains experience during sleep, stability was restored. “It was as though we were giving the neural networks the equivalent of a good night’s rest,” said Watkins.

The discovery came about as the research team worked to develop neural networks that closely approximate how humans and other biological systems learn to see. The group initially struggled with stabilizing simulated neural networks undergoing unsupervised dictionary training, which involves classifying objects without having prior examples to compare them to.

“The issue of how to keep learning systems from becoming unstable really only arises when attempting to utilize biologically realistic, spiking neuromorphic processors or when trying to understand biology itself,” said Los Alamos computer scientist and study coauthor Garrett Kenyon.

“The vast majority of machine learning, deep learning, and AI researchers never encounter this issue because in the very artificial systems they study they have the luxury of performing global mathematical operations that have the effect of regulating the overall dynamical gain of the system.”

The researchers characterize the decision to expose the networks to an artificial analog of sleep as nearly a last ditch effort to stabilize them. They experimented with various types of noise, roughly comparable to the static you might encounter between stations while tuning a radio.

The best results came when they used waves of so-called Gaussian noise, which includes a wide range of frequencies and amplitudes. They hypothesize that the noise mimics the input received by biological neurons during slow-wave sleep.

The results suggest that slow-wave sleep may act, in part, to ensure that cortical neurons maintain their stability and do not hallucinate.

The groups’ next goal is to implement their algorithm on Intel’s Loihi neuromorphic chip. They hope allowing Loihi to sleep from time to time will enable it to stably process information from a silicon retina camera in real time.

If the findings confirm the need for sleep in artificial brains, we can probably expect the same to be true of androids and other intelligent machines that may come about in the future.

Watkins will be presenting the research at the Women in Computer Vision Workshop on June 14 in Seattle.


Building a vast digital simulation of the brain could transform neuroscience and medicine and reveal new ways of making more powerful computers” (Markram et al., 2011). The human brain is by far the most computationally complex, efficient, and robust computing system operating under low-power and small-size constraints.

It utilizes over 100 billion neurons and 100 trillion synapses for achieving these specifications. Even the existing supercomputing platforms are unable to demonstrate full cortex simulation in real-time with the complex detailed neuron models.

For example, for mouse-scale (2.5 × 106 neurons) cortical simulations, a personal computer uses 40,000 times more power but runs 9,000 times slower than a mouse brain (Eliasmith et al., 2012).

The simulation of a human-scale cortical model (2 × 1010 neurons), which is the goal of the Human Brain Project, is projected to require an exascale supercomputer (1018 flops) and as much power as a quarter-million households (0.5 GW).

The electronics industry is seeking solutions that will enable computers to handle the enormous increase in data processing requirements. Neuromorphic computing is an alternative solution that is inspired by the computational capabilities of the brain.

The observation that the brain operates on analog principles of the physics of neural computation that are fundamentally different from digital principles in traditional computing has initiated investigations in the field of neuromorphic engineering (NE) (Mead, 1989a).

Silicon neurons are hybrid analog/digital very-large-scale integrated (VLSI) circuits that emulate the electrophysiological behavior of real neurons and synapses. Neural networks using silicon neurons can be emulated directly in hardware rather than being limited to simulations on a general-purpose computer.

Such hardware emulations are much more energy efficient than computer simulations, and thus suitable for real-time, large-scale neural emulations. The hardware emulations operate in real-time, and the speed of the network can be independent of the number of neurons or their coupling.

There has been growing interest in neuromorphic processors to perform real-time pattern recognition tasks, such as object recognition and classification, owing to the low energy and silicon area requirements of these systems (Thakur et al., 2017; Wang et al., 2017).

These large systems will find application in the next generation of technologies including autonomous cars, drones, and brain-machine interfaces. The neuromorphic chip market is expected to grow exponentially owing to an increasing demand for artificial intelligence and machine learning systems and the need for better-performing ICs and new ways of computation as Moore’s law is pushed to its limit (MarketsandMarkets, 2017).

The biological brains of cognitively sophisticated species have evolved to organize their neural sensory information processing with computing machinery that are highly parallel and redundant, yielding great precision and efficiency in pattern recognition and association, despite operating with intrinsically sluggish, noisy, and unreliable individual neural and synaptic components.

Brain-inspired neuromorphic processors show great potential for building compact natural signal processing systems, pattern recognition engines, and real-time autonomous agents (Chicca et al., 2014; Merolla et al., 2014; Qiao et al., 2015).

Profiting from their massively parallel computing substrate (Qiao et al., 2015) and co-localized memory and computation features, these hardware devices have the potential to solve the von Neumann memory bottleneck problem (Indiveri and Liu, 2015) and to reduce power consumption by several orders of magnitude.

Compared to pure digital solutions, mixed-signal neuromorphic processors offer additional advantages in terms of lower silicon area usage, lower power consumption, reduced bandwidth requirements, and additional computational complexity.

Several neuromorphic systems are already being used commercially. For example, Synaptics Inc. develops touchpad and biometric technologies for portable devices, Foveon Inc. develops Complementary Metal Oxide-Semiconductor (CMOS) color imagers (Reiss, 2004), and Chronocam Inc. builds asynchronous time-based image sensors based on the work in Posch et al. (2011).

Another product, an artificial retina, is being used in the Logitech Marble trackball, which optically measures the rotation of a ball to move the cursor on a computer screen (Arreguit et al., 1996). The dynamic vision sensor (DVS) by iniLabs Ltd. is another successful neuromorphic product (Lichtsteiner et al., 2008). Table ​Table11 provides a detailed timeline, with major breakthroughs in the field of large-scale brain simulations and neuromorphic hardware.

Table 1

Timeline of neuromorphic simulation and hardware.

ReferencesContributions
Mead, 1989aInitiated the field of neuromorphic engineering
Mead, 1989bAdaptive Retina: among the first biologically inspired silicon retina chip
Mahowald and Douglas, 1991Silicon Neuron: neuron using subthreshold aVLSI circuitry
Prange and Klar, 1993BIONIC: an emulator with simulation of 16 neurons with 16 synapses
Yasunaga et al., 1990LSI composed on 576 digital neurons
Wolpert and Micheli-Tzanakou, 1996Modeling nerve networks based on the I&F model in silicon
Jahnke et al., 1996NESPINN: SIMD/dataflow architecture for a neuro-computer
Schoenauer et al., 1999MASPINN: a neuro-accelerator for spiking neural networks
Wolff et al., 1999ParSPIKE: a DSP accelerator simulating large spiking neural networks
Schoenauer et al., 2002NeuroPipe-Chip: a digital neuro-processor for spiking neural networks
Furber et al., 2006High-performance computing for systems of spiking neurons
Markram, 2006Blue Brain Project: large-scale simulation of the brain at cellular level
Boahen, 2006Neurogrid: emulating a million neurons in the cortex
Koickal et al., 2007aVLSI adaptive neuromorphic olfaction chip
Djurfeldt et al., 2008Brain-scale computer simulation of the neocortex on the IBM Blue Gene
Maguire et al., 2007BenNuey: platform comprises up to 18 M neurons and 18 M synapses
Izhikevich and Edelman, 2008Simulation of thalamocortical with 1011 neuron and 1015 synapses
Ananthanarayanan et al., 2009Cortical simulations with 109 neurons, 1013 synapses
Serrano-Gotarredona et al., 2009CAVIAR: a 45 k neuron 5 M synapse 12 G connects/s AER hardware
Schemmel et al., 20082010BrainScales: a wafer-scale neuromorphic hardware
Seo et al., 2011CMOS neuromorphic chip with 256 neurons and 64 K synapses
Cassidy et al., 2011EU SCANDLE: one-million-neuron, single FPGA neuromorphic system
Moore et al., 2012Bluehive project: simulation with 256 k neurons and 256 M synapses
Zamarreno-Ramos et al., 2013AER system with 64 processors, 262 k neurons, and 32 M synapses
Furber et al., 2014SpiNNaker: Digital neuromorphic chip with multicore System-on-Chip
Merolla et al., 2014TrueNorth: IBM introduces the TrueNorth “neurosynaptic chip”
Benjamin et al., 2014Neurogrid: A mixed-analog-digital large-scale neuromorphic simulator.
Park et al., 2014IFAT: neuromorphic processor with 65 k-neuron I&F array transceiver
Wang et al., 2014An FPGA framework simulating 1.5 million LIF neurons in real time
Qiao et al., 2015A spiking neuromorphic processor with 256 neurons and 128 K synapses
Park et al., 2017HiFAT-IFAT: reconfigurable large-scale neuromorphic systems
Cheung et al., 2016Neuromorphic processor capable of simulating 400 k neurons in real-time
Pani et al., 2017An FPGA platform with up to 1,440 Izhikevich neurons
Moradi et al., 2018DYNAP-SEL: neuromorphic mixed-signal processor with self-learning
Davies et al., 2018Loihi: Intel neuromorphic chip with “self-learning chip”
Wang and van Schaik, 2018; Wang et al., 2018DeepSouth: cortex simulator up to 2.6 billion LIF neurons

Provided by Los Alamos National Laboratory

1 COMMENT

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Questo sito usa Akismet per ridurre lo spam. Scopri come i tuoi dati vengono elaborati.