Scientists have taken a striking step toward decoding visual information from the brain by reconstructing short movies using neural activity recorded from mice.
Researchers at University College London (UCL) used signals from individual neurons in the animals’ primary visual cortex to recreate 10-second video clips. The work, published in eLife, offers researchers a new way to examine how the brain converts visual information into an internal representation of the world.
The achievement is not the equivalent of recording a perfect video directly from a mouse’s thoughts. Instead, scientists used patterns of neural activity and a computational model to infer which visual input could have produced those signals.
That distinction is crucial because the researchers are ultimately interested in something deeper: understanding how what exists in the outside world differs from what the brain actually represents.
How scientists turned brain signals into video
The team worked with publicly available data from the 2023 Sensorium Competition.
The dataset contained movies shown to mice, along with recordings of activity from neurons in the animals’ primary visual cortex, known as V1. Researchers also had information about pupil diameter, pupil position and the animals’ running behavior. Neural activity from about 8,000 neurons was available for each of 10 mice.
The brain activity was recorded using two-photon calcium imaging, a technique that allows scientists to monitor activity in individual brain cells by tracking changes in calcium-related fluorescence.
The researchers then used DwiseNeuro, a dynamic neural encoding model that had previously performed best in the Sensorium 2023 competition. The model normally works in the forward direction, predicting how neurons should respond when a mouse watches a particular video.
The UCL team effectively reversed that process.
From a blank screen to a reconstructed movie
The researchers began with a blank video and asked the model to estimate how the recorded neurons would have responded if the mouse had been looking at that blank screen.
They compared that predicted activity with the neurons’ actual activity while the mouse watched a movie.
An optimization process then repeatedly changed the pixels in the blank video. The objective was to make the model’s predicted neural responses increasingly similar to the activity that had actually been recorded from the mouse.
This process, based on gradient descent, gradually transformed the blank sequence into a reconstruction that resembled the visual stimulus presented to the animal.
In total, the researchers reconstructed ten natural 10-second movies from data involving five mice.
The model was tested on an unseen movie
One of the most important aspects of the experiment was testing whether the system could reconstruct something it had not simply memorized.
The researchers applied their method to neural activity recorded while a mouse watched a video that was not included in the model’s training material.
The system was able to reconstruct a 10-second clip that corresponded to the unseen visual stimulus. The researchers also found that reconstruction accuracy improved as more individual neurons were included in the analysis.
That finding suggests that capturing a sufficiently large population of neurons is important for recovering detailed information about the visual scene.
How accurate were the reconstructions?
The researchers evaluated their results using spatiotemporal, or pixel-by-pixel, correlation between the original and reconstructed videos.
The study reports substantial improvement in reconstruction quality compared with earlier approaches, although the reconstructed videos are still far from being exact copies of the original visual input. The researchers say image resolution and the amount of the visual field that can be reconstructed remain limitations.
The goal, therefore, is not simply to produce a visually convincing movie. It is to build a quantitative tool that allows scientists to compare an external stimulus with its representation inside the brain.
Why the study could change how scientists study vision
For decades, neuroscience research has shown that the brain does not simply function like a camera.
Light entering the eye is transformed into neural signals, which are then processed through increasingly complex networks. The brain extracts patterns, emphasizes particular features and integrates sensory information with the animal’s behavior and internal state.
The UCL researchers want to use their reconstruction method to study precisely where those transformations occur.
By comparing the original movie with the neural reconstruction, scientists could potentially identify which aspects of a scene are preserved, distorted, amplified or discarded by visual processing.
That could provide a new way to investigate phenomena such as predictive coding, selective attention and perceptual learning.
A mouse does not “see” the reconstructed video
There is an important misconception to avoid when describing this research.
The experiment does not establish exactly what a mouse consciously experiences. Scientists reconstructed visual information represented by activity in the primary visual cortex. That neural representation is only one stage of the much larger visual-processing system.
Nor does the study show that researchers can currently decode arbitrary memories, dreams or private thoughts from a mouse brain.
The reconstruction works because the researchers have a controlled visual stimulus, detailed recordings from thousands of neurons and a trained model capable of linking visual inputs with patterns of neural activity.
What comes next?
The researchers say the technique could eventually be used to investigate how the brain’s representation of the world changes across different circumstances.
More comprehensive neural recordings could improve reconstruction quality, while experiments involving different visual scenes and behavioral conditions could help reveal how perception is altered by attention, learning or other internal processes.
The study also demonstrates the value of combining large-scale neural recordings with increasingly sophisticated computational models. As scientists gain access to recordings from more neurons, the boundary between observing brain activity and inferring the information encoded within it may continue to shift.
For now, the reconstructed movies offer a glimpse into a remarkable scientific possibility: rather than asking only what an animal sees, researchers can begin comparing the world presented to its eyes with the version of that world encoded by its brain.
