Seminar: 3/14 - Thomas Naselaris
11:00 to 12:30 PM at:
Helen Wills Neuroscience Institute, UC Berkeley
"Picturing the mind's eye: Decoding the structure and meaning of natural images from human brain activity."
Thomas Naselaris, Ryan Prenger, Kendrick Kay, Michael Oliver, Jack Gallant
When you look at an image, the information transmitted through your eyes is converted into neural activity. The relationship between an image and evoked neural activity can be captured with an encoding model. An encoding model can be thought of as a box that takes arbitrary images as inputs, and outputs a prediction of the evoked brain activity. One of the main challenges of cognitive and systems neuroscience is to construct encoding models for all stages of visual processing. Accurate encoding models might also be useful for solving another interesting problem, brain reading. To perform brain reading, an encoding model would be run in reverse, so that measured neural activity enters as input, and a prediction of the stimulus that evoked it emerges as the output. A general brain reading device would be extremely interesting and useful. For instance, measurements of brain activity obtained during sleep could be used to produce a pictorial representation of dreams. In this talk, we describe a new algorithm that takes us several steps toward this goal. In brief, we use fMRI to measure brain activity in the visual cortex of human subjects while they view thousands of natural images. We then construct encoding models that described the stimulus-response mapping function for single voxels, and we reverse these encoding models to perform several different kinds of brain-reading: in image identification the brain activity evoked by one image is used to pick that specific image out of a stack of potentially millions of other images; in image reconstruction brain activity is used to reconstruct an actual picture of the image that evoked it. We will provide examples of these results, and discuss the technical and ethical implications of this work.