Professor Jack Gallant from the University of California, Berkeley, tells us how his team are building an atlas to the semantic system and revealing how our cerebral cortex turns language into meaning.
Read more about this research at: https://researchfeatures.com/2016/07/05/make-meaning-language/
Original publication:
https://dx.doi.org/10.1016%2Fj.neuron.2012.10.014
In this episode, we’re going to be talking about Professor Jack Gallant’s work on ‘Decoding Language in the Brain’.
Professor Jack Gallant from the University of California, Berkeley, tells us how his team are building an atlas to the semantic system and revealing how our cerebral cortex turns language into meaning.
The human brain is one of the most complex biological machines on Earth. Despite many previous attempts to understand its workings, we may still have a long way to go. However, Professor Jack Gallant’s team at the Gallant Lab, University of California, Berkeley, have taken a significant step forward into how the brain turns language into meaning. Developing new techniques in the process, the research team are attempting to map the semantic system, the regions of the brain that handle the complicated stream of data we call language
The semantic system is distributed across much of the cerebral cortex. It is vital to the lives of modern human’s, allowing us to communicate and understand our diverse thoughts, opinions and emotions. The term semantics is derived from the Ancient Greek word for ‘significant’, and studies the meaning we draw from incoming sensory information. The challenge for the researchers at the Gallant Lab is to understand the relationship between signifiers, such as words and phrases, and their denotation – what the words actually mean to us.
First of all, we’re going to talk about Building a giant atlas…
Until now, little has been discovered about the functional and anatomical organisation of the semantic system. In their recent study published in Nature, Alexander Huth, Jack Gallant and their colleagues set out to map the functional regions of the brain involved in semantic processing. “Our goal was to build a giant atlas that shows how one specific aspect of language is represented in the brain, in this case semantics, or the meanings of words,” said Jack Gallant. This difficult task involved recording functional MRI (fMRI) data of seven subjects’ cerebral cortex while they listened to natural narrative speech. As the subjects listened to autobiographical stories from The Moth Radio Hour, a popular US radio show, the researchers recorded exactly which parts of the brain were activated by different words. This information was then used to build mathematical models that predict brain activity in response to the words heard. These models were shown to predict responses in the cerebral cortex relatively well by playing new stories to the subjects and comparing the predicted activity against the actual fMRI observations.
However, to visualise these complex models and identify the centres of different types of information was the next challenge. The 985-dimensional semantic models were reduced to the 3 most important dimensions, using principal component analysis to preserve as much information as possible. The team then applied this information to a 3D model of the brain to generate the fascinating semantic map visualisations lTo compare these results across subjects, the team developed a new analysis technique with the equally carefully crafted acronym PrAGMATiC. This Probabilistic And Generative Model of Areas Tiling the Cortex approach found similar functional areas in different subjects, allowing for anatomical variability in each subject
The cerebral cortex is a sheet of neural tissue that wraps around the brain. Only a few millimeters thick, the cortex is highly folded so we can fit more in processing power. Unfortunately, these folds rather complicate the process of mapping the activity across the sheet. To overcome this problem the research team inflated the model cortex using a computational process. Projecting the data onto this simplified brain model creates a much clearer visualisation. These flat maps are easier to interpret and the team suggest that the technique could be very useful for many other fMRI studies.
Now to move on to Word association…
“Our study found that different individuals have remarkably similar semantic maps” comments Huth. For example, all seven of the subjects showed a particular area for words related to people. Interestingly, that area is surrounded by others selective for visual, tactile and number words. Groups of related concepts such as these suggest semantic associations are a basic organising principle for the brain and the meaning of language. A region that represents the word ‘victim’ also responds to ‘killed’, ‘convicted’ and ‘murdered’. However, all of the subjects in this study are native English speakers. It remains to be discovered how these semantic maps change with different languages and cultures – one of the aims of the continuing research.
The colourfully tiled visualisations of these distinct areas for different information provide the first comprehensive view into how meaning is represented around the cerebral cortex. They show that language is processed across very broad regions of the brain, not limited to a few areas as previously thought. Furthermore, the images show that semantic activity is roughly as large and varied in both hemispheres of the brain. This may force some scientists to rethink their belief that language only involves the left hemisphere. However, this belief was inherited from studies of language production, not comprehension as studied here, leaving plenty of room for debate and further study.
Now, let’s look at decoding language in the brain…
“It’s possible that this approach could be used to decode information about what words a person is hearing, reading, or possibly even thinking from fMRI data” says Huth. The team are some way off a general purpose language decoder, however in the future it could make a huge difference to those suffering from communication disorders such as ALS or locked-in syndrome. Future studies may help to understand how the semantic system differs with language disorders like dyslexia and autism. The work could also help develop tools to study how people recover from brain injuries and strokes.
Currently, they are working on applying the method to map other kinds of language information, such as phonemic information, syntactic information and narrative. More research is also planned to verify the semantic atlas they have developed, hoping to test subjects listening to different stimuli, reading instead of listening and listening in a different language. Semantic maps could vary between individuals from different cultures and who speak other languages. The team also want to look at the links between the semantic system and other senses, for example, “what is the relationship between the representation of a visual scene containing a dog, and the representation of a story about a dog?”
And finally, What is fMRI?
Neural processes are difficult to measure at high resolution. Functional Magnetic Resonance Imaging measures brain activity indirectly by detecting changes in blood flow, oxygenation and volume. The assumption is that fMRI signals are closely related to synaptic activity just upstream of the point of measurement.
Using fMRI technology has been validated in previous studies, which show that fMRI signals are closely correlated with local field potentials and local multi-unit activity – other measurements that are closely associated with activity of excitatory neurons. “That said, it is prudent to remain agnostic about the precise relationship between fMRI signals and neural signals” states Huth.
Leave a Reply