Ad

Scientists put visions of letters in blind people's brains

Stimulating the brain in specific ways can generate mental images of simple shapes

Meredith Schmehl

Neurobiology

Duke University

Surrounded by the buzz of medical equipment, a blind man raises his hand to a touch screen. Pop! A vision of the letter "N," placed in his brain, flashes through his mind. He traces his finger across the screen, replicating the vision with perfect form.

It sounds like science fiction. But in a recent study at Baylor College of Medicine, researchers made the blind see. A team led by neurosurgeon Daniel Yoshor "drew" letters of the alphabet on blind people's brains by giving them specific patterns of electrical zaps. These patterns caused the participants to "see" the letters in their mind's eye. The results could improve medical devices for people who have experienced other types of sensory or motor loss.

A man traces a shape, stimulated in his mind, on a computer screen with his finger

A participant in the study is shown drawing a letter that was stimulated in his brain

Screenshot via Daniel Yoshor on YouTube

The researchers accomplished this by giving patterns of small electrical stimulations to the visual cortex. The visual cortex is one of the hubs in the brain that responds to what we see. This region contains a spatial map of our field of view, meaning particular sets of cells respond to visual information coming from particular locations in our line of sight. Turn on a light on the left side of your field of view, and one set of cells will respond by shooting off an electrical signal. Turn on a light on the right side, and a different set of cells will respond.

Yoshor's team took advantage of this map in a clever way. Because the cells in the visual cortex respond to patterns of light in space, the scientists could reverse the process — give a tiny electrical zap to a particular group of cells and cause someone to perceive a spot of light at a specific location. They performed brain surgery on blind adults to implant a small electrical device with several points of contact to the visual cortex. Each point could be activated individually or in combination with others to stimulate the brain in precise patterns. By carefully controlling the combinations of activated areas, the researchers could cause someone to "see" a specific shape, such as a letter of the alphabet.

But there was one problem. Other researchers had tried similar experiments, but they hadn't been successful in creating a detailed image in someone's mind. These previous experiments stimulated discrete patches of cells, like a shape drawn with dotted lines. But instead of seeing a continuous shape, people just saw several blobs of light. In order to create a detailed mental image, Yoshor's team had to improve on existing techniques for stimulating the brain.

They realized that they could borrow a technique that's common in cochlear implants and auditory therapy to improve the way they stimulated the brain's visual center. This new technique allowed them to replace their dotted sketch with a single continuous line to "draw" shapes such as letters on the brain. They reasoned that the brain might be able to make more sense of a stimulation pattern that is drawn continuously, since the pattern would more closely match the appearance of most letters in daily life.

To achieve this continuous drawing in a person's brain, the researchers gradually changed the level of activation of each point of contact. Instead of turning each point just on or off, they raised or lowered the dial on each one to create a wide range of activity in the corresponding brain cells. When they gradually increased the activation of one point of contact while gradually decreasing the activation of another, they caused people to "see" a continuous line of light corresponding to the space between the two active points of contact in the brain. Participants described the image "like a line being drawn" in their mind's eye.

Now that the researchers had improved their technique for stimulating the brain, they were able to test whether complex activation patterns could cause people to perceive shapes like letters. They stimulated groups of several points in the brain, adjusting the dial at each point to draw lines in spatial arrangements that matched the form of real letters. They asked people to replicate the shapes they saw on a touch screen, and also to respond aloud to indicate which letters they perceived.

prosthetic arm

The results from this study may help people better control their prosthetics

Photo by ThisisEngineering RAEng on Unsplash

The participants were consistently able to reproduce and name letters that matched the stimulation pattern drawn on their brains. When asked to name letters in rapid succession, they accurately named nearly 90 percent at a rate of 85 shapes per minute. In other words, the new stimulation technique successfully generated clearly shaped letters in people's minds.

But the technique isn't perfect. The activation patterns aren't precise enough to allow people to tell the difference between similar shapes, like the letters "C" and "O". And the participants in this experiment became blind as adults, meaning they already knew what the letters looked like. It's likely that it would have been more difficult to replicate the experiment's results in someone who had been born blind and had never seen letters or other shapes.

Nevertheless, this experiment is a promising step forward in scientists' ongoing effort to create a device that could restore vision to those who have lost it. With further improvements in the technology, scientists might be able to stimulate the brain with a finer resolution and generate more detailed mental images. The results could have implications for other types of sensory function, too. Finer stimulation methods could improve pitch perception in users of cochlear implants, where a version of this technique has already been tried, and could help restore a detailed sense of touch in cases of nerve damage.

Besides sensory systems, the results might also improve medical devices that help with motor control. For example, prosthetic limbs rely on technology that reads activity in the motor regions of the brain. Scientists could use their new knowledge about stimulating the brain to read and interpret the brain activity that controls a prosthetic, allowing people to control their new limbs in more precise ways.

It's clear that the results uncovered by Yoshor's team are a crucial development in the ongoing quest to create a visual aid for the blind. In the future, the technology might allow blind people to see the complex objects that surround them in real time.

Comment Peer Commentary

We ask other scientists from our Consortium to respond to articles with commentary from their expert perspective.

Kathryn Vaillancourt

Neuroscience

McGill University

Thanks for writing this article, Meredith, it highlights exactly the kind of spectacular research that made me want to be a scientist in the first place!

I especially liked how you used the dotted line analogy to explain how this study made improvements to the resolution of techniques that were already available. It really makes me think of what could be next: shading? Or even adding color?

Of course that would require stimulating more/different parts of the brain, but perhaps as their method gets less invasive!

Tareq Yousef

Dalhousie University

Wow! I thought this was a great choice of topic for an article. I remember reading this paper when it came out, being very pleased that artificial vision is continuing to progress. I think an interesting take-away for readers might be the retinotopy of the visual cortex, which you alluded to when describing the spatial map. I think a reader might find it interesting to know that not only do different visual cortex cells respond to different places of the visual field, but they are arranged so that the cells next to each other in the visual cortex also correspond to spots in the visual field that are next to each other. This maintenance of retinotopy (as you are already an expert in!) is what is so beautiful and efficient about the visual system, and is what allows this kind of technology (though novel and in its first stages) to work.

Dori Grijseels

Neuroscience

University of Sussex

This is such an interesting topic, and a great explanation. The first thing that came to mind for me was an experiment that I participated in on haptic (the science of touch), where they found that participants were better able to distinguish shapes drawn on their hands when they were drawn as a line rather than presented as a single shape. It seems like this is a common feature of our sensory systems. I wonder how much this is related to our tendency for exploratory behaviors (such as eye saccades).

I do think an important point to be made surrounding this, is the ethics. In order for a technique like this to work, the participants had to be implanted with electrodes. This is most definitely not risk-free (see e.g. this article). Of course, if this technique can help blind people to see again, that is great, but not everybody might want to have, or have access to, such a procedure. Ultimately, it should not replace other means of accessibility.

Vidya Ganapati

Swarthmore College

Fascinating article. At the end, you mention that this technology may “allow blind people to see complex objects that surround them in real time.” What do you think are the major barriers to go from recognizing letters to seeing arbitrary objects? How quickly can an object be drawn and at what resolution? Will software and hardware advances be needed to distill a scene into a drawing, or do you think existing off-the-shelf components can be used?

Meredith Schmehl responds:

Hi Vidya, these are great questions! Some of the major barriers include translating the techniques developed here for drawing 2D objects (letters) to later draw 3D images of everyday objects, as well as increasing the resolution of the perceived images. And as you mentioned, the speed of the drawing is another barrier - although the scientists in this experiment were able to draw each image in less than one second, the technique isn’t fast enough to convey complex information about a visual scene, such as objects in motion. To improve in these areas, scientists will need to develop finer stimulation methods, and participants may need to be trained to recognize particular stimulation patterns.