AI Decodes Brain Waves of Paralyzed Individuals into Real-Time Audio

Men who have lost their ability to speak can engage in real-time conversations and even sing using brain-controlled synthetic voices.

The brain-computer interface captures neural activity through electrodes implanted in the brain, instantly creating audio sounds that match intended pitch, intonation, and emphasis.

“This represents a breakthrough in instantaneous speech synthesis, achieving this within 25 ms,” says Sergei Stavisky from the University of California, Davis.

While advancements are needed to improve speech clarity, Maitreyee Wairagkar, also at UC Davis, notes that the individual who lost his speech due to amyotrophic lateral sclerosis expresses happiness and feels that he has found his true voice.

Existing speech neurospheres that utilize brain-computer interfaces typically require a few seconds to convert brain activity into sound. Stavisky mentions that this delays natural conversation and if the connection falters, it can feel like speaking on a poor-quality phone call.

To create a more seamless speech experience, Wairagkar, Stavisky, and their team implanted 256 electrodes in the areas of the male brain responsible for facial muscle control necessary for speech. In subsequent sessions, they introduced thousands of sentences on a screen, recorded brain activity, and prompted the subject to vocalize with specific intonations.

“For instance, phrases like ‘How are you today?’ or variations such as ‘How are you? today?’ can significantly alter the meaning of sentences,” explains Stavisky. “This approach allows for a richer, more natural dialog, marking a significant advancement over previous technologies.”

The researchers utilized an AI model trained to link particular patterns of neural activity with corresponding words and tonal variations, resulting in synthetic speech that mirrors both the content and emotional delivery intended by the user.

The AI was trained with audio recordings from before the male’s condition deteriorated, employing voice-cloning technology to ensure the synthetic speech bore a resemblance to his original voice.

In another phase of the study, researchers attempted to teach him to sing a simple melody with varying pitches, with their models accurately interpreting the intended pitch in real time and adjusting the produced singing voice accordingly.

He also utilizes the system to communicate spontaneously, making sounds such as “hmmm,” “eww,” and forming words, as noted by Wairagkar.

“He’s a remarkably articulate and intelligent individual,” says David Brandman from UC Davis. “Despite his paralysis, he has continued to participate actively in work and engage in meaningful conversations.”