Brain Computer Interface Decodes Speech and Facial Expressions

Researchers at the University of California San Francisco have developed a brain computer interface that can lets someone with severe paralysis communicate with both speech and facial expressions, in the form of a digital avatar. The breakthrough advances what has been possible, with previous brain computer interface systems providing speech only, and allows people to communicate more completely, encompassing facial expressions, which are an important aspect of natural communication. The system includes electrodes that intercept brain signals that are intended for the muscles of the face, essentially decoding complex facial expressions.

Brain computer interfaces have provided a window into the minds of those with severe paralysis who may otherwise struggle to communicate, or who may not be able to communicate at all. Controlling motorized wheelchairs and other robotic assistance devices is one aspect of such technologies, but communication remains one of the most important.

Despite this, to date, brain computer interfaces have focused on offering basic speech capabilities. However, human communication is more sophisticated than basic speech, and includes a myriad of complex facial expressions and other body language. In this latest development, these researchers have pioneered the use of a digital avatar as a means for a severely paralyzed woman to communicate using a repertoire of facial expressions that accompany speech.

Moreover, the new system can decode these brain signals at a speed of 80 words per minute, which is a marked improvement in speed over pre-existing commercial technologies. The researchers attached electrodes to areas of the participant’s brain that are involved in speech and facial expressions, and trained the system over time to recognize the signals that corresponded to certain words and facial expressions.

“The accuracy, speed and vocabulary are crucial,” said Sean Metzger, a researcher involved in the study. “It’s what gives a user the potential, in time, to communicate almost as fast as we do, and to have much more naturalistic and normal conversations.”

Here’s a video from UCSF about the technology: