Painting With Your Mind
From music to brainwaves: building brain-control interfaces so people who have lost the ability to move can create again, and the mission behind Tesseract.
Everything I've built, the speakers, the coral reef cameras, the robot musicians, the painting robots, quietly led to this, though I didn't know it at the time. The project is called Tesseract, and its purpose is to let people who have lost the ability to move express themselves creatively again.
We started, as I've written elsewhere, by turning live music into paintings. A robot listened to a performance and painted it, and we spent a lot of time on the academic side, pushing at questions of copyright, arguing that the musician, not the robot, owns the art. But about two years into that work, a different question started to take over. We were already training a system on a complex input signal, music, and doing multi-dimensional analysis of it. Wouldn't it be far more meaningful if the input signal were brainwaves?
The personal motivation was close to home. My dad had Parkinson's, so I watched a progressive, degenerative disease take away someone's abilities up close. And I kept thinking about all the people who used to be able to create and no longer can, someone who's quadriplegic, someone in a wheelchair, someone with a degenerative condition that took their hands. There are extraordinary artists trapped in bodies that can't express what's inside them. Audio was an easy input signal to work with, you just need a microphone. But if we could move from audio to brain control, we could give creative expression back to people who had lost it.
This year, in just the last couple of months, I've been able to demonstrate exactly that, twice.
The first was on stage at the AI Engineer conference in Singapore, with a woman named Kai Ming. She's an artist from a family of artists, but she suffers from a progressive disease and can no longer hold a paintbrush without pain. We gave her a wireless, Bluetooth-enabled headset. Using her attention and other signals, she can paint. We read EEG signals, brain activity, along with EMG, the electrical signals from muscle movement, to pick up what she wants to do, and then we paint with that information.
The second was about a week ago, and I'll carry it with me for the rest of my life. A fifteen-year-old girl named Sofia. She has an extremely rare condition, a form of leukodystrophy sometimes referred to as HABC, that affects the myelin and the part of the brain that controls fine motor movement. She used to be able to move and talk, and she used to paint when she was five. She can't do any of that now. She can't speak, she can't move her arms, she's in a wheelchair.
We put the headset on her, and she painted. Two paintings, one meant to be a flower, one of her favorite animal, a sloth, controlled by her mind, timing her intentions with the strokes. And there was this moment where she suddenly came alive. She smiled, this mischievous grin, and reached out to hold her mom's hand. Everyone at the booth was crying. It was the most special moment.
Her mother put it better than I could. She talked about being taken back to when Sofia was five and would say, "Hey mommy, I want to draw a flower, a house, the garden", and how this gave her a glimpse of what could come back.
That's when I understood why I'd spent four years on this. When I started, it was music and academic papers and arguments about copyright. I didn't set out to build brain-control interfaces so that people with disabilities could express themselves. You don't decide that in advance. You follow your curiosity, you follow the passion, you take the steps in front of you, and then a moment like Sofia reaching for her mother's hand tells you what it was all for. I didn't know the reason at the time. Now I do.
And creative expression, as moving as it is, is really just the beginning. The technology today, with a simple EEG headset, can reliably detect around thirty words that almost anyone can produce by putting the headset on, think of an apple, and we can tell you're thinking of an apple. Thirty control signals is enough to do a lot. Jackie, a PhD researcher whose company works on brain interfaces, and who I only met about two months ago, and I are now building a higher-resolution interface that will do far more.
Because here's the bigger vision. So much of the anxiety about AI is that it's taking jobs. But a lot of the jobs AI creates need a human in the loop, someone supervising an automated process, catching the thing that goes wrong. Those jobs don't require you to be in the room. You could be at home. Which means a person with a disability who can't easily leave the house could do that work, and be gainfully employed. Think of a "dark factory", a fully automated plant that doesn't even need the lights on, still needing someone to watch and intervene when something falls or fails. That's a role a person controlling a computer with a brain interface could fill.
There are people with disabilities who've lived with severe depression precisely because they couldn't contribute, couldn't be part of things. This is potentially a whole new economy for them, creative expression first, because it brings joy immediately, and then real, dignified employment.
That's the mission of Tesseract. It isn't really a company for profit; it's a company for purpose. From audio to hardware to software to AI, all of it has delivered me to a point where I think I can help millions of people find fulfillment in their lives. If I can bring together the right hardware and the right software to make that happen, I'll consider this the thing I was meant to build.
This piece touches on serious illness and disability. If any of it resonates with something you're going through personally, please reach out to a professional or someone you trust for support.