Teaching Sophia to Sing

After the robot musician at the Opera House, I kept finding my way back to robots, and one of the most fascinating detours was Sophia, the humanoid robot from Hanson Robotics in Hong Kong.

At the time, Sophia was probably the most sophisticated robot in the world, and her most striking feature was her skin. It looked genuinely real, but it could also be manipulated by motors inside her head, which is what let her expressions look natural. She became famous, she was even granted citizenship in a country. But she was, and is, more of a spectacle than a working machine. She isn't built to do tasks on a manufacturing floor. Her arms were fully 3D printed, made for gesturing, waving, saying hi, not for real manipulation.

My contribution was to teach her to sing.

She already had a speaking voice, but this was around the COVID period, before the large language models we know now. ChatGPT wasn't something ordinary people had heard of. What we had was much more classical machine learning. In technology terms it feels like decades ago, even though it was only a few years, that's how fast this field moves.

Sophia's speaking voice had originally been built from recordings of a real woman in Ireland, who went into a studio and carefully voiced an enormous set of sounds. The building blocks of speech aren't just letters; they're phonemes, the individual sounds that make up the words we say. Capturing those in high detail is what made the voice sound so good. The company behind that work had a remarkable track record; they'd previously taken recordings of John F. Kennedy's voice and reconstructed the speech he was driving to deliver on the day he was assassinated, the version that now lives at his memorial in the U.S.

But no one had ever asked them to make that voice sing. That was new. And singing well is not just hitting perfect pitch over and over, that sounds robotic, like heavy auto-tune, which was exactly what we were trying to avoid. We wanted her to sound natural: vibrato, style, notes that fade and bend. So beyond the machine-learning training you might imagine, I did a lot of hands-on programming based on pitch, timing, and frequency, adding vibrato to the tails of notes, introducing a bit of randomization and imperfection, deliberately making the pitch not perfect. The artistry lived in the nuance.

Once she could sing, we did a live duet: Sophia and Jimmy Fallon on The Tonight Show. Millions of people watched it. It was a lot of fun.

Then I went further and started teaching her to play the piano, and that meant rebuilding her body. Her arms were designed to grasp, to close around a cup. Piano playing needs something completely different: fingers that move up and down like little pistons, that separate and strike keys independently. So, working with the mechanical and mechatronics engineers at the company, I fully redesigned her arms. And it worked. Right before COVID, we had Sophia playing piano and singing at the same time.

We were genuinely one step away from something remarkable: an improvisational musician. Imagine an AI that improvises, with a human in the loop giving it feedback, a simple "that sounded interesting" as a plus-one, or "no, that was weird" as a signal not to go there again. Over time it would learn its own musical taste. We were right on the edge of that.

Then COVID hit and killed the project. It stopped there.

But Sophia was never really a destination for me. Looking back, she was one of a series of touchpoints, the Opera House robot, Sophia, the painting robots that came later, each one pulling me deeper into robotics and into the same obsession: using machines not to replace human creativity, but to express it. That thread didn't end when the Sophia project stopped. It's exactly what I'm still chasing now.