A Robot Musician at the Sydney Opera House

My first real step into robotics didn't come from a lab or a research paper. It came from walking into the wrong room.

It was around 2014, and I was at Accenture. I'd flown to Bangalore, and I happened to wander into a room where there was a robot, a collaborative robot, about the size and torso of a person, on a pedestal, with two arms and a cute screen for a face. I recognized it. It was a Baxter robot from Rethink Robotics. Rethink doesn't exist anymore, but at the time Baxter was one of the first collaborative robots people were putting on the manufacturing floor. Its whole appeal was that you didn't need to be a software engineer to program it: you just physically moved its arms to where you wanted them, and it learned the positions. Move it here, move it there, and it would repeat the task.

Watching it, I had two thoughts at once. The first was that it was a genuinely cool machine. The second was that it had such a boring life, pick this up, move it there, pick this up, move it there, forever. I thought: let's make it a musician. Let's do something creative and entertaining with it.

The chance came months later, back in Australia. Accenture and Adobe are big partners, and the person leading that relationship told me the Adobe Symposium couldn't use the Sydney Convention Centre that year because it was being renovated, so it was going to be held at the Opera House instead. What could we do that was special? I said I had just the idea. We take the robot, turn it into a musician, and put on a live show in the Sydney Opera House concert hall.

Everyone thought I was crazy. Naturally.

But by then I'd built a team inside Accenture called the Liquid Studio, full-stack engineers, hardware engineers, mechatronics engineers, all working together. We'd done drone work, including detecting rust and corrosion on leaky pipes under bridges using drones and early computer vision, OpenCV, object recognition, OCR, the early days of applied AI. So we knew how to build things.

For the concert, we gave Baxter special hands. With these robots you can attach whatever end effector you want, a parallel gripper, a humanoid hand, suction cups for picking up circuit boards. We built custom end effectors holding marimba mallets, three in each hand. Then we trained the robot using Google's Magenta project, a Google Brain music project of that era. The training happened in TensorFlow, running in the cloud, but once we had the model we put it on a Raspberry Pi, dropped that in a backpack, and strapped the backpack to the robot. A piano was connected to Baxter over MIDI, so the robot could effectively listen to what the human pianist was playing and improvise in response.

We set the whole thing up and performed it live in the Sydney Opera House.

I also earned a title I still hold inside Accenture: the only managing director to ever expense a grand piano.

And, true to form, the project got cancelled twice before it happened. What saved it was a deal I made back in Bangalore: if the India innovation center helped fund it, I'd bring the robot there afterward to use for their own proofs of concept. They said yes. I came back to Australia, where everyone assumed the project was dead, with about half a million dollars in funding, and we did it.

There was a strategy underneath the spectacle. A robot improvising live in front of a studio audience was, in a sense, thought leadership as a soft sell. If a partner or prospective client saw that we could pull that off, they could trust us to handle the comparatively simple stuff, robotic process automation, ordinary IT. The music justified itself many times over.

But for me, it was something more than a sales device. It was the first time I'd put creativity and technology on a stage together and watched them become one thing. Every robotics project I've done since traces back to that moment in a room in Bangalore, looking at a machine with a boring life and deciding to make it play music.