Tesseract: Turning Music Into Paintings
Salvador Dali, a four-dimensional cube, NeurIPS papers, and the copyright question that reframed the robot as an instrument, not an author.
The name of my project, Tesseract (tesseract.art), comes from Salvador Dali. To explain why, I have to talk about a four-dimensional cube and a painting of the crucifixion.
Dali created a picture of a Jesus figure suspended in midair, and behind it is what looks almost like a three-dimensional cross. That cross is actually an unfolded tesseract. A tesseract is a four-dimensional cube. We can't see in four dimensions or truly picture them, but these higher-dimensional geometric objects exist mathematically. Dali was fascinated by exactly this. He worked with professors, including mathematicians around Brown University, who were studying multi-dimensional mathematics, and he wanted to understand and depict art in more than three dimensions.
Here's the intuition that captivated him, and me. Think about a shadow. A three-dimensional object casts a two-dimensional shadow onto a surface, a projection from three dimensions down into two. In the same way, a four-dimensional object can cast a three-dimensional "shadow" into our world. An unfolded tesseract shows up as six cubes; the full tesseract has something like twenty-four faces in four dimensions. That idea, of projecting something higher-dimensional down into a form we can perceive, is where the whole project started.
There was a smaller inspiration too: synesthesia, the condition where senses cross over, where a person might experience sound as color, or smell as color. It's a beautiful way of turning one kind of perception into another, and it sat in the back of my mind. But the real spark was Dali and his obsession with dimensions.
Because our first approach was to do exactly that with music. We took a live musical performance and treated it as a multi-dimensional input signal, then analyzed it across many dimensions to create art. Music isn't just loud or soft, bass or treble. In any given interval you can ask how many notes are happening, what the intensity is, and many subtler qualities beyond simple loudness or frequency. We analyzed all of that and turned it into a painting, an acrylic painting produced by a robot with special end effectors I built for the purpose.
I worked on this initially with Dr. Richard Savery in Australia, and we published about it a couple of times at NeurIPS, the Neural Information Processing Systems conference, where we were accepted into the Creative AI track. The first paper was about the artwork itself and how we generated it. The second year, more recently, we built diffusion models using a style LoRA, a low-rank adaptation that sits on top of a diffusion system and lets you generate images in a particular artist's style.
That style work led to one of my favorite demonstrations. We had an artist in residence with a distinctive caricature drawing style. We took fifty of her caricatures, people with long hair and short hair, young and old, glasses and no glasses, beard and no beard, and used them to train the style. After that, we could apply her style to any photo. At the Super AI conference, we could take anyone's picture, turn it into a caricature that looked exactly like the artist had drawn it, and then have the robot paint it.
People sometimes assume the point of all this is to replace artists. It's the opposite. Our whole intention was to use AI to give people creative superpowers, to increase creativity, not take it away.
And doing this seriously forced us into a genuinely interesting question about copyright. When a robot paints a live performance, who owns the result? Our view is that the musicians do. The painting wouldn't exist if the musicians hadn't performed. The robot is just an arm, a prosthetic, an instrument. When someone plays the saxophone, nobody says the saxophone owns the copyright; the person does. The robot arm is no different.
That question, the robot as instrument, not author, turned out to be the doorway to everything that came next. Because once you see the robot as a prosthetic for human creativity, you start asking who else could use one. And that's the thread that led me from turning music into paintings to helping people who can't hold a paintbrush paint with their minds.