Piano Genie

Have fun pretending you’re a piano virtuoso with the help of AI. Use the 1-8 numbered keys in your keyboard or touch the colored buttons to play a full 88 key piano. The more you pretend to be a real player, the better the AI will make you sound.

When playing you decide the timing of the notes and have some control over the “shape” of the melody: whether it should move up or down in pitch, and by how much. The Piano Genie AI will assist you by choosing which piano keys to play to make you sound more like a professional player.

Piano Genie was programmed without coding in any theoretical rules about harmony or composition. Its AI is based on a neural network that was trained with 1400 performances from the International Piano e-Competition. We can say that through this training it learned on its own, and by example, what piano music should sound like.

How it works

Neural Networks are AI systems inspired by biological brains. Similar to how groups of neurons work, they take an input (e.g. signals from the nervous system) and produce an appropriate output (e.g. activate a muscular response). The network learns to give “intelligent” responses through a training process: it receives many examples of input, has its performance evaluated on each, and then is given feedback so it can improve over time.

For a long time most AI systems required the programmer to precisely describe how to perform a task to the computer. Neural networks, on the other hand, are commonly used for tasks where describing the decision making process is not possible because of its complexity. They’re used for a very large variety of tasks such as image and voice recognition, playing games, developing drugs and vaccines, driving cars, or even creating music.

The Piano Genie AI was built by first creating two separate neural networks: an encoder and a decoder. The encoder receives a melody (a sequence of notes) represented as the numbers 1-88 (the keys of the piano) as input and produces a sequence with the same length but using only the numbers 1 through 8. The decoder reverses the process by taking this simplified sequence and outputting a melody that once again uses the different 88 notes a piano can produce. By connecting the encoder and decoder together the melody passes through a sort of funnel.

Of course it’s not always possible to perfectly recover the original melody once it passes through the encoder and decoder. With a full piano each note of a melody can be any of 88 available pitches. When the notes of a melody are encoded using only 8 symbols it means that after each note there are only eight possible ways to continue. We can decode it back to something that hopefully sounds similar, but will probably not be the same.

There’s not a unique way to encode the notes using 8 different values so they can later be decoded back and the creators of Piano Genie didn’t know which one was the best. This is the role of AI: by training them, the neural networks can find the best way to do this conversion.

Both the encoder and decoder networks in Piano Genie were trained simultaneously using the 1400 piano performances mentioned. The networks were evaluated by looking at how closely the output of the decoder matched the original melody. The encoder was also evaluated based on how similar the “shape” of the 8 note sequence was to the original (for instance, if the notes in the melody went up and then down, the numbers of the encoded sequence should go up and then down as well).

The top image shows a piano roll visualization of a real performance. Below, the sequence generated by the encoder using 8 different symbols. You can see they both share similar shapes.

Source: Google Magenta

Once the encoder and decoder learned to perform both tasks well enough, the authors of Piano Genie took the decoder (by itself) and connected its input to the 8 colored on-screen buttons. When you play on Piano Genie’s keyboard, the decoder receives your sequence of key presses and tries to decode it into a piano melody. The decoder network is attempting to reconstruct a piano improvisation that actually never took place.

Why it works

We wouldn’t consider every possible combination of piano notes musical, just like not every random combination of letters is a sentence. Music has structure: it follows patterns, scales, motifs. When the encoder and decoder are trained, they learn how to compress music (that is, express it using a smaller amount of symbols) by finding the common structure and patterns in the examples we give them. They work with the musically interesting sequences and ignore the arbitrary ones.

When the decoder neural network is connected to the 8 Piano Genie buttons, it produces note sequences that share structure to the ones in the songs it was trained on.

Further experiments

Check out the settings button in Piano Genie. You’ll find options for using MIDI instruments for input or output. There’s also an option for using a Makey Makey kit to play the piano using everyday objects.

The rock band The Flaming Lips used Piano Genie on its live shows to play the piano by touching fruit in a bowl.

Some things to think about

In the I AM A.I. Exhibition

The musical exhibit AI Jam, also part of the Google Magenta project, lets up to two people play a duet with an AI using a piano keyboard and drum pads. It also uses a neural network trained on many hours of actual performances, which tries to play something similar to what the user just did.

Learn how to visit.

The official Piano Genie page at Google Magenta
The original Piano Genie paper
The Piano Genie source code

  Text is available under the Creative Commons Attribution License.

Continue exploring