One of the most interesting concepts I discovered during my initial research was concatenative sound synthesis. The main idea is simple: given a database of sound snippets, concatenate or mix these sounds in order to create new and original ones. The concept is vague enough to guarantee rich literature in solutions and ideas with plenty of interesting research projects, such as CataRT and earGram.

My previous experience with dimensionality-reduction techniques on different domains led me to believe that matrix factorization of spectrograms would be an interesting way to perform concatenative synthesis. Allow me to explain.

A spectrogram is a 3D representation of audio that displays which frequencies of a signal are active over time. Non-negative matrix factorization (NMF) can be applied to the spectrogram in order to decompose it into a set of units, each composed of a component and an activation.

Components and activations can be seen as fine-grained representations of the original spectrogram. These elements decompose the original signal into separate units that, when joined together, are able to (approximately) reconstruct the original spectrogram. They also possess mathematical properties that allow us to perform some very interesting tasks.

First, each component and activation combination can be independently reconstructed. This means that NMF can be used as a non-supervised sound source separation technique, grouping related frequencies into units, in a lot of cases amounting to the instruments used in the original track.

Second, components and activations can be changed. For example, you can swap the components in a sound, effectively making one instrument sound like another.

This is not limited to the same source sound. We can swap components of different sounds, allowing us to make instruments sound like instruments from other tracks!

We can even replace all the components in a sound, while retaining the rhythmic structure but completely changing the way it sounds.

With these properties in mind, I created a tool that uses NMF to perform concatenative sound synthesis. This tool allows me to decompose an audio file into components and activations, then swap them with components from a database of good components, which I carefully built over time. This way, I can extract instruments from songs that I like and interactively find interesting variations for the way they sound.

Another very interesting area of research in the audio domain these days is the identification and separation of specific sound sources. Given some audio, can we remove the vocals or extract the percussion?

With the advent of deep learning models, there was a substantial increase in the quality and availability of tools for this task.

After testing a few available open-source tools, I came to several conclusions. First, imperfect separations are sometimes the most interesting, especially the results from models trained to separate something that isn't in the source sound (e.g., extract vocals from an instrumental track).

Second, it is hard to predict how good the separation on a sound source will be. Trial and error is your best friend here.

To help in the difficult task of finding good separations, I've developed a command-line tool that uses several models and parameter combinations to produce many different separations for a specific source of audio. This allows me to get a good overview of interesting separations that can be extracted. For example, if I like the percussion on a track, I let my tool produce dozens of different percussion separations, and then select which one sounds the best given the context I'm going to use it in.

Another extremely interesting and underrated task in the audio domain is translating audio into MIDI notes. Melodyne is state of the art in the area, but Ableton's Convert Audio to Midi works really well too.

Both the concatenative synthesis and source separation tools allow us to extract fine-grained separations from an original sound source. I found that those tools create interesting, although imperfect, translations into MIDI. Resynthesizing this MIDI allows us to generate intriguing variations that "sound like" the original, although in unexpected ways.

All of these techniques became part of my music creation process.

I start by sampling loops from songs I like, and then use the concatenative synthesis and source separations tools to extract interesting musical ideas (e.g., melodies, percussion, ambiance). These results can be used directly, with a little FX (Replica XT, Serum FX), or translated into MIDI and resynthesized (Serum, VPS Avenger). This gives me the building blocks of the song (the tracks), each a few bars long, which I carefully combine using Ableton Live's session view into something that sounds "good".

Later, each track is expanded by sampling more loops from the original songs until I have plenty of content to create something with enough structure to be called a "song".

The biggest challenge in the whole process was dealing with imperfection. Concatenative synthesis commonly produces harsh-sounding results, sound source separations are rarely clean, and MIDI translations don't always make sense. Instead of obsessing over perfection and going down the path of carefully cleaning all the aggressive leftovers created by these tools, I decided to embrace imperfection and use it as the unifying theme of the album.

Melodies with unusual progressions, rough low-fi percussion, lush pads for masking undesired noises, and strange-sounding samples all give the album its distinctive character.