iota 

Sound Design With OpenAI's Jukebox (2022)

 

Overview

iota is an experimental piece created with the assistance of Jukebox by OpenAI, a neural network that generates music and elementary singing in a wide selection of genres and artist styles. For this experiment, the model was used as a means of generating a new music sample from the stems of an existing song to inspire a new one. I used Google Collab Pro to run the code on one of Google's graphic cards to be able to run this test. It served as a sound design tool.

‍

Documentation

The process was broken into three stages. Synthesize, Sample Extraction and Reconstruction.

‍
Synthesize:

He wrote a short song comprised of eight sounds, each created using solely wavetable synthesis. Then they were assembled into 22 second 4-bar loops.

‍

Sample Extraction:

Each loop was fed to Jukebox. The model then attempted to produce a new music sample by continuing the loop for a given length of time while adhering to a number of parameters that were set in place. In Jukebox, you can choose from the their database of artists and genres and provide your own lyrics as well. The most interesting results were the result of mismatching the artist and genre, for example Frank Sinatra and rock music. The same lyrics were provided for each run, simply sticking to "ooh" and "aah".

​

The initial generation of the audio goes through a process called downsampling, which results in poor-quality samples with apparent noise. The next stage is a process called upsampling, which essentially restores the audio to a higher-quality state. Upsampling the audio felt unnecessary because he believed the audio was more then salvageable. The natural grit of the sample was used as a way of texturing the sound. He regularly tends to inject white or pink noise into his own sample creation process to achieve a more textured result, so this seemed like an interesting alternative. The outcome varied from pure noise and ghostly voices to fragments of these somewhat beautiful textures. This was achieved while still maintaining the rough musical structure of the original audio. 
‍

Reconstruction:

The audio generated by Jukebox was used as the source material for a new song. The audio was predominantly manipulated using a variety of granular synthesis processes. The process consisted of essentially scrubbing through the sample for a section that was tonal, isolating it and reconstructing it into a composite of micro-loops and a myriad of ambient textures.

‍

‍Result / Analysis Audio Examples

Documentation of each stage can be found below.

WARNING: the samples generated by Jukebox are sometimes a bit shrill.