Five5quare
Beautiful music and hypnotic animation are at your fingertips. If you can bend your fingers, you can play a song!
Project type
A visual musical instrument for non-musicians, made with Max Horwich (audio, code) and Amena Hayat (hardware, code).
My role
Visual animation, code, prototyping
Outcome
- A working instrument: four flex sensors on a glove, a step sequencer and a paint canvas
- A score anyone can read: color is the instrument, position is pitch, opacity is volume
- Five synced animations, each shape moving to its own sound
Idea
An instrument for people who can't read music
Inspired by Kandinsky from Chrome Music Lab, I wanted to explore how drawing could become a way to create music. The challenge was to balance structure and spontaneity, giving users enough constraints to create coherent compositions while leaving room to experiment and freestyle.
The result combines a traditional step sequencer, which loops different instruments, with a drawing-based instrument and wearable gesture controls to create and manipulate melodies. The interface is designed to be intuitive enough for anyone to play, regardless of musical experience.
The prototype
A visual language for music
As an early POC of the idea, I put together an early visualization of the interface as something to aim towards. Each attribute of this graphic corresponds to a musical aspect
- ColorA separate channel playing a different sound
- ShapeThe texture and timbre
- PositionThe pitch
- OpacityThe volume
My collaborator Max translated the graphic into music as an early start
Next, I made a POC of a basic drawing program, with two different oscillators mapped to the x and y axes.
POC of web interface and serial control
As the flex sensor was giving me erratic values, I mapped the sliders to a potentiometer to simulate the glove.
Changing the tempo for different sound families seemed to be a good way to quickly add a rich level of expression.
For the tangible control, we narrowed down on a gestural interface using 4 flex sensors, controlled by an Adafruit Flora board, sewn onto the glove for 4 fingers.
Animation
Making shapes move to the music
I figured out the different sound analyzers in the Tone.js library and used that data to animate each shape.
Circle, kick
Tone.progress gives the value of the kick loop over time. It's mapped to the circle in reverse, so it swells to its peak as it hits the kick.
Square, bass
A waveform analyzer on the bass. It took me nearly half a day to get this working! Mapping the values to a line only moved a few points, leading to much frustration. Eventually I used the array to make the square vibrate to its waveform.
Triangle, snare
I used the waveform again to slightly shake the triangle as it hit the snare.
Dynamic dots, piano
The piano loop has 4 sets of chords, each with 3 notes. With help from Shivanku, I used Tone.progress as a global timer, broke it into three phases, and played a different circle animation at each.
Paint, lead
The paint program plays different notes as you paint across the screen, mapped to different zones on the canvas.
Hardware
The glove
In parallel, Amena was making progress on the glove. It uses a simple circuit of 4 flex sensors wired to the pins of the Flora board.
The step sequencer is where you loop different instruments, and the drawing program and glove combine to freestyle with the music.