Creative Piano
This project explores how tangible physical interactions can be mapped to dynamic digital feedback using basic electronics and browser-based programming. The goal was to create an accessible, playful, and immersive musical instrument by combining capacitive touch sensing with 3D visuals and synchronized audio. Built using an Arduino Uno and a p5.js sketch, the outcome is an interactive interface where touching conductive pads triggers audiovisual responses on screen and plays musical notes through the browser.
Concept and Design
The project was inspired by the layout and playability of a traditional piano. Seven capacitive touchpads were arranged horizontally to simulate seven musical keys, each corresponding to a note from A to G. These touchpads were constructed from copper tape and wired to the Arduino using 1MΩ resistors and the CapacitiveSensor library, with digital pin D2 as the send pin and D4–D10 as receive pins. Touching a pad activates a signal sent via serial communication to a p5.js sketch running in the browser.
Visually, the p5.js interface features a WEBGL canvas with seven 3D cubes spaced evenly across a horizon-like scene. When a user touches one of the physical pads, the corresponding cube rotates, scales up, and releases a short burst of animated particles. Each key also triggers a distinct .mp3
file that plays the associated note. To complete the feedback loop, an LED mounted next to each pad lights up on activation, enhancing the physical response.
Implementation and Technical Challenges
The Arduino side of the system uses the CapacitiveSensor library for reliable touch detection. Serial data is transmitted using Serial.println()
to send a numerical value (1 to 7) to the browser. These numbers are received in p5.js using the Web Serial API. Each value maps to a cube on the canvas, a sound file, and an LED output. Due to the limited number of digital pins on the Arduino Uno, analog pins A0–A5 were repurposed as digital outputs for controlling LEDs, alongside digital pin D12.
One major technical hurdle was encountered when attempting to load all sound files within a loop in p5.js using loadSound()
. This approach caused the browser to silently fail to load the audio. The issue was resolved by loading each sound file individually, using separate loadSound()
calls with explicit success and error handlers.
Another issue involved unstable serial communication, particularly when switching between the Arduino IDE and browser. Ensuring the serial monitor was closed before running the p5.js sketch, introducing delays in the Arduino setup()
function, and adding robust error handling in the JavaScript code helped address this. Additionally, adding a 1MΩ pull-down resistor from pin D2 to GND improved signal reliability.
User Testing and Feedback
To evaluate the interface, I conducted informal user testing without providing any instructions. Users were able to understand the instrument’s function immediately due to its intuitive piano-like layout. Most could successfully trigger both audio and visuals without any guidance.
However, two issues emerged. First, the interface only featured seven keys, which users noticed as an incomplete octave. This design limitation was due to hardware constraints and the number of available input pins on the Arduino. Second, users reported a small but perceptible delay between touching the pad and hearing the sound, which slightly detracted from the interactive experience. Despite these drawbacks, users found the interface fun and engaging, and appreciated the multi-sensory feedback through visuals, sound, and lights.
Reflection
Overall, the project succeeded in creating a satisfying and creative interactive system that blends physical computing with browser-based media. The integration of touch, sound, and 3D visuals offered a cohesive and enjoyable user experience, demonstrating how simple hardware and software tools can be used to build meaningful interactions.
There are several areas for potential improvement. Adding an eighth key would allow users to play a full musical scale, which would greatly improve the musicality of the instrument. Reducing latency between touch and audio playback, possibly by optimizing serial reading or switching to a faster communication protocol, would also enhance responsiveness. Finally, some users have noted during the showcase that it would have been more interesting if piano keys could be pressed simultaneously. My personal vision for this project is making it gamified: when the cubes light up on the screen, inviting the user to plays corresponding keys to reproduce a certain song or melody.
Tools and Materials
-
Hardware: Arduino Uno, copper tape, 1MΩ resistors, jumper wires, 7 LEDs
-
Libraries: CapacitiveSensor (Arduino), p5.js (sound, WEBGL, serial)
-
Software: Arduino IDE, Chrome browser, p5.js Web Editor
-
Languages: C++ (Arduino), JavaScript (p5.js)
Conclusion
This project showcases how physical and digital systems can be seamlessly integrated to create interactive, expressive instruments. By leveraging capacitive sensing, serial communication, and browser technologies, the Capacitive Touch Musical Interface offers a compelling example of creative technology that invites play, experimentation, and multisensory engagement.
let port; let reader; let sounds = {}; let labels = ['A', 'B', 'C', 'D', 'E', 'F', 'G']; let musicNotes = ['C', 'D', 'E', 'F', 'G', 'A', 'B']; let allLoaded = false; let cubeSizes = {}; let targetSizes = {}; let cubeRotations = {}; let targetRotations = {}; let flippedState = {}; let cubeYOffsets = {}; let targetYOffsets = {}; let particles = []; let baseSize = 50; let pulseSize = 100; let jumpHeight = 40; let sizeLerpSpeed = 0.2; let rotationLerpSpeed = 0.15; let positionLerpSpeed = 0.2; let labelLayer; function preload() { soundFormats('mp3'); for (let label of labels) { sounds[label] = loadSound(`${label}.mp3`, () => console.log(` Loaded ${label}.mp3!`), err => console.error(`❌ Failed to load ${label}.mp3`, err) ); } } function setup() { createCanvas(1000, 500, WEBGL); noStroke(); labelLayer = createGraphics(width, height); labelLayer.textAlign(CENTER, CENTER); labelLayer.textSize(16); labelLayer.textFont('sans-serif'); labelLayer.fill(20); for (let label of labels) { cubeSizes[label] = baseSize; targetSizes[label] = baseSize; cubeRotations[label] = 0; targetRotations[label] = 0; flippedState[label] = false; cubeYOffsets[label] = 0; targetYOffsets[label] = 0; } let connectButton = createButton("Connect to Arduino"); connectButton.size(200, 40); connectButton.position((windowWidth - 200) / 2, height + 40); connectButton.class("connect-button"); connectButton.mousePressed(connectToArduino); } function draw() { background(135, 206, 250); camera(0, -100, 600, 0, -100, 0, 0, 1, 0); push(); translate(0, 100, 0); rotateX(HALF_PI); fill(220); plane(3000, 3000); pop(); push(); translate(0, 0, -1000); fill(135, 206, 250); plane(3000, 2000); pop(); ambientLight(100); pointLight(255, 255, 255, 0, -300, 300); directionalLight(200, 200, 200, -0.5, -1, -0.3); let spacing = 120; let totalWidth = spacing * (labels.length - 1); let startX = -totalWidth / 2; for (let i = 0; i < labels.length; i++) { let label = labels[i]; cubeSizes[label] = lerp(cubeSizes[label], targetSizes[label], sizeLerpSpeed); cubeRotations[label] = lerp(cubeRotations[label], targetRotations[label], rotationLerpSpeed); cubeYOffsets[label] = lerp(cubeYOffsets[label], targetYOffsets[label], positionLerpSpeed); let x = startX + i * spacing; let y = -baseSize / 2 + 100 - cubeYOffsets[label]; push(); translate(x, y, 0); rotateX(cubeRotations[label]); fill(0, 102, 204); specularMaterial(0, 102, 204); shininess(20); box(cubeSizes[label]); pop(); } for (let i = particles.length - 1; i >= 0; i--) { particles[i].update(); particles[i].display(); if (particles[i].lifespan <= 0) particles.splice(i, 1); } labelLayer.clear(); for (let i = 0; i < musicNotes.length; i++) { let spacing = 120; let totalWidth = spacing * (labels.length - 1); let x = -totalWidth / 2 + i * spacing; let screenX = width / 2 + x; let screenY = height / 2 + 130; labelLayer.text(musicNotes[i], screenX, screenY); } resetMatrix(); image(labelLayer, 0, 0); } function triggerCube(label) { targetSizes[label] = pulseSize; targetYOffsets[label] = jumpHeight; setTimeout(() => { targetSizes[label] = baseSize; targetYOffsets[label] = 0; }, 150); flippedState[label] = !flippedState[label]; targetRotations[label] = flippedState[label] ? PI : 0; let spacing = 120; let totalWidth = spacing * (labels.length - 1); let x = -totalWidth / 2 + labels.indexOf(label) * spacing; let y = 100 - baseSize; for (let i = 0; i < 15; i++) { particles.push(new Particle(x, y, 0)); } } function keyPressed() { let keyUp = key.toUpperCase(); if (sounds[keyUp]) { sounds[keyUp].play(); triggerCube(keyUp); } } async function readSerial() { while (port.readable) { try { const decoder = new TextDecoderStream(); const inputDone = port.readable.pipeTo(decoder.writable); const inputStream = decoder.readable; const reader = inputStream.getReader(); while (true) { const { value, done } = await reader.read(); if (done) { console.warn(" Serial stream ended. Trying to reconnect..."); reader.releaseLock(); break; } if (value) { const clean = value.trim().toUpperCase(); console.log(" Received from Arduino:", clean); if (sounds[clean]) { sounds[clean].play(); triggerCube(clean); } } } } catch (err) { console.error("❌ Serial read error:", err); break; } await new Promise(resolve => setTimeout(resolve, 1000)); } console.log("❌ Serial not readable anymore."); } async function connectToArduino() { try { port = await navigator.serial.requestPort(); await port.open({ baudRate: 9600 }); readSerial(); console.log("✅ Serial connected"); } catch (err) { console.error("❌ Connection failed:", err); } } class Particle { constructor(x, y, z) { this.pos = createVector(x, y, z); this.vel = createVector(random(-1, 1), random(-3, -1), random(-0.5, 0.5)); this.lifespan = 60; } update() { this.pos.add(this.vel); this.vel.y += 0.05; this.lifespan -= 2; } display() { push(); translate(this.pos.x, this.pos.y, this.pos.z); fill(255, 150, 0, this.lifespan * 4); noStroke(); sphere(5); pop(); } }
Arduino code:
#include <CapacitiveSensor.h> // CapacitiveSensor(sendPin, receivePin) CapacitiveSensor pads[7] = { CapacitiveSensor(2, 4), // A CapacitiveSensor(2, 5), // B CapacitiveSensor(2, 6), // C CapacitiveSensor(2, 7), // D CapacitiveSensor(2, 8), // E CapacitiveSensor(2, 9), // F CapacitiveSensor(2, 10) // G }; const char notes[7] = {'A', 'B', 'C', 'D', 'E', 'F', 'G'}; bool touched[7] = {false}; int ledPins[7] = {A0, A1, A2, A3, A4, A5, 12}; // LED output pins long threshold = 100; // Adjust after testing raw values void setup() { delay(1000); // Allow time for USB/Serial to stabilize Serial.begin(9600); for (int i = 0; i < 7; i++) { pads[i].set_CS_AutocaL_Millis(0xFFFFFFFF); // Disable autocalibration pinMode(ledPins[i], OUTPUT); digitalWrite(ledPins[i], LOW); } } void loop() { for (int i = 0; i < 7; i++) { long reading = pads[i].capacitiveSensor(10); // Reduced samples for speed bool isTouched = (reading > threshold); digitalWrite(ledPins[i], isTouched ? HIGH : LOW); if (isTouched && !touched[i]) { Serial.println(notes[i]); touched[i] = true; } else if (!isTouched && touched[i]) { touched[i] = false; } } // No delay(5); loop runs as fast as possible for lower latency }