Music Producer — Final Project

“Music gives a soul to the universe, wings to the mind, flight to the imagination, and life to everything.” — Plato

Concept

I truly love music. I actually realized, while looking at my past projects, that I have implemented music in many of then as a core element. I think it’s maybe because of my surroundings and how I grew up playing so many instruments, and for my final project, I wanted others to feel this unique connection to music too. I wanted to give a space for users to experiment with sound and visuals so that they would want to create music, which is very different than only listening to it. So, that is how my project was born.

“The only way to do great work is to love what you do.” — Steve Jobs

In general, I wanted to build something that actually feels like a creative tool and not just a demo. The idea is that you are a music producer. You start by recording your own voice, and then you build a whole song on top of it using different instruments like piano, drums, and bass, all controlled through physical buttons connected to an Arduino. Everything you play gets layered and looped, and at the same time the system generates a live visual artwork based on what you are creating.

I wanted to make something where every single action has a visual response, so that by the end you do not just have a song, you have something that looks like a living piece of art. The kind of thing where even if you have no idea how to make music, you can still produce something that feels like yours.

It is designed for anyone who wants to experiment with sound and visuals in a fun and intuitive way. The controls are simple enough that you do not need to be a musician, but expressive enough that if you are, you can actually do something interesting with it.

How It Works

Interaction Design

The whole experience is split into three modes: Piano, Drums, and Bass. In each mode, the four Arduino buttons trigger different sounds, piano notes, drum hits, or bass notes. The potentiometer controls the playback speed of any loops you have selected. A green LED on the Arduino lights up when you are recording and goes red when you stop.

On the screen you have a recordings panel that you can toggle open and close, recording buttons, and mode selectors. The center of the screen always shows a Siri-like wave visualization that responds to everything playing. And depending on which mode you are in, a different kind of art generates in the background as you play. Subtle drifting lines for piano, glowing expanding circles for drums, and thickening colored waves for bass.

The flow is: record your voice, pick a mode, play sounds on top, record the whole thing as a loop, loop it, and keep building on top of that.

Arduino

The Arduino does only one thing and that is reading the physical inputs and sending them to p5.js. It does not make any decisions about what the sounds mean. It sends the state of all four buttons and the potentiometer value every 50 milliseconds, formatted as b1,b2,b3,b4,pot. It also listens for messages back from p5 to control the LEDs. LED:1 turns the green on and LED:0 goes back to red.

I kept the Arduino side as clean and minimal as possible on purpose. So that the user gets that p5.js is the brain and Arduino is just the hands.

You can find the full Arduino source code here

p5.js

The sketch handles literally everything else: the audio system, the visuals, the UI, the serial communication, and the art generation. It has multiple screens (menu, instructions, main), a full recording and playback system using p5.SoundRecorder and p5.SoundFile, an FFT analyzer for the wave visualization, a particle system for the art, and a styled HTML button system injected directly from JavaScript.

You can also see the full p5.js source code here

How This Was Made

The starting point for the audio visualization was the “Sound Visualization: Frequency Analysis with FFT“ video by Daniel Shiffman on The Coding Train. I had never used p5.FFT before and that video really broke it down in a way I could actually follow. From there I understood how fft.waveform() gives you the time domain signal and fft.getEnergy() gives you the overall loudness, which is exactly what I needed to make the wave lines spread based on how loud the audio is. I spent a lot of time with that one and went back to it probably like four or five times throughout the project.

For understanding how p5.SoundRecorder and p5.SoundFile worked together I went back to The Coding Train’s p5.Sound library series starting from episode 11.1. The documentation alone was not enough for me, I needed to see it in action to understand the difference between what the recorder does versus what the SoundFile stores and when it is actually ready to use.

For the HTML buttons inside p5, like styling them with CSS classes and using createButton() and addClass(), I found The Coding Train’s p5.js DOM tutorials really useful. I did not know you could inject a style tag directly from JavaScript in setup and have it affect your buttons in real time. That one I kind of figured out by going through the p5.js reference library and experimenting a lot, but the DOM series gave me the foundation to even know that was possible.

For the particle system, the piano lines, the drum circles, I referenced The Coding Train’s Nature of Code series on particle systems for how to structure a particle with a life, a decay, and properties that change over time. I had seen it before but applying it here in a way that felt musical and not just random took a lot of trial and error honestly.

For the serial communication between Arduino and p5.js I followed what we learned in class.

I also used ChatGPT AI, mainly for debugging. A big portion of the time I spent on this project was debugging, and having something I could explain the problem to and get a clear answer helped a lot. I always made sure I understood what it was suggesting before implementing it, because if I did not understand it I could not fix the next thing that broke on top of it.

What I’m Proud Of

The Mode System

The thing I am most proud of conceptually is how the same four buttons completely change meaning depending on which mode you are in. In piano mode they are notes, in drums mode they are different drum hits, in bass mode they are bass lines. And all of that switches with one click, no lag, no confusion. Building that felt simple in theory but getting the visual art to also switch correctly, and making sure the wave visualization responded differently in bass mode, that took a lot of coordination between different parts of the code. This is the core of the whole interaction design and I think it works really well.

The Recordings Panel

The recordings panel was one of the more technically satisfying things I built. It is a slide-in panel inside the canvas, not a separate screen, where you can see all your recordings, click to play or stop them, double click to rename them, and delete selected ones. What makes it tricky is that all the interaction has to be calculated manually because it is drawn on the canvas and not a real HTML element. So I had to make sure the coordinates in drawRecordingsPanel() matched exactly with the hit detection in mousePressed(), otherwise clicking a recording would trigger the wrong one or nothing at all.

// y has to match exactly between drawing and clicking
let y = 196 + i * 46;
if (mouseX < 252 && mouseY > y - 14 && mouseY < y + 24) {

Also managing the selectedRecordings array, tracking which ones are playing, and deleting them in descending order so the indexes do not shift mid-loop, that took me more debugging than I expected and I was very happy when it finally worked correctly.

The Voice Wave

The wave visualization is honestly the thing I am most proud of technically. The idea is that all 9 lines overlap in the center when it is quiet and spread apart as the audio gets louder. I had to figure out a math system where each line gets a t value from -1 to 1, and the amplitude of that line is abs(t) times baseAmp, so the outermost lines spread the most and the center line never moves. Then I flip the direction for the top half versus the bottom half so they mirror each other like a reflection.

let t   = map(k, 0, N - 1, -1, 1);
let amp = abs(t) * baseAmp;
let dir = t >= 0 ? 1 : -1;

The Art Per Mode

Making every sound trigger a different visual response was something I really cared about from the beginning. For piano it is thin drifting lines that appear across the screen and slowly fade. For drums the circles are sized by drum type, the kick spawns big glowing circles and the hihat spawns tiny ones. For bass the wave lines thicken in the color that belongs to the note you are pressing. None of it is random in a meaningless way, every parameter was chosen to match the feeling of that sound. That intentionality is something I am really proud of.

Areas for Future Improvement

Honestly one of the biggest things missing right now is the ability to export your final song as an audio file. You can loop everything and listen to it but there is no way to actually save what you made and share it with someone, which feels like a pretty big missing piece for a tool that is supposed to be about music production.

I would also love to add volume control per recording. Right now all selected recordings play at the same level, which means if you stack too many loops it gets muddy and hard to hear what is what. Being able to raise or lower individual loops would make it way more usable as an actual production tool.

The piano art could also be a lot more interesting. Right now the lines just drift horizontally and fade, but I wanted them to feel more like a magic keyboard that reacts specifically to which note you are pressing, not just that a note was pressed. I did not get there in time.

And if I had another two weeks I would probably add some kind of BPM sync so that the loops actually stay in time with each other instead of playing freely. Right now if you start two recordings at slightly different moments they drift out of sync pretty fast, which limits how musical the final result can actually be.

Overall I am really proud of this project. It is the most complex thing I have built so far and it actually works the way I imagined it when I first wrote down the concept. That does not always happen so it feels good when it does.

SCHEMATIC

IMAGES

VIDEO DEMO

User Interaction with my Project

For this stage, I had my brother test my project without explaining anything to him beforehand, and I recorded the interaction to see how he would naturally use it.

Overall, he understood most of it pretty quickly. He was able to figure out how to record audio, switch between modes, and use the buttons to trigger sounds. That made me feel like the core idea and interaction design are working well.

One of the main issues he noticed was with the sound samples, especially the piano and drums. They felt delayed when pressing the buttons, which made the system feel unresponsive. I realized this was because the audio files had a bit of silence at the beginning, so I went back and trimmed each sound manually so that they start exactly when the note or drum hit begins. That made a big difference in how responsive everything feels.

Another thing that confused him was the layout. The recordings were too close to the waveform visualization, which made the interface feel messy. Based on that, I moved all the recordings into a separate tab that you can open and close. That made the interface much cleaner and easier to understand.

He also tried to record the piano, drums, and bass directly, which wasn’t something I had implemented yet. At that point, the system only recorded audio from the microphone. This helped me realize that I needed to clearly separate those two types of recording. So I added two different options: one for recording voice through the mic, and another one for recording the full loop so it can repeat.

Another issue was that recordings kept accumulating and there was no way to delete them. He expected that to be possible, so I added a delete function inside the recordings panel.

In terms of understanding the system, everything else was pretty clear. The only thing he didn’t really understand was the potentiometer, mostly because he didn’t know what it was. The instructions explain it, but it’s still something that depends on the user’s familiarity with hardware.

Overall, his feedback was really helpful. It helped me fix small but important details like responsiveness, layout, and clarity.

At this point, the functionality is working really well. For the final version, I want to focus more on the design and make it feel more intentional and artistic, because right now it still looks pretty plain even though the interaction itself is solid.

Full p5js code till now

////// MUSIC PRODUCER FINAL //////

//INPUT FROM ARDUINO
let port;

let button1 = 0;
let button2 = 0;
let button3 = 0;
let button4 = 0;
let potentiometerValue = 0;

//STATE
let screen = "menu";
let mode = "piano";

//AUDIO SYSTEM
let mic, recorder;
let recordings = [];
let selectedRecordings = [];
let recordingNames = [];

// RECORDING STATES
let micRecording = null;
let loopRecording = null;
let isMicRecording = false;
let isLoopRecording = false;
let showRecordings = false;
let newRecordingAdded = false;

//SOUND LIBRARIES 
let pianoSounds = [];
let drumSounds = [];
let bassSounds = [];

//ACTIVE SOUND TRACKING (for button hold)
let activeSounds = {};

// UI ELEMENTS 
let connectButton, instructionsButton, startButton;
let bliseyFont;

// FFT VISUAL
let fft;

//LOAD ASSETS 
function preload() {

  bliseyFont = loadFont("Blisey.otf");

  pianoSounds[0] = loadSound("pianoC.mp3");
  pianoSounds[1] = loadSound("pianoD.mp3");
  pianoSounds[2] = loadSound("pianoE.mp3");
  pianoSounds[3] = loadSound("pianoF.mp3");

  drumSounds[0] = loadSound("kick.mp3");
  drumSounds[1] = loadSound("snare.mp3");
  drumSounds[2] = loadSound("hihat.mp3");
  drumSounds[3] = loadSound("clap.mp3");

  bassSounds[0] = loadSound("Bass1.mp3");
  bassSounds[1] = loadSound("Bass2.mp3");
  bassSounds[2] = loadSound("Bass3.mp3");
  bassSounds[3] = loadSound("Bass4.mp3");
}

//=== SETUP ===
function setup() {
  createCanvas(windowWidth, windowHeight);
  smooth();
  textFont(bliseyFont);

  port = createSerial();

  mic = new p5.AudioIn();
  mic.start();

  recorder = new p5.SoundRecorder();
  recorder.setInput(mic);

  fft = new p5.FFT(0.9, 128);

  setupMenu();
}

//=== MENU ===
function setupMenu() {
  removeAllButtons();

  instructionsButton = createButton("INSTRUCTIONS");
  instructionsButton.position(width/2 - 80, height/2);
  instructionsButton.mousePressed(goToInstructions);

  startButton = createButton("GET STARTED");
  startButton.position(width/2 - 80, height/2 + 50);
  startButton.mousePressed(goToMain);

  connectButton = createButton("CONNECT");
  connectButton.position(20, 20);
  connectButton.mousePressed(() => port.open(9600));
}

//== NAVIGATION ===
function goToInstructions() {
  removeAllButtons();
  screen = "instructions";

  let backButton = createButton("BACK");
  backButton.position(20, height - 60);
  backButton.mousePressed(setupMenu);
}

//=== MAIN INTERFACE ==
function goToMain() {
  removeAllButtons();
  screen = "main";

  // recordings tab toggle
  let recordingsTab = createButton("RECORDINGS");
  recordingsTab.position(20, 100);
  recordingsTab.mousePressed(() => {
    showRecordings = !showRecordings;
    newRecordingAdded = false;
  });

  // mic recording (voice only)
  let micButton = createButton("🎤");
  micButton.position(40, height - 180);
  micButton.size(70, 70);
  micButton.mousePressed(toggleMicRecording);

  // loop recording (full system)
  let loopRec = createButton("REC LOOP");
  loopRec.position(width/2 - 100, height - 100);
  loopRec.size(120, 50);
  loopRec.mousePressed(startLoopRecording);

  let loopStop = createButton("STOP LOOP");
  loopStop.position(width/2 + 40, height - 100);
  loopStop.size(120, 50);
  loopStop.mousePressed(stopLoopRecording);

  // mode selection
  let pianoButton = createButton("🎹");
  pianoButton.position(width - 100, 100);
  pianoButton.size(60, 60);
  pianoButton.mousePressed(() => mode = "piano");

  let drumsButton = createButton("🥁");
  drumsButton.position(width - 100, 180);
  drumsButton.size(60, 60);
  drumsButton.mousePressed(() => mode = "drums");

  let bassButton = createButton("🎸");
  bassButton.position(width - 100, 260);
  bassButton.size(60, 60);
  bassButton.mousePressed(() => mode = "bass");
}

//=== CLEAN BUTTONS ===
function removeAllButtons() {
  selectAll('button').forEach(b => b.remove());
}

//=== SERIAL READ ===
function readSerial() {
  let data = port.readUntil("\n");

  if (data) {
    let values = data.trim().split(",");

    if (values.length == 5) {
      button1 = int(values[0]);
      button2 = int(values[1]);
      button3 = int(values[2]);
      button4 = int(values[3]);
      potentiometerValue = int(values[4]);
    }
  }
}

//=== MIC RECORDING ===
function toggleMicRecording() {
  userStartAudio();

  if (!isMicRecording) {
    micRecording = new p5.SoundFile();
    recorder.record(micRecording);
    isMicRecording = true;
    port.println("LED:1");
  } else {
    recorder.stop();
    recordings.push(micRecording);
    recordingNames.push("Recording " + recordings.length);
    newRecordingAdded = true;
    isMicRecording = false;
    port.println("LED:0");
  }
}

//=== LOOP RECORDING ===
function startLoopRecording() {
  userStartAudio();

  loopRecording = new p5.SoundFile();
  recorder.record(loopRecording);

  isLoopRecording = true;
  port.println("LED:1");
}

function stopLoopRecording() {
  recorder.stop();

  recordings.push(loopRecording);
  recordingNames.push("Recording " + recordings.length);
  newRecordingAdded = true;

  loopRecording.loop(); // auto loop after recording

  isLoopRecording = false;
  port.println("LED:0");
}

//=== MAIN DRAW LOOP ===
function draw() {
  background(0);
  readSerial();

  if (screen === "menu") drawMenu();
  if (screen === "instructions") drawInstructions();
  if (screen === "main") drawMain();
}

//=== MENU DRAW ===
function drawMenu() {
  fill(255);
  textAlign(CENTER);
  textSize(40);
  text("MUSIC PRODUCER", width/2, height/3);
}

//=== INSTRUCTIONS DRAW ===
function drawInstructions() {
  fill(255);
  textAlign(CENTER);
  textSize(18);

  text(
`You are a DJ.

Record your voice and build music on top of it.

• Mic = voice recording
• Loop = full system recording

• Modes:
🎹 piano
🥁 drums
🎸 bass

• Buttons = trigger sounds
• Potentiometer = pitch control
• LED:
Red = idle
Green = recording`,
  width/2,
  height/2
  );
}

//=== MAIN DRAW ===
function drawMain() {
  noStroke();
  fill(255);
  textAlign(CENTER);
  textSize(30);
  text("MODE: " + mode, width/2, 60);

  drawWave();

  // recordings panel
  if (showRecordings) {
    
    noStroke();
    fill(20);
    rect(0, 150, 250, height);
    
    noStroke();
    fill(180);
    textAlign(CENTER);
    textSize(12);
    text("double click to rename a recording", 125, 170);

    textAlign(LEFT);

    for (let i = 0; i < recordings.length; i++) {
      let y = 200 + i * 40;

      fill(selectedRecordings.includes(i) ? "#00ff99" : 255);
      text(recordingNames[i], 20, y);
    }

    // delete button (panel only)
    let deleteX = 200;
    let deleteY = height - 60;
    
    noStroke();
    fill(120);
    rect(deleteX, deleteY, 30, 30, 5);

    noStroke();
    fill(255);
    textAlign(CENTER, CENTER);
    textSize(8);
    text("delete", deleteX + 15, deleteY + 15);
  }

  // new recording indicator
  if (newRecordingAdded) {
    
    fill(255, 200, 0);
    ellipse(140, 115, 10);
  }

  // pitch control for selected recordings
  let speed = map(potentiometerValue, 0, 1023, 0.5, 2);

  for (let i of selectedRecordings) {
    recordings[i].rate(speed);
  }

  handleButtons();
}

//=== AUDIO VISUALIZATION ===
function drawWave() {

  let waveform = fft.waveform();

  noFill();
  strokeWeight(3);

  stroke(255); drawLine(waveform, 0);
  stroke(0,255,200); drawLine(waveform, 40);
  stroke(255,100,100); drawLine(waveform, -40);
  stroke(100,100,255); drawLine(waveform, 80);
}

function drawLine(wave, offset) {

  beginShape();

  for (let i = 0; i < wave.length; i++) {
    let x = map(i, 0, wave.length, 0, width);
    let y = height/2 + wave[i] * 150 + offset;
    curveVertex(x, y);
  }

  endShape();
}

//=== HARDWARE BUTTONS ===
let p1=0,p2=0,p3=0,p4=0;

function handleButtons() {

  handleHold(1, button1);
  handleHold(2, button2);
  handleHold(3, button3);
  handleHold(4, button4);

  p1=button1; p2=button2; p3=button3; p4=button4;
}

//=== SOUND TRIGGER ===
function handleHold(n, state) {

  let sound;

  if (mode === "piano") sound = pianoSounds[n-1];
  if (mode === "drums") sound = drumSounds[n-1];
  if (mode === "bass") sound = bassSounds[n-1];

  if (state && !activeSounds[n]) {
    sound.play();
    activeSounds[n] = true;
  }

  if (!state && activeSounds[n]) {
    activeSounds[n] = false;
  }
}

//=== MOUSE INTERACTION ===
function mousePressed() {

  if (!showRecordings) return;

  for (let i = 0; i < recordings.length; i++) {

    let y = 200 + i * 40;

    if (mouseX < 250 && mouseY > y - 20 && mouseY < y + 20) {

      if (selectedRecordings.includes(i)) {
        recordings[i].stop();
        selectedRecordings = selectedRecordings.filter(r => r !== i);
      } else {
        recordings[i].loop();
        selectedRecordings.push(i);
      }
    }
  }

  // delete button click
  let deleteX = 200;
  let deleteY = height - 60;

  if (
    mouseX > deleteX &&
    mouseX < deleteX + 30 &&
    mouseY > deleteY &&
    mouseY < deleteY + 30
  ) {
    deleteSelected();
  }
}

//=== RENAME RECORDINGS ===
function doubleClicked() {

  if (!showRecordings) return;

  for (let i = 0; i < recordings.length; i++) {

    let y = 200 + i * 40;

    if (mouseX < 250 && mouseY > y - 20 && mouseY < y + 20) {

      let newName = prompt("Rename recording:");
      if (newName) {
        recordingNames[i] = newName;
      }
    }
  }
}

//=== DELETE RECORDINGS ===
function deleteSelected() {

  selectedRecordings.sort((a, b) => b - a);

  for (let index of selectedRecordings) {
    recordings[index].stop();
    recordings.splice(index, 1);
    recordingNames.splice(index, 1);
  }

  selectedRecordings = [];
}

Music Producer Progress

For this stage of the project, I actually went from just having an idea to having something that works, and that honestly changed how I see everything. Before, it all sounded pretty simple in my head, but once I started actually connecting p5.js with Arduino and dealing with real inputs, I realized how much the interaction itself matters. It’s not just about making things work, it’s about how they feel when you use them.

Right now, the core system is already working. I can record audio directly in p5, save multiple recordings, select one and have it loop, and then change its pitch in real time using the potentiometer. The Arduino buttons are also working, and depending on the mode (piano or drums for now), they trigger different sounds. On top of that, the audio is being visualized as a waveform that reacts live, which already makes everything feel more alive. It’s still pretty raw visually, but it already feels like a mini music tool. I’ll include a video demo showing how it works so far, and also the schematic of the Arduino setup.

The hardest part by far was getting p5.js and Arduino to actually connect. At first, nothing worked. The port wouldn’t open, the sketch wouldn’t connect, and I kept getting errors even though the Arduino code itself was fine. I tried different approaches, switching between p5.serialport and web serial, closing and reopening ports, checking permissions, restarting my computer, and even simplifying everything just to test if anything would connect. It took a while, but once it finally worked, everything else became way easier. That was definitely the moment where the project clicked for me.

Another thing that was tricky was making the buttons have different meanings depending on the mode. It sounds simple, but structuring it cleanly took some thought. I ended up designing it so Arduino only sends raw data, and all the logic happens in p5. That separation made the system way more flexible and easier to expand.

One thing I’m actually really proud of is how the mode system works. Instead of having a lot of controls, I have one set of buttons that changes function depending on whether you’re in piano, drums, or later bass mode. It keeps everything simple but still powerful. I also really like the recording system. Each recording gets saved in an array, and you can select it, loop it, deselect it, and change its pitch whenever you want. The fact that the pitch isn’t fixed and is controlled live by the potentiometer makes it feel much more interactive. The waveform visualization using FFT is also something I like, because even though it’s simple, it already makes the connection between sound and visuals feel real.

There’s still a lot to improve though. I still need to add the bass sounds, design the interface better, and make everything feel more intentional visually. Right now it works, but it doesn’t fully feel like a finished experience. I also want to add a clearer instruction/tutorial section, fix some timing issues with the sounds (some feel slightly delayed), and clean up the audio files, especially for piano and drums. Another big thing is allowing instrument sounds to also be recorded and looped, not just the voice. And visually, I want each element (voice, beats, piano, bass) to look different and more meaningful.

Now that the technical foundation is working, I feel way more confident moving forward. Before, I was mostly worried about whether I could even connect everything and make it function. Now that I’ve solved that, I can actually focus on making it feel good to use. My next steps are refining the interaction, improving the visuals so they’re less generic, making the controls more intuitive, and adding more layering so the final compositions feel richer.

SCHEMATIC LINK

VIDEO OF HOW IT’S WORKING

This reading honestly made me rethink a lot about how design and disability are connected, because I always saw design for disability as something more functional, like just solving a problem, but it’s actually way more about perception and culture. What I found really interesting is how a lot of mainstream design actually comes from disability, like the Eames leg splint   influencing modern furniture, which shows that when designers work with real constraints they come up with better ideas. I also liked the idea of “trickle-up” design because it’s so true today, like voice assistants or accessibility tools that everyone uses now. At the same time, the reading made me question the idea of designing for everyone, because when things try to do too much they end up being confusing instead of inclusive. The part about hiding vs showing disability stood out a lot too, because design often tries to make disability invisible, but that can actually reinforce stigma, while things like glasses became something people want to wear. I do agree that simplicity is important, but not just visually, more in terms of making things actually understandable and usable. But I also think the reading pushes a bit too much toward making disability expressive through design, because not everyone wants that, some people just want something that works and doesn’t draw attention. So for me the main idea is that design isn’t neutral, it shapes how we see people, and instead of choosing one approach, it should give people the option to either hide or express their disability depending on what they want.

Concept

My project is called Music Producer, and it’s basically an interactive system where you take something super simple, like your voice saying “hello”, and turn it into a full song and visual artwork.

You start by recording your voice directly in p5.js. After that, everything becomes about experimenting and building on top of it. The idea is that you’re not just editing sound, you’re playing with it. You can change how it sounds, slow it down, make it higher or lower, and then start adding layers like beats, piano, or bass.

What I find interesting is that the same controls can mean different things depending on what mode you’re in. So instead of having a million controls, you have a simple setup that changes function depending on what you’re trying to do. At the same time, everything you add or change shows up visually, so by the end you don’t just have music, you have something that looks like a living artwork.

Mediums

This project uses p5.js and Arduino, but they do very different things.

• p5.js is where everything actually happens. It records the audio, plays it back, lets you edit it, and also creates all the visuals. It also has different “sections” or modes, like pitch, tempo, piano, beats, and bass.
• Arduino is just the controller. It doesn’t deal with sound directly. Instead, it sends inputs using buttons and a potentiometer. The buttons are used to trigger things like adding beats or notes, and the potentiometer controls continuous changes like pitch or tempo.

So basically, p5.js is the brain, and Arduino is the hands.

Process

First, I’m going to make sure I can record and loop audio in p5.js, since that’s the base of everything.

Then I’ll build the Arduino circuit with the buttons, LEDs, and potentiometer, and make sure it sends clean data through serial.

After that, I’ll connect both systems and start mapping inputs. The important part here is that the controls change depending on the mode you’re in. For example:

• in pitch mode, the potentiometer controls pitch
• in tempo mode, it controls speed
• in piano mode, the buttons trigger notes

Then I’ll add different sound layers like beats, bass, and simple melodies.

At the same time, I’ll design the visuals so they match each element:

• voice looks like waveforms moving and changing size
• piano shows points that grow and shrink
• beats look like stars or pulses
• bass looks like slower waves
• different notes have different colors

Finally, I’ll focus on making everything feel smooth, responsive, and easy to understand.

Final Goal and Interaction

The goal is that you can take something super basic, like one word, and turn it into a full composition.

The interaction is very simple but keeps going:

• you record your voice
• you choose a mode
• you use the physical controls to change or add things
• the system responds instantly with sound and visuals
• you keep building

By the end, you have your own song, and at the same time a visual piece that represents everything you created.

Expectations for the Project

I want this project to feel fun and intuitive, not confusing. It should be clear what you’re controlling, and every action should have an immediate result.

I also want people to feel like they’re actually creating something, not just pressing buttons. Even if the input is simple, the output should feel complex and expressive.

Overall, I’m expecting a system where sound, visuals, and interaction all feel connected, and where the user can experiment freely and end up with something that feels like their own piece of art.

Arduino – Harry Potter Music Box

Concept

For this project, I decided to create a mini music pad that can play the Harry Potter theme. The idea came from a small Harry Potter music box that I was gifted when I was younger, since I have always loved Harry Potter and I am a big fan. I also play the piano, so I wanted to design something that felt similar to an instrument and could simulate specific notes like a simplified keyboard.

Process

For the hardware, I used four colored buttons, a potentiometer, and a buzzer to produce sound. I connected each button to a digital pin, the potentiometer to 5V, GND, and analog pin A0, and the buzzer to another digital pin. All components were connected to ground appropriately.

For the coding part, it was a bit challenging at first to find the right notes. I experimented with a basic Arduino buzzer note library and helped myself with Chat GPT to adjust different frequencies until they sounded closer to the melody I wanted. Since the Harry Potter theme has more than four notes, I used the potentiometer as an analog input to change the pitch of the notes. This allowed me to expand the range of sounds using only four buttons. Although it is not perfectly accurate, the result is still very recognizable and captures the essence of the original theme.

// ==============================
// MINI MUSIC PAD - HARRY POTTER SONG
// ==============================


// BUTTON PINS 
int b1 = 2; // E
int b2 = 3; // G
int b3 = 4; // B
int b4 = 5; // C 

// BUZZER 
int buzzer = 8;

// POTENTIOMETER 
int pot = A0;


// NOTES 
#define E4 329
#define G4 392
#define B4 494
#define C5 510


// ==============================
// SETUP
// ==============================
void setup() {

  pinMode(b1, INPUT_PULLUP);
  pinMode(b2, INPUT_PULLUP);
  pinMode(b3, INPUT_PULLUP);
  pinMode(b4, INPUT_PULLUP);

  pinMode(buzzer, OUTPUT);
}


// ==============================
// LOOP
// ==============================
void loop() {

  int potValue = analogRead(pot);

  // HIGH / LOW pitch mode
  bool highPitch = potValue > 512;

  float multiplier;

  if (highPitch) {
    multiplier = 1.3; // slightly higher pitch
  } else {
    multiplier = 0.85; // slightly lower pitch
  }

  if (digitalRead(b1) == LOW) {
    play(E4, multiplier);
  }

  else if (digitalRead(b2) == LOW) {
    play(G4, multiplier);
  }

  else if (digitalRead(b3) == LOW) {
    play(B4, multiplier);
  }

  else if (digitalRead(b4) == LOW) {
    play(C5, multiplier);
  }

  else {
    noTone(buzzer);
  }
}


// ==============================
// FUNCTION
// ==============================
void play(int note, float mult) {
  tone(buzzer, note * mult);
}

Thing That I’m Proud Of

One part of the project I am especially proud of is how I used the potentiometer to control the pitch. This idea allowed me to overcome the limitation of having only four buttons. Initially, I tried pressing multiple buttons at once to create different sounds, but it did not work as expected. Then I realized I could use the potentiometer to dynamically change the pitch, which was a much more effective solution. I am also proud of the overall design of my mini music pad, as it is intuitive to use and much cleaner than my earlier versions, where the wires were more disorganized.

// HIGH / LOW pitch mode
  bool highPitch = potValue > 512;

  float multiplier;

  if (highPitch) {
    multiplier = 1.3; // slightly higher pitch
  } else {
    multiplier = 0.85; // slightly lower pitch
  }

Overall Reflection

Overall, I really enjoyed this project. It allowed me to combine something personal, like my love for Harry Potter and music, with technical skills such as coding and circuit design. The process involved a lot of experimentation and problem-solving, especially when trying to match the melody. This project also made me feel nostalgic, since the song reminds me of my childhood. I am very happy with the final result and how I was able to turn a simple Arduino setup into an interactive musical experience.

SCHEMATIC

VIDEO

Reading Reflections

Reading A Brief Rant on the Future of Interaction Design and the Response honestly changed the way I see technology in a way I did not expect at all. I agree with the author, but what surprised me the most is that I had never even realized this was a problem in the first place. It is not that our devices are bad, but more that they are limiting us from using our full potential, especially when it comes to how we physically interact with the world.

One thing that really stuck with me was when I started thinking about the idea of using devices with your eyes closed. I asked myself what I could actually do like that, and the only thing I could come up with was typing on my laptop keyboard. And then I realized that typing is one of the only truly tactile interactions we still have, and it is not even part of the screen itself. Everything else, especially touchscreens, feels like just sliding your finger on glass with almost no physical feedback. The reading describes this as “Pictures Under Glass,” and I think that is such a perfect way to put it, because it really does feel disconnected from what you are doing.

It also made me think about why we keep simplifying technology more and more. At first it seems like progress, because everything becomes easier and more accessible, but at the same time, it feels like we are reducing the way we interact with the world to something very basic, almost like we are being treated as toddlers. And that idea really stayed with me, because I had never questioned it before. I never thought that maybe we actually deserve more complex and richer ways of interacting, instead of everything being reduced to tapping and swiping.

I also kept thinking about the comparison between the real world and digital interfaces. In real life, our hands are constantly feeling, adjusting, reacting. The reading gives examples like holding a book or a glass and understanding things like weight and position without even thinking about it. That made me realize how much information we are losing when everything becomes flat and two dimensional. It made me wonder if we are actually moving forward or backward by making everything more and more screen based.

At the same time, I thought about things like movie theaters. It took so long for us to even be able to record and display images, but now we have 3D, 4D, and experiences where you can feel water or heat from what is happening on screen. That feels like a step toward engaging more of our senses. But then I started thinking, what if we did not need glasses to see in 3D, or what if devices could actually communicate through touch in a more real way? Something closer to real life, like materials that change shape or give feedback. That is kind of what the author is pushing for with the idea of a “dynamic medium that we can see, feel, and manipulate.”

At some point, though, I also started questioning how far this can really go. If we keep trying to make technology more and more like real life, are we just trying to recreate reality itself? Because the real world is already the most advanced “interface” we have. So it made me think about where the limit is, and whether the goal should be to replicate reality or to create something entirely new.

Overall, this reading really opened my mind. It made me realize that interaction design is not just about making things look nice or easy to use, but about deeply understanding human capabilities and not ignoring them. I had never questioned touchscreens before, but now I cannot stop thinking about how much more is possible and how much we might be missing out on by staying with what we have.

Production: LED Lights Memory Game

Concept

The concept of my project was to create a memory-based LED game inspired by the small game we watched in class. In that example, a light would turn on and the first person to press the button on their side would win. I found that idea very engaging, so I wanted to build something similar but with a stronger focus on memory and sequence. While looking for ideas, I came across the concept of a “Simon Says” style memory game, which immediately reminded me of a toy I used to play with as a child, similar to a Pop It memory game with lights and patterns. This connection made the idea more personal and interesting to develop, so I decided to create a game where the Arduino generates a sequence of LED lights and the player must repeat it correctly using buttons.

Process

To build this project, I started by learning the basic components separately. I watched several tutorials by Paul McWhorter, which helped me understand more things on how LEDs and buttons can work, as well as how to structure Arduino code using functions, loops, and conditionals. I also explored the Arduino documentation to better understand how different functions work and how Arduino programming compares to tools like p5.js. The good thing is that both use similar logical structures, which made it easier for me to understand the code. I programmed the Arduino to generate a random sequence, display it using LEDs, and then check the user’s input through buttons. I also added a startup LED test to verify that the circuit was working correctly before the game begins.

// LED MEMORY GAME


// PIN SETUP

// LED pins
int redLED = 12;
int yellowLED = 9;
int greenLED = 6;
int blueLED = 3;

// Button pins
int redButton = 11;
int yellowButton = 8;
int greenButton = 5;
int blueButton = 2;


// GAME VARIABLES
int sequence[10];   // stores up to 10 steps
int level = 0;      // current level


// ==============================
// SETUP 
// ==============================
void setup() {

  // Set LED pins as OUTPUT
  pinMode(redLED, OUTPUT);
  pinMode(yellowLED, OUTPUT);
  pinMode(greenLED, OUTPUT);
  pinMode(blueLED, OUTPUT);

  // Set button pins as INPUT (with internal pull-up)
  pinMode(redButton, INPUT_PULLUP);
  pinMode(yellowButton, INPUT_PULLUP);
  pinMode(greenButton, INPUT_PULLUP);
  pinMode(blueButton, INPUT_PULLUP);

  // Start random generator
  randomSeed(analogRead(0));


  // ==============================
  // STARTUP LED TEST
  // ==============================

  // Turn on each LED one by one
  digitalWrite(redLED, HIGH);
  delay(300);
  digitalWrite(redLED, LOW);

  digitalWrite(yellowLED, HIGH);
  delay(300);
  digitalWrite(yellowLED, LOW);

  digitalWrite(greenLED, HIGH);
  delay(300);
  digitalWrite(greenLED, LOW);

  digitalWrite(blueLED, HIGH);
  delay(300);
  digitalWrite(blueLED, LOW);

  delay(500); // small pause before game starts
}


// ==============================
// MAIN LOOP 
// ==============================
void loop() {

  addStep();        // add new random color
  showSequence();   // show LED sequence

  bool correct = checkPlayer(); // check user input

  if (correct == false) {
    gameOver();     // if wrong → game over
    resetGame();    // restart
  }

  delay(1000);
}


// ==============================
// FUNCTIONS
// ==============================


// Add a new random step
void addStep() {
  sequence[level] = random(0, 4);
  level++;
}


// Show the sequence using LEDs
void showSequence() {

  for (int i = 0; i < level; i++) {

    turnOnLED(sequence[i]);
    delay(500);

    turnOffAll();
    delay(250);
  }
}


// Turn on the correct LED
void turnOnLED(int number) {

  if (number == 0) digitalWrite(redLED, HIGH);
  if (number == 1) digitalWrite(yellowLED, HIGH);
  if (number == 2) digitalWrite(greenLED, HIGH);
  if (number == 3) digitalWrite(blueLED, HIGH);
}


// Turn off all LEDs
void turnOffAll() {

  digitalWrite(redLED, LOW);
  digitalWrite(yellowLED, LOW);
  digitalWrite(greenLED, LOW);
  digitalWrite(blueLED, LOW);
}


// Check player input
bool checkPlayer() {

  for (int i = 0; i < level; i++) {

    int buttonPressed = waitForButton();

    // Show feedback
    turnOnLED(buttonPressed);
    delay(300);
    turnOffAll();

    // Check correctness
    if (buttonPressed != sequence[i]) {
      return false;
    }
  }

  return true;
}


// Wait for button press
int waitForButton() {

  while (true) {

    if (digitalRead(redButton) == LOW) {
      delay(200); // debounce
      return 0;
    }

    if (digitalRead(yellowButton) == LOW) {
      delay(200);
      return 1;
    }

    if (digitalRead(greenButton) == LOW) {
      delay(200);
      return 2;
    }

    if (digitalRead(blueButton) == LOW) {
      delay(200);
      return 3;
    }
  }
}


// Game over animation
void gameOver() {

  for (int i = 0; i < 3; i++) {

    digitalWrite(redLED, HIGH);
    digitalWrite(yellowLED, HIGH);
    digitalWrite(greenLED, HIGH);
    digitalWrite(blueLED, HIGH);

    delay(300);

    turnOffAll();

    delay(300);
  }
}


// Reset game
void resetGame() {
  level = 0;
}

Difficulties

One of the main difficulties I encountered was related to the physical design of the circuit, especially the buttons. It was challenging to make the buttons clearly visible and easy to press during gameplay. At times, the wires made the setup look cluttered and slightly uncomfortable to use. To improve this, I adjusted the layout of the breadboard and repositioned it to the left side instead of the right, which made the buttons more accessible. However, I still believe the design could be improved. A possible solution would be to use shorter wires, since the ones included in the kit are relatively long and make the circuit less organized.

Thing that I’m Proud Of

One aspect of the project that I am particularly proud of is the feedback system when the player loses. I programmed all the LEDs to blink simultaneously several times, creating a clear and satisfying visual indication that the game is over. This small detail enhances the user experience and makes the game feel more complete and interactive.

Reflection Overall

Overall, I found this project very enjoyable and rewarding. It allowed me to combine creativity with technical skills and to better understand how hardware and software interact. I especially liked how I could take a simple idea and gradually build it into a functional game. If I were to improve this project in the future, I would like to add sound effects, such as tones that play when each LED lights up, to make the experience more immersive. Although I have not yet implemented this feature, it is something I would like to explore next. This project helped me gain confidence in working with Arduino and problem-solving in both coding and physical design.

PHOTO OF PHYSICAL CIRCUIT

PHOTO OF SCHEMATIC

Reading Reflections

Making Interactive Art

Reading this text me rethink what art even is, especially interactive art, because I feel like I’ve always thought of art as something where the artist is trying to say something very specific, and the audience is supposed to understand that meaning. But here it’s kind of the opposite. It’s saying that if you control too much, if you explain too much, you’re actually limiting the experience. And I find that really interesting but also a bit uncomfortable, because it means you’re giving up control over your own work. Like, you create something, but then people might completely misunderstand it, or interact with it in a way you didn’t expect, and that’s supposed to be part of it. I think what stood out to me the most is this idea that the artwork is not the final product, but more like the beginning of a conversation, and the audience kind of finishes it. At the same time, I feel a bit conflicted, because I wonder if that means the artist’s intention doesn’t matter as much anymore, or if it just becomes too vague. But I do like the idea that interaction is more real when it’s not forced, when people figure things out on their own. It makes the experience more personal instead of just following instructions. So I think the main thing I take from this is that good interactive art is not about telling people what to think, but about creating a space where they can think for themselves, even if that means losing some control over what your work becomes.

Physical Computing’s Greatest Hits (and misses)

This article made me realize something that I hadn’t really thought about before, which is how much creativity is not about doing something completely new, but about how you reinterpret things that already exist. At first, I kind of agreed with that instinct of “if it’s already been done, it’s not original,” but reading this made me see that that idea is actually very limiting. The author shows how the same types of projects keep coming back, like gloves or sensors or interactive spaces, but what changes is how people give meaning to them. And I think that’s the part that stood out to me the most, because it’s not really about the technology itself, it’s about the interaction and what it makes people feel or do. At the same time, I also feel a bit conflicted, because some of these projects sound very repetitive or even a bit shallow, like things that look cool but don’t really have depth. So it made me question where the line is between something that is genuinely creative and something that is just visually interesting. I think overall the article is kind of saying that originality doesn’t come from the idea itself, but from the intention behind it, but I’m not sure if I fully agree, because I still feel like at some point things can become too repetitive. But I do like the idea that you shouldn’t immediately discard something just because it’s been done before, because that mindset probably stops a lot of good ideas before they even start.

Norman,“Emotion & Design: Attractive things work better”

In my opinion, the most interesting takeaway from the reading is when it talks about this “gut feeling” we have, like a sixth sense that immediately tells us what we want or do not want to deal with just based on how something looks. It is like a small light turns on in your brain from the beginning, and that already affects how willing you are to interact with something and how you are going to deal with it.

I also found the connection to the debugging video we saw earlier in class really clear. We talked about how when you are stressed or anxious, your thinking becomes very narrow and you get stuck. The reading explains this as “tunnel vision,” where you focus too much on one thing and cannot see other solutions. That might be useful in a survival situation, but for debugging or any kind of creative problem solving, it is the worst mindset to be in. It makes a lot of sense now why working while stressed feels so unproductive, because your brain is literally limiting your ability to think creatively.

Again, what stood out to me the most, especially because I am interested in design, is the idea that emotions happen before conscious thought. Because of course your brain is already reacting to a design before you even process it. I used to think usability and aesthetics were kind of separate, like one is how it works and the other is how it looks, but now I see that they are actually connected. This quote: If something makes you feel good, you are more likely to overlook small issues and just have a better experience overall. Is actually crazy to me, because I think this applies to many thinks, like for example the apple ecosystem. because of the aesthetics of it and how smoothly it runs even when it it loading, you could be waiting more time for something to download in a mac than on a windows laptop and you would feel like the mac is working better just for the aesthetics of it.

In future projects, I want to use design to influence how the user feels from the beginning. Since as the reading shows us, if I can make someone feel calm and comfortable, they will probably think more clearly, be more flexible, and have a much smoother experience using what I create.

Her Code Got Humans on the Moon

Reading about Margaret Hamilton really blew my mind. It is honestly incredible that a woman, and a working mother at that, was the one who made the moon landing possible during a time when women were not even encouraged to go into technical fields. What is even crazier to me is that she was the only one thinking about a backup plan or worst case scenarios. NASA literally told her that error checking code was not necessary because “astronauts were trained to be perfect.” To me, that is such a wild assumption because humans make mistakes all the time, but back then they just ignored that reality.

It actually makes so much sense to me that a mother would think this way. She saw her daughter, Lauren, accidentally crash a simulator by pressing a button that was not supposed to be used during flight, and she immediately understood that users will eventually do something they are not supposed to do. She used that to try to protect the astronauts from themselves, and it is honestly lucky she did. An astronaut on Apollo 8 made that exact same mistake, and Hamilton’s team had to spend nine hours figuring out how to fix it and get them home safely.

This whole idea of debugging and making systems intuitive for the user basically started with her. Before she came along, software was not even included in the official Apollo budget or schedule. She had to fight for her work to be taken seriously, even creating the term “software engineering” so it would be respected like other fields. It is insane to think that this one woman, working in what she called the “Wild West” of coding, helped create an entire field that now impacts the whole world. She did not just help us get to the moon, she helped shape the foundation of the modern digital world.

Something that genuinely drives me crazy (and it’s not in the reading) are the old washing machines that were in dorm laundry rooms. Not even because they’re complicated, but because you literally cannot tell what they are doing. You press start, sometimes it locks, sometimes it doesn’t, sometimes it just stops and you don’t know if it’s broken or thinking. I’ve stood there many times not knowing if I should open it, wait, or restart it. And the worst part is everyone reacts differently, some people keep pressing buttons, some unplug it, some hit it (honestly valid).

But after reading Norman I realized the problem is not that the machine is complicated but it’s that it has no feedback and no signifiers. The interface doesn’t communicate what state the machine is in. Norman explains that good design should make the possible actions discoverable and understandable. But here you don’t know if is it washing? paused? locked error?

The machine technically works, but the interaction fails. The simplest improvement would actually not be adding more buttons but adding better communication. A small progress bar, a timer that updates, or even a message like “Door will unlock in 30 seconds” would fix almost all the confusion. Norman talks about how users shouldn’t need instructions for simple objects, and when they do it means the design is wrong  . A washing machine should not require guessing or trial and error.

This connects a lot to interactive media. When someone opens a sketch or website, they don’t read instructions first. They try to understand it immediately. So discoverability becomes really important. Norman calls this human-centered design, which is designing based on how people actually behave, not how we wish they behaved  .

I realized my own project actually needed these ideas too. For example, at first my visualization was confusing because nothing indicated when the music would start, how to make it stop, and even at the beginning it still takes a bit to load and you don’t know why (even though its because the image and sound is uploading). After thinking about Norman’s principles, the play button became a signifier: it communicates the action you should take. Also the movement of the dots works as feedback. When the music gets louder and the dots expand, the system is telling you your action (playing the song) had an effect.

In interactive media especially, feedback is essential, for example, as Norman explains, users keep pressing elevator buttons when they don’t receive feedback. That’s literally what happens in digital projects too, if nothing responds, users assume it’s broken. So animations, hover effects, and sound reactions are not just decoration, they are communication.

Another idea from the reading is conceptual models. People build a mental explanation of how something works. If the system behaves differently from what they expect, they get confused. My sketch actually depends on this a little I would say: people assume music causes motion, so when the dots pulse with sound, it feels intuitive. They don’t need instructions.

Overall, the reading made me realize interactive art still has to be usable. Even if something is artistic, the viewer should understand how to interact with it. Good design is not just aesthetics, it is communication between the system and the user.