Button Beats!

My final project is a game inspired by a popular game called Piano Tiles called Button Beats. Users are able to physically play the game using physical buttons, while the gameplay is displayed on a laptop screen and recreated in p5js, with some differences like a life powerup through the special golden tile.

Video of gameplay: https://drive.google.com/file/d/1q-BEMe4s6vl2vXgGhi7uOwFdDXdhTPeO/view?usp=sharing

Arduino

My arduino program is in charge of sending all the serial inputs from the push buttons when a player presses a key on the piano to the p5js. This is similar to the musical instrument assignment we did in class except the speaker is not in the arduino but rather music playing on the computer from p5 as long as the player pressed the right key in the right time frame. The arduino code reads the serial input from the button and sends that to p5 as A, B, C, or D depending on which button was clicked. The code for this can be seen below:

const int buttonPins[] = {2, 3, 4, 5};
const char buttonChars[] = {'A', 'B', 'C', 'D'};
const int numButtons = 4;

void setup() {
  Serial.begin(9600);
  for (int i = 0; i < numButtons; i++) {
    pinMode(buttonPins[i], INPUT_PULLUP);
  }
}

#sends the serial of the button pressed as the letter A, B, C, or D 
void loop() {
  for (int i = 0; i < numButtons; i++) {
    if (digitalRead(buttonPins[i]) == LOW) {
      Serial.println(buttonChars[i]);
      delay(200);
    }
  }
}

P5JS

The p5js runs the code for the entire game. It sets up all the game states and all the game logic. The player will see a welcome screen at the start explaining the rules and a way to connect to the Arduino. Once the user presses enter, the game begins, the music starts playing, and the tiles are randomly generated for the user to physically press on the arduino. When a button is pushed on the arduino, the p5js receives this and follows the logic provided in p5js to ensure that there is currently a tile in that lane and marks that as pressed. If not, the player loses a life. Also, if the player fails to press the button before the tile leaves the screen, they lose a life. If a player loses all their lives and makes a mistake, the game goes to the game over screen and gives the player the option to restart. Here are pictures of all the game states:

Throughout the game, there is a chance to gain extra lives if the golden tile appears which has a 2.5% probability of appearing  at any time in the game. The black tiles also appear randomly, but do speed up as the game goes on which increases the game’s difficulty. The score is kept based on how many tiles you pressed before losing all your lives. Here is the link to the game: https://editor.p5js.org/izza.t/full/moFOZkumG

The circuit consists of the 4 push buttons connected to pins 2, 3, 4, and 5 as well as ground. Here is the schematic below:

I’m particularly proud of the way the game handles the inputs and creates a game that looks pretty similar to the original piano tiles itself. It was really fun from a user perspective to be able to use actual buttons to play the game instead of tapping on a screen. The game working and knowing when you’ve missed a tile or when a tile goes off screen is also something I’m pretty proud of.

One thing though is that the tiles are randomly generated and there is only one song. In the future, it would be nice to have it such that the keys are synced up to the song as is in the game Piano Tiles itself as well as options to select different songs.

Concept Overview

“Sketch and Switch” is a modern take on the Etch-A-Sketch, but with a twist, where the user can draw with random colored lines with mechanical knobs, as this version uses Arduino hardware to control a digital canvas through two potentiometers and a button. The potentiometers allow left/right and up/down movement of the drawing point, and a button press toggles drawing mode while randomly altering the line’s color and thickness.

Project Interaction

Video interaction: https://drive.google.com/file/d/1h1HtV-_-JUEKgieFiu1-NDM2Pb2Thfwr/view?usp=sharing

Interaction Design

Users interact with two potentiometers and a button:

• • Left Potentiometer: Controls horizontal (X-axis) movement.
  • Right Potentiometer: Controls vertical (Y-axis) movement.
  • Button: Toggles drawing mode and changes the drawing color randomly.

Arduino Code

const int potX = A0;         // Potentiometer for horizontal movement
const int potY = A1;         // Potentiometer for vertical movement
const int buttonPin = 2;     // Pushbutton for toggling draw mode

int lastButtonState = HIGH;              
unsigned long lastDebounceTime = 0;      // Timestamp of last button state change
unsigned long debounceDelay = 20;        // Debounce time to avoid false triggers

void setup() {
  Serial.begin(115200);                  
  pinMode(buttonPin, INPUT_PULLUP);      
}

void loop() {
  int xVal = analogRead(potX);           // Read horizontal potentiometer value
  int yVal = analogRead(potY);           // Read vertical potentiometer value

  int reading = digitalRead(buttonPin);  
  int btnState = LOW;                    // Default button state to not pressed

  if (reading != lastButtonState) {
    lastDebounceTime = millis();
  }

  if ((millis() - lastDebounceTime) > debounceDelay) {
    btnState = (reading == LOW) ? 1 : 0;  // Set button state (pressed = 1)
  }

  lastButtonState = reading;

  // Send formatted data: x, y, buttonState
  Serial.print(xVal);
  Serial.print(",");
  Serial.print(yVal);
  Serial.print(",");
  Serial.println(btnState);

  delay(20); // Small delay to reduce serial flooding
}

Circuit Schematic

p5.js Code

let port;
let connectButton, disconnectButton, finishButton, startButton, saveButton, statusText;
let xPos = 0;
let yPos = 0;
let drawingEnabled = false;
let isConnected = false;
let prevX, prevY;
let lastButtonState = 0;
let started = false;
let tutorialShown = false;
let currentColor;
let studentImg;
let tutorialButton;

function preload() {
  studentImg = loadImage("Shamsa.PNG"); // preload image for the intro screen
}

function setup() {
  createCanvas(1280, 720); // fixed landscape canvas size
  background("#F5F5DC");

  port = createSerial(); // setup WebSerial port
  currentColor = color(random(255), random(255), random(255)); // initial random color

  // Setup start screen
  startButton = createButton("Start");
  styleButton(startButton);
  positionCenter(startButton, 0, 60);
  startButton.mousePressed(() => {
    startButton.hide();
    tutorialShown = true;
    showTutorialScreen(); // go to tutorial screen before drawing
  });

  statusText = createP("Status: Not connected");
  statusText.position(10, 10);
  statusText.hide(); // hidden until drawing mode begins
}

function styleButton(btn) {
  // Apply consistent style to all buttons
  btn.style("background-color", "#CF877D");
  btn.style("color", "black");
  btn.style("border-radius", "10px");
  btn.style("padding", "10px 15px");
  btn.style("font-size", "14px");
  btn.style("border", "none");
}

function positionCenter(btn, offsetX, offsetY) {
  // Center button horizontally/vertically with optional offset
  btn.position((width - btn.size().width) / 2 + offsetX, (height - btn.size().height) / 2 + offsetY);
}

function showTutorialScreen() {
  clear();
  background("#F5F5DC");

  // Instructions and disclaimer
  textAlign(CENTER);
  fill("#a8423d");
  textSize(32);
  text("Welcome to Sketch & Switch!", width / 2, 80);

  textSize(20);
  fill(0);
  text(
    "Disclaimer:\nThe blue knobs may be difficult at first, so twist them slowly and gently.\n" +
    "The one on the right moves ↑↓, and the one on the left moves ←→",
    width / 2, 160
  );

  text(
    "Instructions:\n1. Press 'Connect' to connect to your drawing device\n2. Twist knobs to move\n" +
    "3. Press the button on the board to change color (it will be randomized)\n" +
    "4. When finishing the drawing, click 'Finish Drawing' to clear it,\n" +
    "   or click 'Save as PNG' to download your art.\n\n Tip: Clockwise = ↑ or →, CounterClockwise = ↓ or ←",
    width / 2, 320
  );

  // Begin actual drawing
  tutorialButton = createButton("Start Drawing");
  styleButton(tutorialButton);
  tutorialButton.position(width / 2 - 70, height - 100);
  tutorialButton.mousePressed(() => {
    tutorialButton.remove();
    clear();
    background(255);
    started = true;
    setupDrawingUI(); // load UI controls for drawing
  });
}

function setupDrawingUI() {
  // Create control buttons
  connectButton = createButton("Connect");
  connectButton.mousePressed(() => {
    if (!port.opened()) {
      port.open("Arduino", 115200); // open WebSerial at 115200 baud
    }
  });
  styleButton(connectButton);

  disconnectButton = createButton("Disconnect");
  disconnectButton.mousePressed(() => {
    if (port.opened()) {
      port.close(); // safely close serial port
    }
  });
  styleButton(disconnectButton);

  finishButton = createButton("Finish Drawing");
  finishButton.mousePressed(() => {
    background(255); // clear canvas
    drawingEnabled = false;
  });
  styleButton(finishButton);

  saveButton = createButton("Save as PNG");
  saveButton.mousePressed(() => {
    saveCanvas("drawing", "png"); // download current canvas
  });
  styleButton(saveButton);

  positionUI(); // arrange buttons
  statusText.show();
}

function positionUI() {
  // Align control buttons along the top
  let baseX = width / 2 - 250;
  let y = 10;
  connectButton.position(baseX, y);
  disconnectButton.position(baseX + 130, y);
  finishButton.position(baseX + 260, y);
  saveButton.position(baseX + 420, y);
}

function draw() {
  if (!started) {
    // Intro screen only if tutorial not yet shown
    if (!tutorialShown) {
      background("#F5F5DC");
      textAlign(CENTER, CENTER);
      textSize(48);
      fill("#a8423d");
      text("Sketch & Switch!", width / 2, height / 2 - 100);

      textSize(24);
      fill(0);
      text("Press Start to Begin", width / 2, height / 2 - 40);

      imageMode(CENTER);
      image(studentImg, width / 4, height / 2 - 30, studentImg.width / 3, studentImg.height / 3);
    }
    return;
  }

  // Serial data handling (reads only once per frame to prevent lag)
  if (port.opened()) {
    isConnected = true;
    statusText.html("Status: Connected");

    let data = port.readUntil("\n");
    if (data && data.trim().length > 0) {
      processSerial(data.trim()); // pass cleaned data to be handled
    }
  } else {
    isConnected = false;
    statusText.html("Status: Not connected");
  }

  // Draw a small dot when not drawing (cursor)
  if (!drawingEnabled && isConnected) {
    fill(currentColor);
    noStroke();
    ellipse(xPos, yPos, 6, 6);
  }
}

function processSerial(data) {
  // Parse "x,y,button" format from Arduino
  let parts = data.split(",");
  if (parts.length !== 3) return;

  let xVal = int(parts[0]);
  let yVal = int(parts[1]);
  let btn = int(parts[2]);

  if (isNaN(xVal) || isNaN(yVal) || isNaN(btn)) return;

  // Map potentiometer values to canvas dimensions
  xPos = map(xVal, 0, 1023, 0, width);
  yPos = map(yVal, 0, 1023, 0, height);

  // Toggle drawing mode when button is pressed
  if (btn === 1 && lastButtonState === 0) {
    drawingEnabled = !drawingEnabled;
    currentColor = color(random(255), random(255), random(255));
    prevX = xPos;
    prevY = yPos;
  }

  // Draw if in drawing mode
  if (drawingEnabled) {
    stroke(currentColor);
    strokeWeight(2);
    line(prevX, prevY, xPos, yPos);
    prevX = xPos;
    prevY = yPos;
  }

  lastButtonState = btn; // update for debounce
}

Arduino and p5.js Communication

Communication is established via serial connection:

• • Arduino: Sends comma-separated values (X, Y, ButtonState) at a set interval.
  • p5.js: Reads incoming serial data, parses the values, and updates the drawing accordingly.

Highlights

I’m proud of the fact that the push button consistently functions to toggle between the drawing modes of randomized thickness and color. Moreover, with optimal baud rate adjustment and data handling, the potentiometers also function with minimal lag, showing smoother and finger movement.

In addition, the project also has clear on-screen instructions and simple0 controls to allow users across any age range to easily become involved, whether they are experts or complete newbies to physical computing.

Areas for Future Improvement

While Sketch & Switch functions smoothly, there’s still plenty of room to build on its foundations:

• • Add a feature where it allow users to position the drawing point before enabling drawing mode, giving them more control over line placement.
  • Adding a color wheel and thickness slider in the UI so users can manually choose colors and line widths, rather than relying solely on randomness.
  • Add an undo button to let users correct mistakes without having to clear the entire canvas.
  • Replace the current components with larger potentiometers and a larger push button for improved tactile feedback and easier control.

For my final project, I created a hybrid digital-physical escape room-style game inspired by timed bomb defusal sequences in video games. The entire experience blends physical interactivity through Arduino with dynamic visuals and logic in p5.js. The result is a tense, fast-paced game where players must race against the clock to complete four distinct challenges and successfully input a final disarm code before the device “explodes.”

The core idea was to simulate a bomb defusal setup using varied mini-games—each one testing a different skill: speed, logic, memory, and pattern recognition.

How the Game Works

The digital game runs in p5.js and connects to a single Arduino board that handles inputs from physical buttons, rotary encoders, and switches. Players are given 80 seconds to complete four escalating stages:

1. Button Mash Challenge – Tap a physical button repeatedly until a counter hits the target.

2. Math Riddle Quiz – Use a button to select answers and confirm; one wrong answer ends the game.

3. Note Match – Listen to a played note and match it using one of four physical options.

4. Morse Code Challenge – Decode a flashing Morse signal and reproduce it using a button.

After all four stages, the player must recall and enter a 4-digit code, derived from the hidden logic behind each stage. If they enter it correctly, the bomb is defused and a green screen confirms success. Otherwise—boom. Game over.

A countdown timer runs persistently, and sound effects, animations, and images change based on player progress, creating an immersive narrative.

Video Demo:  https://drive.google.com/drive/folders/1xghtShbdS5ApygD3-LrT41DRQsbnQ98U?usp=share_link

Hardware & Physical Build

This game relies heavily on Arduino to provide tactile interaction. Here’s how each component contributes:

• Button Mash: A simple digital input wired to a button switch.

• Math Quiz: A designated button allows users to scroll through numeric answers, with a button to lock in their choice.

• Note Match: A speaker plays a pitch generated from Arduino, and players must select the correct note using four distinct buttons.

• Morse Code: The p5 screen shows a pattern, which the player must replicate with button presses (dots and dashes).

To enhance the look, I created screen graphics for each stage and embedded them as assets into the p5.js sketch. I also used audio cues (success/failure sounds) to give it more feedback.

Code + Serial Integration

The p5.js sketch acts as the game engine and controller, managing state transitions, timing, visuals, and logic. Arduino handles all the physical input and sends data to p5.js over serial using a consistent message format.

Initially, I experimented with sending raw characters for stage signals and player responses, but ran into reliability issues. Eventually, I switched to using numeric values and simple prefixes which made parsing much more predictable.

There’s a small but critical serial timing issue to manage — making sure Arduino doesn’t flood the buffer, and that p5 reads and trims data consistently. I handled this using readUntil("\n").trim() on the p5 side and line breaks on the Arduino side.

I also implemented a game reset trigger — pressing “R” after the game ends resets both the p5 and Arduino states and lets the player start over without refreshing the browser.

Arduino Code:

// initialize connections
const int BUTTON_PINS[] = {2, 3, 4, 5}; 
const int BUZZER_PIN = 8;
const int CONFIRM_PIN = 7;
const int POT_PIN = A0;
const int RED_LED_PIN_1 = 9;
const int YELLOW_LED_PIN_1 = 10;
const int GREEN_LED_PIN_1 = 12;
const int RED_LED_PIN_2 = 13;
const int YELLOW_LED_PIN_2 = 11;
const int GREEN_LED_PIN_2 = 6;

// initialize all game variables
int currentPressed = -1;
int targetNote = -1;
bool newRound = true;
bool morsePressed = false;
unsigned long morseStart = 0;
int buttonMashCount = 0;
int currentGame = 0;

bool bombDefused = false;
bool bombExploded = false;
bool gameEnded = false;

unsigned long gameStartTime;
const unsigned long GAME_DURATION = 80000; 

bool inCountdown = false;
unsigned long lastBeepTime = 0;
unsigned long beepInterval = 1000;

int blinkState = 0; 
unsigned long lastBlinkTime = 0;
unsigned long blinkInterval = 400;
bool ledOn = false;

void setup() {
  Serial.begin(9600);
  while (!Serial);
  // setup and initialize all physical connections
  for (int i = 0; i < 4; i++) pinMode(BUTTON_PINS[i], INPUT_PULLUP);
  pinMode(BUZZER_PIN, OUTPUT);
  pinMode(CONFIRM_PIN, INPUT_PULLUP);
  pinMode(POT_PIN, INPUT);
  pinMode(RED_LED_PIN_1, OUTPUT);
  pinMode(YELLOW_LED_PIN_1, OUTPUT);
  pinMode(GREEN_LED_PIN_1, OUTPUT);
  pinMode(RED_LED_PIN_2, OUTPUT);
  pinMode(YELLOW_LED_PIN_2, OUTPUT);
  pinMode(GREEN_LED_PIN_2, OUTPUT);

  randomSeed(analogRead(A1));
  gameStartTime = millis();
}

void loop() {
  if (Serial.available()) {
    String input = Serial.readStringUntil('\n');
    input.trim();

    if (input == "RESET") {
      resetGame(); // reset game variables when reset command is received
      return;
    }
    // to identify target note sent by arduino
    if (input.startsWith("NOTE:")) {
      targetNote = input.substring(5).toInt();  // parses 6th character which holds numeric value
      newRound = true;
      return;
    }
    // if bomb is defused on p5, it sends input to arduino and bomb is defused here as well
    if (input == "DEFUSED"){
      bombDefused= true; 
      gameEnded = true;
      return;
    }
    // in case user makes a mistake in games, p5 sends exploded to arduino
    if (input == "EXPLODED") {
      bombExploded = true;
      gameEnded = true;
      Serial.println("EXPLOSION_ACK");
      return;
    }
    // to parse game sent to arduino each time a challenge is completed and we move to next one
    currentGame = input.toInt();

    if (currentGame == 0) buttonMashCount = 0;
    if (currentGame == 2) newRound = true;
  }
  // when bomb is defused or explodes
  if (gameEnded) {
    noTone(BUZZER_PIN);
    return;
  }
  // turn of all leds 
  if (bombExploded || bombDefused) {
    digitalWrite(RED_LED_PIN_1, LOW);
    digitalWrite(YELLOW_LED_PIN_1, LOW);
    digitalWrite(GREEN_LED_PIN_1, LOW);
    digitalWrite(RED_LED_PIN_2, LOW);
    digitalWrite(YELLOW_LED_PIN_2, LOW);
    digitalWrite(GREEN_LED_PIN_2, LOW);

    noTone(BUZZER_PIN);
  }

  unsigned long elapsed = millis() - gameStartTime;
  // handles blinking of leds alternatively until 30 seconds are left 
  unsigned long remaining = GAME_DURATION - elapsed;
  if (!gameEnded && !bombDefused) {
  if (remaining > 30000) {
    if (millis() - lastBlinkTime >= 400) {
      lastBlinkTime = millis();
      ledOn = !ledOn;

      digitalWrite(RED_LED_PIN_1, LOW);
      digitalWrite(YELLOW_LED_PIN_1, LOW);
      digitalWrite(GREEN_LED_PIN_1, LOW);
      digitalWrite(RED_LED_PIN_2, LOW);
      digitalWrite(YELLOW_LED_PIN_2, LOW);
      digitalWrite(GREEN_LED_PIN_2, LOW);


      if (ledOn) {
        if (blinkState == 0) {
          digitalWrite(GREEN_LED_PIN_1, HIGH);
          digitalWrite(GREEN_LED_PIN_2, HIGH);}
        else if (blinkState == 1) {
          digitalWrite(YELLOW_LED_PIN_1, HIGH);
          digitalWrite(YELLOW_LED_PIN_2, HIGH);}
        else if (blinkState == 2) {
          digitalWrite(RED_LED_PIN_1, HIGH);
          digitalWrite(RED_LED_PIN_2, HIGH);}

        blinkState = (blinkState + 1) % 3;
      }
    }
  }
  // last 30 seconds yellow starts blibking with beeps
  else if (remaining > 13000) {
    if (millis() - lastBlinkTime >= 500) {
      lastBlinkTime = millis();
      ledOn = !ledOn;

      //ensure other LEDs are off
      digitalWrite(RED_LED_PIN_1, LOW);
      digitalWrite(RED_LED_PIN_2, LOW);
      digitalWrite(GREEN_LED_PIN_1, LOW);
      digitalWrite(GREEN_LED_PIN_2, LOW);

      // Yellow blinking
      digitalWrite(YELLOW_LED_PIN_1, ledOn ? HIGH : LOW);
      digitalWrite(YELLOW_LED_PIN_2, ledOn ? HIGH : LOW);
    }
    // beeps
    if (millis() - lastBeepTime >= 1000) {
      lastBeepTime = millis();
      tone(BUZZER_PIN, 1000, 100);
    }
  }
  // last 10 seconds red is blinking with faster beeps
  else if (remaining > 3000) {
    if (millis() - lastBlinkTime >= 300) {
      lastBlinkTime = millis();
      ledOn = !ledOn;
      digitalWrite(RED_LED_PIN_1, ledOn ? HIGH : LOW);
      digitalWrite(RED_LED_PIN_2, ledOn ? HIGH : LOW);
    }

    if (millis() - lastBeepTime >= 500) {
      lastBeepTime = millis();
      tone(BUZZER_PIN, 1200, 100);
    }
  }
}
  // bomb exploded cause time is up
  if (elapsed >= GAME_DURATION && !bombDefused) {
    bombExploded = true;
    gameEnded = true;
    Serial.println("EXPLODED");
    return;
  }
  // Serial input
  

  switch (currentGame) {
    case 0: handleButtonMash(); break;
    case 1: handleMathQuiz(); break;
    case 2: handleNoteMatch(); break;
    case 3: handleMorseCode(); break;
  }
}
// to handle physicak input for each game
void handleButtonMash() {
  static unsigned long lastPressTime = 0;
  static bool lastButtonState = HIGH;
  bool currentState = digitalRead(CONFIRM_PIN);
  // each press sends 1 to p5, which increments counter for current presses 
  if (lastButtonState == HIGH && currentState == LOW && millis() - lastPressTime > 200) {
    buttonMashCount++;
    lastPressTime = millis();
    Serial.println("1");
  }
  lastButtonState = currentState;
}
// resetGame function defined to reset all game variables and start game again
void resetGame() {
  bombDefused = false;
  bombExploded = false;
  gameEnded = false;
  buttonMashCount = 0;
  currentPressed = -1;
  currentGame = 0;
  newRound = true;
  morsePressed = false;
  targetNote = -1;
  morseStart = 0;
  gameStartTime = millis();
  ledOn = false;
  blinkState = 0;
  lastBlinkTime = 0;
  lastBeepTime = 0;
  
  // turn off all LEDs and buzzer
  digitalWrite(RED_LED_PIN_1, LOW);
  digitalWrite(YELLOW_LED_PIN_1, LOW);
  digitalWrite(GREEN_LED_PIN_1, LOW);
  digitalWrite(RED_LED_PIN_2, LOW);
  digitalWrite(YELLOW_LED_PIN_2, LOW);
  digitalWrite(GREEN_LED_PIN_2, LOW);
  noTone(BUZZER_PIN);
}

void handleMathQuiz() {
  static int selectedNum = 0;
  static int lastButtonState = HIGH;
  static unsigned long lastDebounceTime = 0;
  const unsigned long debounceDelay = 200;

  int currentState = digitalRead(BUTTON_PINS[0]); // increment button on pin 2

  // handle incrementing selected number
  if (lastButtonState == HIGH && currentState == LOW && (millis() - lastDebounceTime > debounceDelay)) {
    selectedNum = (selectedNum + 1) % 10;
    Serial.print("SELECT:");
    Serial.println(selectedNum);
    lastDebounceTime = millis();
  }
  lastButtonState = currentState;

  // handle confirmation
  if (digitalRead(CONFIRM_PIN) == LOW) {
    delay(50); 
    // sends selected number to arduino when confirm button is pressed 
    if (digitalRead(CONFIRM_PIN) == LOW) {
      Serial.print("PRESS:");
      Serial.println(selectedNum);
      delay(300); 
    }
  }
}

void handleNoteMatch() {
  static unsigned long toneEndTime = 0;
  static bool isPlayingTarget = false;

  // handle new round target note
  if (newRound) {
    noTone(BUZZER_PIN);
    delay(5);
    digitalWrite(BUZZER_PIN, LOW);
    // plays target note sent by p5
    tone(BUZZER_PIN, getPitch(targetNote), 500);
    toneEndTime = millis() + 500;
    isPlayingTarget = true;
    newRound = false;
    return;
  }

  // handle tone playing completion
  if (isPlayingTarget && millis() > toneEndTime) {
    noTone(BUZZER_PIN);
    isPlayingTarget = false;
  }

  // playing note corresponding to button presses 
  if (!isPlayingTarget) {
    bool anyPressed = false;
    for (int i = 0; i < 4; i++) {
      if (digitalRead(BUTTON_PINS[i]) == LOW) {
        anyPressed = true;
        if (currentPressed != i) {
          noTone(BUZZER_PIN);
          delay(5);
          currentPressed = i;
          tone(BUZZER_PIN, getPitch(i));
          Serial.println(i);
        }
        break;
      }
    }
    // no note should play when button is not pressed
    if (!anyPressed && currentPressed != -1) {
      noTone(BUZZER_PIN);
      currentPressed = -1;
    }
    // send final answer confirmation when button is pressed
    if (digitalRead(CONFIRM_PIN) == LOW) {
      noTone(BUZZER_PIN);
      Serial.println("CONFIRM");
      delay(300);
      newRound = true;
    }
  }
}

void handleMorseCode() {
  static unsigned long lastDebounceTime = 0;
  const unsigned long debounceDelay = 50; // ms
  
  int btn = digitalRead(BUTTON_PINS[0]);  // button on pin 2 is used for sending data
  
  // Button press detection with debouncing
  if (btn == LOW && !morsePressed && (millis() - lastDebounceTime) > debounceDelay) {
    morseStart = millis();
    morsePressed = true;
    lastDebounceTime = millis();
  }
  
  // Button release detection
  if (btn == HIGH && morsePressed) {
    unsigned long duration = millis() - morseStart;
    morsePressed = false;
    
    // short press sends . and long press sends -
    if (duration >= 20) {
      Serial.println(duration < 500 ? "." : "-");
    }
    lastDebounceTime = millis();
    delay(100); 
  }

  // pressing confirm button sends confirm to p5 which then checks if string formed by user matches morse code proivded
  if (digitalRead(CONFIRM_PIN) == LOW) {
    delay(50); // Debounce
    if (digitalRead(CONFIRM_PIN) == LOW) { 
      Serial.println("CONFIRM");
      while(digitalRead(CONFIRM_PIN) == LOW); 
      delay(300);
    }
  }
}
// 4 notes chosen for note match 
int getPitch(int index) {
  int pitches[] = {262, 294, 330, 349};
  return pitches[index];
}

p5js code:

let port;
let connectBtn;
let startBtn;
let baudrate = 9600;

// initiate all flags required
let showWelcome = true;
let showInstructions = false;
let gameStarted = false;
let currentGame = 0;
let gameCompleted = [false, false, false, false];
let codeDigits = [];
let userCodeInput = "";
let correctCode = "";
let bombDefused = false;
let bombExploded = false;
let stageCompleted = false;
let stageCompleteTime = 0;
let stageDigit = -1;
let imgWelcome, imgInstructions, imgButtonSmash, buttonMashSuccessImg, mathQuizSuccessImg, noteMatchSuccessImg, morseCodeSuccessImg, imgMathRiddle, imgNoteMatch, imgMorseCode1,imgMorseCode2, imgBombDefused, imgBombExploded,imgCodeEntry;
let bombSound;
let playedExplosionSound = false;
let successSound;
let playedSuccessSound = false;

// initiate all game variables
let totalTime = 80;
let startTime;

let pressCount = 0;
let targetPresses = 30;
let challengeActive = false;

let selectedNumber = 0;
let correctAnswer = 5;  
let mathAnswered = false;
let feedback = "";

let currentSelection = -1;
let lockedIn = false;
let noteMessage = "";
let noteAnswerIndex = 0;

let morseCode = "";
let userInput = "";
let roundActive = false;
let showSuccess = false;
let showFailure = false;

function preload() {
  imgWelcome = loadImage("start.png");
  imgInstructions = loadImage("instructions.png");
  imgButtonSmash = loadImage("button_smash.png");
  buttonMashSuccessImg = loadImage("stage1_success.png");
  mathQuizSuccessImg = loadImage("stage2_success.png");
  noteMatchSuccessImg = loadImage("stage3_success.png");
  morseCodeSuccessImg = loadImage("stage4_success.png");
  imgMathRiddle = loadImage("math_riddle.png");
  imgNoteMatch = loadImage("note_match.png");
  imgMorseCode1 = loadImage("morse_code1.png");
  imgMorseCode2 = loadImage("morse_code2.png");
  imgBombDefused = loadImage("defused.png");
  imgBombExploded = loadImage("exploded.png");
  bombSound = loadSound('bomb.mp3');
  successSound = loadSound('success.mp3');
  imgCodeEntry = loadImage("code_entry.png")
}

function setup() {
  createCanvas(600, 600);
  textAlign(CENTER, CENTER);

  port = createSerial();

}

function startGame() {
  startTime = millis(); // Set the start time for the timer
  gameStarted = true; // Set the flag to start the game
  currentGame = 0; // Set the current game to 0
  sendGameSwitch(0); // Send game switch signal to Arduino
  startButtonMashChallenge(); // Start the Button Mash Challenge
}

function draw() {
  background(220);
  
  // displays screen for when bomb is defused along with sound effects
  if (bombDefused) {
      image(imgBombDefused, 0, 0, width, height);
    if (!playedSuccessSound) {
        successSound.play();
        playedSuccessSound = true;
      }
      return;
    }
  // displays screen for when bomb is exploded along with sound effects
  if (bombExploded) {
      image(imgBombExploded, 0, 0, width, height);
      if (!playedExplosionSound) {
        bombSound.play();
        playedExplosionSound = true;
      }
      return;
    }

  // Welcome Screen display
  if (showWelcome) {
    image(imgWelcome, 0, 0, width, height);
    return;
  }

  //Instructions Screen display
  if (showInstructions) {
    image(imgInstructions, 0, 0, width, height);
    return;
  }
  // calculates time to keep track of explosion and so on
  let elapsed = int((millis() - startTime) / 1000);
  let remaining = max(0, totalTime - elapsed);
  
  // if time runs out bomb is exploded
  if (remaining <= 0 && !bombDefused) {
    bombExploded = true;
    return;
  }

  // handle all incoming data by reading and sending to function after trimming
  if (port.opened() && port.available() > 0) {
    let data = port.readUntil("\n").trim();
    if (data.length > 0) {
      handleSerialData(data);
    }
  }

  // toggles success screens for all games 
  if (stageCompleted) {
  switch (currentGame) {
    case 0:
      // Show success screen for Button Mash
      image(buttonMashSuccessImg, 0, 0, width, height);
      break;
    case 1:
      // Show success screen for Math Quiz
      image(mathQuizSuccessImg, 0, 0, width, height);
      break;
    case 2:
      // Show success screen for Note Match
      image(noteMatchSuccessImg, 0, 0, width, height);
      break;
    case 3:
      // Show success screen for Morse Code
      image(morseCodeSuccessImg, 0, 0, width, height);
      break;
  }
    // removes success screen afte 3 seconds and moves onto next game
    if (millis() - stageCompleteTime > 3000) {
      codeDigits.push(stageDigit);
      currentGame++;
      sendGameSwitch(currentGame);
      stageCompleted = false;

      // start the next game
      switch (currentGame) {
        case 1: startMathQuiz(); break;
        case 2: startNoteMatchChallenge(); break;
        case 3: startMorseCodeChallenge(); break;
        case 4: correctCode = "4297"; break; 
      }
    }
    return;
  }

  // display game screens using functions defined 
  switch (currentGame) {
    case 0: drawButtonMashChallenge(); break;
    case 1: drawMathQuiz(); break;
    case 2: drawNoteMatchChallenge(); break;
    case 3: drawMorseCodeChallenge(); break;
    case 4: 
      correctCode = "4297";
      if (userCodeInput === "") startCodeEntry();
      drawCodeEntry();
      break;

  }
  // timer display at top of screen
  if (gameStarted && !bombDefused && !bombExploded) {
    textSize(20);
    fill(0);
    textAlign(CENTER, TOP);
    text("Time Remaining: " + remaining + "s", width / 2, 20);
  }

}

function handleSerialData(data) {
  
  if (bombDefused) {
    return; // no data should be handled if bomb has been defused 
  }
  // stop handing data once bomb explodes
  if (data === "EXPLODED") {
    bombExploded = true;
    if (port.opened()) {
      port.write("EXPLODED\n");
    }
    return;
  }
  switch (currentGame) {
    case 0:
      if (data === "1" && challengeActive) {
        pressCount++;
        //  checks success condition, when user presses button 30 times
        if (pressCount >= targetPresses) {
          challengeActive = false;
          // handle necessary flags for this stage and keep track of time for success screen display
          stageDigit = 4;  
          gameCompleted[0] = true;
          stageCompleted = true;
          stageCompleteTime = millis();
        }
      }
      break;

    case 1:
      if (data.startsWith("SELECT:")) {// parses data for this specific game
        selectedAnswer = int(data.substring(7));  // 8th character gives actual numeric value
      } else if (data.startsWith("PRESS:")) {  // for confirm button press
        let val = int(data.substring(6));  // 7th character gives digit confirmed by user 
        // success condition
        if (val === correctAnswer) {
          feedback = "CORRECT";
          stageDigit = 2;
          gameCompleted[1] = true;
          stageCompleted = true;
          stageCompleteTime = millis();
        } else {
          // in case of wrong answer
          bombExploded = true;
          if (port.opened()) {
            port.write("EXPLODED\n");
          }
        }
      }
      break;
    // handling data for note match game
    case 2:
      // if user presses confirm button, checks answer
      if (!lockedIn) {
        if (data === "CONFIRM") {
          lockedIn = true;
          // if correct answer is selected 
          if (currentSelection === noteAnswerIndex) {
            noteMessage = "Correct!";
            stageDigit = 9;
            gameCompleted[2] = true;
            stageCompleted = true;
            stageCompleteTime = millis();
            // if user makes a mistake, they lose
          } else {
            bombExploded = true;
            if (port.opened()) {
              port.write("EXPLODED\n");
            }
          }
        } else if (!isNaN(int(data))) {
          currentSelection = int(data);  // reading data for option selected 
        }
      }
      break;
    // parsing user input based on arduino feedback to concatenate morse code and compare with original string
    case 3:
      if (data === "." || data === "-") {
        userInput += data;
        // if user confirms answer 
      } else if (data === "CONFIRM") {
        if (userInput === morseCode) {
          showSuccess = true;
          stageDigit = 7;
          gameCompleted[3] = true;
          stageCompleted = true;
          stageCompleteTime = millis();
          roundActive = false;
          // in case of incorrect answer
        } else {
          bombExploded = true;
          if (port.opened()) {
            port.write("EXPLODED\n");
          }
        } // displays morse code for 5 seconds for user to memorize then disappears
        setTimeout(() => {
          showSuccess = false;
          showFailure = false;
          userInput = "";
        }, 5000);
      }
      break;

    case 4:
      // handles code entry
      if (data === "CONFIRM") {
        if (userCodeInput.length !== 4) return; // Ignore if code is incomplete
        if (userCodeInput === "4297") {
          bombDefused = true;
        } else {
          bombExploded = true;
          if (port.opened()) {
            port.write("EXPLODED\n");
          }
        }
      }
      break;
  }
}
// to tell arduino to switch to game being sent
function sendGameSwitch(gameNum) {
  if (port.opened()) {
    port.write(gameNum + "\n");
  }
}

// all game display functions
function startButtonMashChallenge() {
  pressCount = 0;
  challengeActive = true;
}

function drawButtonMashChallenge() {
  image(imgButtonSmash, 0, 0, width, height);
  fill(255);
  textSize(44);
  textAlign(CENTER, CENTER);
  text(pressCount, 300,325); 
}

function startMathQuiz() {
  feedback = "";
  correctAnswer = 5; 
  selectedAnswer = 0;
}


function drawMathQuiz() {
  image(imgMathRiddle, 0, 0, width, height);

  fill(29,148,94);
  rect(width / 2 - 40, 350, 80, 80);
  fill(0);
  textSize(48);
  text(selectedAnswer, width / 2, 370);

}

function startNoteMatchChallenge() {
  lockedIn = false;
  noteMessage = "";
  currentSelection = -1;
  noteAnswerIndex = floor(random(0, 4));
  sendNoteChallenge(noteAnswerIndex);
}

function drawNoteMatchChallenge() {
  image(imgNoteMatch, 0, 0, width, height);
  
  textSize(24);
  textAlign(CENTER, CENTER); 
  fill(0);

  let labels = ["C", "D", "E", "F"];
  let size = 80;
  let spacing = 20;
  let totalWidth = labels.length * size + (labels.length - 1) * spacing;
  let startX = (width - totalWidth) / 2;
  let y = 300;

  for (let i = 0; i < labels.length; i++) {
    let x = startX + i * (size + spacing);
    
    if (i === currentSelection) {
      fill(0, 0, 255);
    } else {
      fill(255);
    }

    rect(x, y, size, size);

    fill(0);
    text(labels[i], x + size / 2, y + size / 2); 
  }

  fill(0);
  textSize(20);
  text(noteMessage, width / 2, height - 50);
}


function startMorseCodeChallenge() {
  morseCode = "..-.--.";
  userInput = "";
  roundActive = true;
  showSuccess = false;
  showFailure = false;
  setTimeout(() => {
    roundActive = false;
  }, 5000);
}
// displays image with code for 5 seconds for user to memorize code 
function drawMorseCodeChallenge() {
  if (roundActive) {
    image(imgMorseCode1, 0, 0, width, height);
  } else {
    image(imgMorseCode2, 0, 0, width, height);}
  fill(50,50,50);
  textSize(24);
  text("User input: " + userInput, width / 2, 300);
}

function drawCodeEntry() {
  image(imgCodeEntry, 0, 0, width, height);
  textSize(24);
  fill(0);
  text(userCodeInput, width / 2, 170);

  for (let i = 0; i <= 9; i++) {
    let x = 140 + (i % 5) * 80;
    let y = 220 + floor(i / 5) * 80;
    fill(200);
    rect(x, y, 60, 60);
    fill(0);
    text(i, x + 30, y + 30);
  }

  fill(255);
  rect(width / 2 - 35, 410, 70, 40);
  fill(0);
  text("Clear", width / 2, 420);
}

function mousePressed() {
  // handles navigation from welcome screen to instructions screen and instructions screen and back
  if (showWelcome) {
    if (mouseX > 112 && mouseX < 224 && mouseY > 508 && mouseY < 547) {
      try {
        // creating serial connection
        if (!port.opened()) {
          let usedPorts = usedSerialPorts();
          if (usedPorts.length > 0) {
            port.open(usedPorts[0], baudrate);
          } else {
            port.open(baudrate);
          }
        }
        console.log("Connected to serial!");
        startGame();
        showWelcome = false;
      } catch (err) {
        console.error("Connection failed:", err);
      }
    }
    
    if (mouseX > 275 && mouseX < 544 && mouseY > 506 && mouseY < 545) {
      showInstructions = true;
      showWelcome = false;
    }
    return;
  }

  if (showInstructions) {
    // Click anywhere to go back
    showInstructions = false;
    showWelcome = true;
  }
  // checks code entry
  if (currentGame === 4 && !bombDefused && !bombExploded) {
    for (let i = 0; i <= 9; i++) {
      let x = 140 + (i % 5) * 80;
      let y = 220 + floor(i / 5) * 80;
      if (mouseX > x && mouseX < x + 60 && mouseY > y && mouseY < y + 60) {
        userCodeInput += i;
        return;
      }
    }

    // clear button
    if (mouseX > width / 2 - 30 && mouseX < width / 2 + 30 &&
        mouseY > 400 && mouseY < 440) {
      userCodeInput = userCodeInput.slice(0, -1);
      return;
    }
    // successful code entry defuses bomb successfully
    if (userCodeInput.length === 4) {
      if (userCodeInput === correctCode) {
        bombDefused = true;
        port.write("DEFUSED\n");
      }
      else bombExploded = true;
    }
  }
}
// for arduino to choose note sent as correct note and play it for users to guess 
function sendNoteChallenge(noteIndex) {
  if (port.opened()) {
    port.write("NOTE:" + noteIndex + "\n");
  }
}
function startCodeEntry() {
  userCodeInput = "";
}

// resets all game variables for reset functionality once game ends 
function resetGame() {
  showWelcome = true;
  showInstructions = false;
  gameStarted = false;
  currentGame = 0;
  gameCompleted = [false, false, false, false];
  codeDigits = [];
  userCodeInput = "";
  correctCode = "";
  bombDefused = false;
  bombExploded = false;
  stageCompleted = false;
  stageCompleteTime = 0;
  stageDigit = -1;
  pressCount = 0;
  challengeActive = false;
  selectedAnswer = 0;
  correctAnswer = 5;
  feedback = "";
  currentSelection = -1;
  lockedIn = false;
  noteMessage = "";
  noteAnswerIndex = 0;
  morserrCode = "";
  userInput = "";
  roundActive = false;
  showSuccess = false;
  showFailure = false;
  playedExplosionSound = false;
  playedSuccessSound = false;
}
// if user presses key once game is over, it restarts everything
function keyPressed() {
  if (key === 'R' || key === 'r') {
    resetGame();
    port.write("RESET\n");
  }
}

Challenges & Lessons Learned

• Serial Port Management: One recurring headache was managing serial port connections on browser refreshes and game resets. I had to add logic to prevent re-opening already open ports to avoid exceptions.

• Real-Time Feedback: Timing and responsiveness were crucial. Since the game runs on a strict timer, any lag in serial communication or missed input could break the experience. Careful buffering and validation were necessary.

• Game Flow Management: Keeping track of game state across 5 different modes, plus timers and sounds, took careful design. The stageCompleted flag and a timed transition window after each success proved essential.

For this final project, I built an interactive game that represents the mental world of a child’s mind—heavily inspired by Inside Out. The digital part of the game was made using p5.js, and it connects to three physical Arduino setups: two for the moving characters (Joy and Sadness) and one for the glowing memory balls.

You play by dodging obstacles and collecting star-shaped powerups. There are two types of obstacles: the memory bridge (which the player slides under) and the memory ball cluster (which you jump over). As the player moves through the level, their real-life emotions move along a race track: Joy advances when the player collects powerups, and Sadness moves forward every time an obstacle is hit.

If Sadness reaches the end first (if user hit 3 obstacles), a memory ball glows blue. If Joy wins (3 powerups), it glows yellow.

Hardware + Fabrication:

• I designed the track pattern in Illustrator and laser-cut it .

• I downloaded STL files of Joy and Sadness from Printables, 3D printed them, and painted them with acrylic paint. Then i laser cut a stand and glued them to it so that they were on top of the arduino boards that were connected to the wheels.

• I used SolidWorks to model the memory ball, 3D printed it, and added LEDs connected to breadboards inside then connected the 2 breadboards to a single arduino.

• 

  

   

  

Code + Arduino Integration:

At first, I wanted to send letters (‘O’, ‘P’, etc.) from p5 to Arduino to trigger events, but it just wasn’t reliable. After about 8 hours of trial and error, I figured out that using numbers instead of characters made the communication consistent.

Another issue was the Arduinos themselves. Because the boards are inside the moving characters, they were getting damaged by all the motion and collisions. They’d randomly stop working even though the wiring was fine. I had to replace the connections multiple times, and one of the boards is still a little unstable.

Arduino Code – Joy & Sadness Movement (Boards 1 & 2):

const int STBY = 6;

// Motor A
const int AIN1 = 4;
const int AIN2 = 5;
const int PWMA = 3;

// Motor B
const int BIN1 = 7;
const int BIN2 = 8;
const int PWMB = 9;

void setup() {
  Serial.begin(9600);

  pinMode(STBY, OUTPUT);
  pinMode(AIN1, OUTPUT);
  pinMode(AIN2, OUTPUT);
  pinMode(PWMA, OUTPUT);
  pinMode(BIN1, OUTPUT);
  pinMode(BIN2, OUTPUT);
  pinMode(PWMB, OUTPUT);

  digitalWrite(STBY, HIGH); // Enable motor driver
}

void loop() {
  if (Serial.available()) {
    char command = Serial.read();

    if (command == 0 || command == 1) {
      moveForward(200); // Set speed here (0–255)
      delay(1000);       // Adjust this value to travel the desired cm (experiment to tune)
      stopMotors();
    }
  }
}

void moveForward(int speed) {
  // Motor A forward
  digitalWrite(AIN1, HIGH);
  digitalWrite(AIN2, LOW);
  analogWrite(PWMA, speed);

  // Motor B forward
  digitalWrite(BIN1, HIGH);
  digitalWrite(BIN2, LOW);
  analogWrite(PWMB, speed);
}

void stopMotors() {
  // Stop Motor A
  analogWrite(PWMA, 0);
  digitalWrite(AIN1, LOW);
  digitalWrite(AIN2, LOW);

  // Stop Motor B
  analogWrite(PWMB, 0);
  digitalWrite(BIN1, LOW);
  digitalWrite(BIN2, LOW);
}

Arduino Code – Memory Ball LEDs (Board 3):

// Pins for Motor Driver (L298N or similar)
#define MOTOR1_PIN1 2
#define MOTOR1_PIN2 3
#define MOTOR2_PIN1 4
#define MOTOR2_PIN2 5

void setup() {
  Serial.begin(9600);
  pinMode(MOTOR1_PIN1, OUTPUT);
  pinMode(MOTOR1_PIN2, OUTPUT);
  pinMode(MOTOR2_PIN1, OUTPUT);
  pinMode(MOTOR2_PIN2, OUTPUT);
}

void loop() {
  if (Serial.available()) {
    char cmd = Serial.read();
    if (cmd == 'O') { // 'O' for Obstacle
      // Move both motors forward for 1 second
      digitalWrite(MOTOR1_PIN1, HIGH);
      digitalWrite(MOTOR1_PIN2, LOW);
      digitalWrite(MOTOR2_PIN1, HIGH);
      digitalWrite(MOTOR2_PIN2, LOW);
      delay(1000);
      stopMotors();
    }
  }
}

void stopMotors() {
  digitalWrite(MOTOR1_PIN1, LOW);
  digitalWrite(MOTOR1_PIN2, LOW);
  digitalWrite(MOTOR2_PIN1, LOW);
  digitalWrite(MOTOR2_PIN2, LOW);
}

The p5.js Sketch:

The sketch handles the game logic—collision detection, button controls, score tracking, and serial communication with  Arduinos.

• Character controller with jump/slide mechanics

• Obstacle spawning at set intervals

• Collision detection that updates the win count for Joy or Sadness

• Serial output that sends a number to each board depending on the in-game event

What I’m Most Proud Of:

Definitely the connection between digital and physical. Seeing Joy or Sadness physically move when something happens in the game is so rewarding. It brings the internal emotional world of the game into the real world.

Also, figuring out how to manage serial communication across 3 Arduinos (despite many boards dying on me) felt like a huge accomplishment.

If I Had More Time:

• Improve the Arduino casing so that the wires don’t come loose so easily

• Add a small screen showing who’s winning in real-time

P5 Code: 

let port1, port2;
let connectBtn1, connectBtn2;
let player;
let obstacles = [];
let laneWidth;
let lanes = [0, 1, 2];
let score = 0;
let gameSpeed = 5;
let obstacleSpawnCounter = 0;
let bg1, bg2;
let gameState = "start";
let startButton;
let powerUps = [];
let powerUpSpawnCounter = 0;
let gameMusic;
let obstacleHits = 0;
let powerUpHits = 0;
const MAX_HITS = 3;

function preload() {
  bg2 = loadImage("background.png");
  bg1 = loadImage("headquarters.png");
  gameMusic = loadSound("Song.mp3");
}

function setup() {
  createCanvas(windowWidth, windowHeight);
  laneWidth = width / 3;
  setupGame();

  port1 = createSerial();
  port2 = createSerial();

  startButton = createButton("Start Game");
  startButton.position(width / 2 - 50, height / 2 + 40);
  startButton.size(100, 40);
  startButton.mousePressed(() => {
    gameState = "playing";
    startButton.hide();
  });

  connectBtn1 = createButton("Connect Arduino 1 (Obstacle)");
  connectBtn1.position(50, 50);
  connectBtn1.mousePressed(() => port1.open(9600));

  connectBtn2 = createButton("Connect Arduino 2 (Powerup)");
  connectBtn2.position(50, 100);
  connectBtn2.mousePressed(() => port2.open(9600));
}

function setupGame() {
  player = new Player();
  obstacles = [];
  score = 0;
  gameSpeed = 5;
  obstacleSpawnCounter = 0;
  powerUps = [];
  powerUpSpawnCounter = 0;
  obstacleHits = 0;
  powerUpHits = 0;

  if (gameState === "start" && startButton) {
    startButton.show();
  }
}

function draw() {
  background(bg2);

  if (gameState === "start") {
    drawStartScreen();
  } else if (gameState === "playing") {
    if (!gameMusic.isPlaying()) {
      gameMusic.loop();
    }
    runGame();
  } else if (gameState === "gameover") {
    if (gameMusic.isPlaying()) {
      gameMusic.stop();
    }
    drawGameOverScreen();
  }
}

function drawStartScreen() {
  fill(255);
  background(bg1);
  textAlign(CENTER, CENTER);
  textSize(48);
  text("INSIDE OUT RUNNER", width / 2, height / 2 - 120);

  textSize(24);
  text("Instructions:", width / 2, height / 2 - 50);
  textSize(20);
  text("Up Arrow: Jump over Memory Ball Cluster (obstacle)", width / 2, height / 2 - 15);
  text("Down Arrow: Slide under Memory Bridge (obstacle)", width / 2, height / 2 + 5);
  text("Left/Right Arrows: Move between the 3 lanes", width / 2, height / 2 + 25);

  textSize(24);
  text("Objective:", width / 2, height / 2 + 100);
  textSize(20);
  text("Collect 3 stars for Joy to win!", width / 2, height / 2 + 130);
  text("Hit 3 obstacles for Sadness to win!", width / 2, height / 2 + 150);
}

function drawGameOverScreen() {
  background(bg1);
  fill(255, 50, 50);
  textAlign(CENTER, CENTER);
  textSize(48);
  text("Game Over", width / 2, height / 2 - 40);
  textSize(24);
  text("Final Score: " + floor(score / 10), width / 2, height / 2 + 10);
  text("Press R to Restart", width / 2, height / 2 + 50);

  if (obstacleHits >= 3) {
    text("Sadness Wins!", width / 2, height / 2 + 80);
  } else if (powerUpHits >= 3) {
    text("Joy Wins!", width / 2, height / 2 + 80);
  }
}

function runGame() {
  gameSpeed += 0.0005;
  background(bg2);

  player.update();
  player.show();

  obstacleSpawnCounter++;
  if (obstacleSpawnCounter > max(90 - gameSpeed * 5, 40)) {
    obstacles.push(new Obstacle());
    obstacleSpawnCounter = 0;
  }

  for (let i = obstacles.length - 1; i >= 0; i--) {
    if (obstacles[i].offscreen()) {
      obstacles.splice(i, 1);
      continue;
    }
    obstacles[i].move();
    obstacles[i].show();

    if (obstacles[i].hits(player)) {
      obstacleHits++;
      obstacles.splice(i, 1);
      if (port1 && port1.opened()) {
        port1.write(0);
      }
      if (obstacleHits >= MAX_HITS) {
        gameState = "gameover";
      }
    }
  }

  powerUpSpawnCounter++;
  if (powerUpSpawnCounter > 300) {
    powerUps.push(new PowerUp());
    powerUpSpawnCounter = 0;
  }

  for (let i = powerUps.length - 1; i >= 0; i--) {
    if (powerUps[i].offscreen()) {
      powerUps.splice(i, 1);
      continue;
    }
    powerUps[i].move();
    powerUps[i].show();

    if (powerUps[i].hits(player)) {
      powerUpHits++;
      powerUps.splice(i, 1);
      score += 100;
      if (port2 && port2.opened()) {
        port2.write(1);
      }
      if (powerUpHits >= MAX_HITS) {
        gameState = "gameover";
      }
    }
  }

  score += 1;
  fill(255);
  textSize(24);
  textAlign(LEFT);
  text("Score: " + floor(score / 10), 10, 30);

  textAlign(RIGHT);
  text("Obstacles: " + obstacleHits + "/3", width - 10, 30);
  text("Powerups: " + powerUpHits + "/3", width - 10, 60);
  textAlign(LEFT);
}

function keyPressed() {
  if (gameState === "start" && key === " ") {
    gameState = "playing";
  } else if (gameState === "gameover" && (key === "r" || key === "R")) {
    setupGame();
    gameState = "playing";
  }

  if (gameState === "playing") {
    if (keyCode === LEFT_ARROW) {
      player.move(-1);
    } else if (keyCode === RIGHT_ARROW) {
      player.move(1);
    } else if (keyCode === UP_ARROW) {
      player.jump();
    } else if (keyCode === DOWN_ARROW) {
      player.slide();
    }
  }
}

class Player {
  constructor() {
    this.lane = 1;
    this.x = laneWidth * this.lane + laneWidth / 2;
    this.baseY = height - 50;
    this.y = this.baseY;
    this.r = 20;
    this.gravity = 1;
    this.velocity = 0;
    this.isJumping = false;
    this.isSliding = false;
    this.slideTimer = 0;
  }

  show() {
    fill(255);
    noStroke();
    if (this.isSliding) {
      ellipse(this.x, this.y + this.r / 2, this.r * 2, this.r);
    } else {
      ellipse(this.x, this.y, this.r * 2);
    }
  }

  move(dir) {
    this.lane += dir;
    this.lane = constrain(this.lane, 0, 2);
    this.x = laneWidth * this.lane + laneWidth / 2;
  }

  jump() {
    if (!this.isJumping && !this.isSliding) {
      this.velocity = -20;
      this.isJumping = true;
    }
  }

  slide() {
    if (!this.isJumping && !this.isSliding) {
      this.isSliding = true;
      this.slideTimer = 30;
    }
  }

  update() {
    this.y += this.velocity;
    this.velocity += this.gravity;

    if (this.y >= this.baseY) {
      this.y = this.baseY;
      this.velocity = 0;
      this.isJumping = false;
    }

    if (this.isSliding) {
      this.slideTimer--;
      if (this.slideTimer <= 0) {
        this.isSliding = false;
      }
    }
  }
}

function drawEmotionCloud(x, y) {
  const colors = ['#A066FF', '#FF6666', '#FFFF66', '#66FF66', '#66CCFF'];
  for (let i = 0; i < 5; i++) {
    fill(colors[i % colors.length]);
    noStroke();
    ellipse(x + i * 20, y + sin(i) * 8, 30, 30);
  }
}

function drawMemoryBridge(x, y) {
  fill("purple");
  beginShape();
  for (let dx = -60; dx <= 60; dx += 5) {
    let dy = -20 * sin(PI * (dx + 60) / 120);
    vertex(x + dx, y + dy);
  }
  for (let dx = 60; dx >= -60; dx -= 5) {
    let dy = -10 * sin(PI * (dx + 60) / 120);
    vertex(x + dx, y + dy + 20);
  }
  endShape(CLOSE);

  stroke(0);
  strokeWeight(2);
  for (let dx = -50; dx <= 50; dx += 20) {
    let dy = -20 * sin(PI * (dx + 60) / 120);
    line(x + dx, y + dy, x + dx, y + dy - 10);
  }

  noFill();
  strokeWeight(2);
  beginShape();
  for (let dx = -50; dx <= 50; dx += 5) {
    let dy = -20 * sin(PI * (dx + 60) / 120);
    vertex(x + dx, y + dy - 10);
  }
  endShape();
}

// --- Obstacle Class ---
class Obstacle {
  constructor() {
    this.lane = random(lanes);
    this.x = laneWidth * this.lane + laneWidth / 2;
    this.y = 350;
    if (random(1) < 0.5) {
      this.type = "high";
    } else {
      this.type = "low";
    }
    this.color = color("blue");
  }

  move() {
    this.y += gameSpeed;
  }

  show() {
    if (this.type === "low") {
      drawMemoryBridge(this.x, this.y);
    } else {
      drawEmotionCloud(this.x, this.y);
    }
  }

  hits(player) {
    let playerRadius = player.isSliding ? player.r * 0.5 : player.r;
    let px = player.x;
    let py = player.isSliding ? player.y + player.r / 2 : player.y;

    let collisionX = abs(this.x - px) < 60 / 2 + playerRadius;
    let collisionY = abs(this.y - py) < 40 / 2 + playerRadius;

    if (collisionX && collisionY) {
      if (this.type === "high" && !player.isJumping) return true;
      if (this.type === "low" && !player.isSliding) return true;
    }
    return false;
  }

  offscreen() {
    return this.y > height + 40;
  }
}

// --- PowerUp Class ---
class PowerUp {
  constructor() {
    this.lane = random(lanes);
    this.x = laneWidth * this.lane + laneWidth / 2;
    this.y = 350;
    this.size = 30;
    this.color = color(255, 174, 66);
  }

  move() {
    this.y += gameSpeed;
  }

  show() {
    push();
    fill(this.color);
    noStroke();
    translate(this.x, this.y);
    beginShape();
    for (let i = 0; i < 10; i++) {
      let angle = TWO_PI * i / 10;
      let r = i % 2 === 0 ? this.size : this.size / 2;
      let sx = cos(angle) * r;
      let sy = sin(angle) * r;
      vertex(sx, sy);
    }
    endShape(CLOSE);
    pop();
  }

  offscreen() {
    return this.y > height + this.size;
  }

  hits(player) {
    let d = dist(this.x, this.y, player.x, player.y);
    return d < this.size + player.r;
  }
}

For this step of the project, I focused on user testing and started making some updates based on how people interacted with the game. The concept is a physical memory-based game where two characters move forward when the player makes a correct choice, but if they hit an obstacle, a glowing memory ball (controlled by another Arduino) lights up.

Right now, I’m still working on the Arduino connection between the boards. Each moving character has its own Arduino, and I’ve written code that lets them move forward when they receive a command — but syncing the movement between the two boards and handling serial communication properly is still a work in progress. I’m also still refining how everything connects with the third Arduino, which controls the obstacle LED reaction in the memory balls.

During testing, I noticed that sometimes one board would move while the other didn’t, or the LED wouldn’t react consistently. So I’ve been going back into the code and checking the timing, delays, and serial input handling. It’s been a bit tricky trying to get all three Arduinos to communicate clearly without any delays or missed signals.

This week, I’m focusing on:

**Finalizing the Arduino code for the two moving boards
**Getting the serial communication between the three Arduinos to work smoothly

The part I’m most proud of so far is how the basic movement is already working and how the obstacle-triggered lights bring the whole thing to life. Once the connection part is sorted out, I think the physical interactions will feel much more natural and immersive.

For this assignment, Izza and I worked together to do the 3 tasks.

Task 1: 

For this task, we used a potentiometer to control whether the ellipse was moving to the left or the right on the horizontal axis. We did this by mapping the values of the potentiometer (0-1023) on the horizontal axis. Then, when we turned the potentiometer the value would translate to a position on the horizontal axis that the ball would move to. We had some difficulties with the delay between the arduino and p5js as sometimes we’d have to wait a couple seconds before the code would update in p5. Here is the code for the arduino:

void setup() {

  Serial.begin(9600);

}




// Read potentiometer value (0–1023) and sends to p5js

void loop() {

  int sensorValue = analogRead(A0);

  Serial.println(sensorValue);

  delay(1);

}

Task 2:

For this task, we had to do something that controlled the brightness of an LED on the arduino breadboard through p5js. So, we decided to create a dropdown for the user to pick between 1-10 to control the brightness of the LED with 1 being off and 10 being maximum brightness. We then mapped that to the 0-255 range for the brightness of an LED and sent that to the arduino which would control how brightly the LED would light up for a few seconds. On the arduino we simply had one bulb connected to digital pin 9. The arduino code can be seen below:

void setup() {

  Serial.begin(9600);

  pinMode(9, OUTPUT);

}




//gets the serial converted value from p5js

void loop() {

  if (Serial.available() > 0) {

    int brightness = Serial.parseInt(); 

    brightness = constrain(brightness, 0, 255); //make sure the value isn't out of range

    analogWrite(9, brightness);

  }

}

Task 3:

In this task, we had to take already existing code and alter it such that every time the ball bounced, one LED light on the arduino lit up, and the wind was controlled by an analog sensor. For controlling our wind, we used a potentiometer once again as we could make it such that values above 512 would move the ball to the east (right) and values below 512 would move the ball towards the west (left). On the arduino, we connected a potentiometer at analog pin A0 and an LED light on digital pin 9. We then used p5js to recieve that serial input from the potentiometer and map it to the wind. Whether it bounced being true or false is also what makes the LED light up. Once again, we did experience a delay between the potentiometer value and the wind in p5. The arduino code can be seen below:

const int potPin = A0;

const int ledPin = 9;

bool ledOn = false;

unsigned long ledTimer = 0;

const int ledDuration = 100;




void setup() {

  Serial.begin(9600);

  pinMode(ledPin, OUTPUT);

}




void loop() {

  // Read potentiometer and send value

  int potValue = analogRead(potPin);

  Serial.println(potValue);




  // If LED was turned on recently, turn it off after some time

  if (ledOn && millis() - ledTimer > ledDuration) {

    digitalWrite(ledPin, LOW);

    ledOn = false;

  }




  // recieve signal on whether the ball bounced from p5.js

  if (Serial.available()) {

    String input = Serial.readStringUntil('\n');

    input.trim();




    if (input == "bounce") {

      digitalWrite(ledPin, HIGH);

      ledOn = true;

      ledTimer = millis();

    }

  }




  delay(10); // Slight delay for stability

}

Lastly, here is the link to the video showing the LED light up and the ball being “blown” away by the value sent from the potentiometer:

https://drive.google.com/file/d/140pGv-9DMPd1gCa1xMn_LR3pR_pphx47/view?usp=sharing

Concept & Goals

Inspiration: Give a plant a “voice” through digital animation and sound, fostering empathy and care.

The idea was to monitor ambient light and human touch on the plant, translate those signals into a friendly digital avatar’s mood, color-changing lamp, moving leaf, and background music.

• • Goals:

    1. Awareness of plant wellbeing via playful tech.

    2. Interaction through capacitive touch (DIY sensor) and light sensing.

    3. Empathy by giving the plant a way to “talk back.”

Video

Avatar

Setup

Hardware Overview

1. Capacitive Touch Sensor (DIY)

• • Pins: D4 (send) → 1 MΩ resistor → D2 (receive + foil/copper tape).

  • Library: Paul Badger’s CapacitiveSensor. Downloaded for the Arduino code.

  • Assembly:

    1. Connect a 1 MΩ resistor between pin 4 and pin 2.

    2. Attach copper tape to the receiving leg (pin 2) and wrap gently around the plant’s leaves.

    3. In code: CapacitiveSensor capSensor(4, 2);

2. Photoresistor (Ambient Light)

• • Pins: LDR → +5 V; 10 kΩ → GND; junction → A0.

  • Function: Reads 0–1023, mapped to 0–255 to control lamp intensity.

3. Push-Button (Music Control)

• • Pins: Button COM → D7, NO → GND (using INPUT_PULLUP).

4. Mood LEDs

• • Pins:

    • Green LED1: D12 → 330 Ω → LED → GND

    • Red LED2: D13 → 330 Ω → LED → GND

  • Behavior:

    • Red LED on when touch > high threshold (indicates that the plant does not like the touch).

    • Green LED on when touch < low threshold (the plant is calm and likes the touch).

Arduino Code

#include <CapacitiveSensor.h>

// Capacitive sensor: send→ 4, receive→ 2
CapacitiveSensor capSensor(4, 2);

// Mood LEDs
const int ledPin   = 12;
const int ledPin2  = 13;

// Photoresistor
const int photoPin = A0;

// Push-button + button LED
const int buttonPin    = 7;  // COM→7, NO→GND


// Hysteresis thresholds
const long thresholdHigh = 40;
const long thresholdLow  = 20;

// Debounce
const unsigned long debounceDelay = 50;
unsigned long lastDebounceTime    = 0;
int           lastButtonReading   = HIGH;

// State trackers
bool musicOn = false;
bool led1On  = false;

void setup() {
  Serial.begin(9600);
  delay(100);

  // Let p5.js know we start OFF
  Serial.println("MUSIC_OFF");

  // Mood LEDs
  pinMode(ledPin,   OUTPUT);
  pinMode(ledPin2,  OUTPUT);

  // Capacitive sensor raw
  capSensor.set_CS_AutocaL_Millis(0);

  // Button LED off
  pinMode(buttonLedPin, OUTPUT);
  digitalWrite(buttonLedPin, LOW);

  // Push-button with pull-up
  pinMode(buttonPin, INPUT_PULLUP);
}

void loop() {
  // Button toggle (only prints MUSIC_ON / MUSIC_OFF) 
  int reading = digitalRead(buttonPin);
  if (reading != lastButtonReading) {
    lastDebounceTime = millis();
  }
  if (millis() - lastDebounceTime > debounceDelay) {
    static int buttonState = HIGH;
    if (reading != buttonState) {
      buttonState = reading;
      if (buttonState == LOW) {            // you pressed
        musicOn = !musicOn;
        Serial.println(musicOn ? "MUSIC_ON" : "MUSIC_OFF");
        digitalWrite(buttonLedPin, musicOn ? HIGH : LOW);
      }
    }
  }
  lastButtonReading = reading;

  // Capacitive
  long sensorValue = capSensor.capacitiveSensor(30);
  Serial.println(String("TOUCH:") + sensorValue);

  // Mood LED hysteresis 
  if (!led1On && sensorValue > thresholdHigh) {
    led1On = true;
  } else if (led1On && sensorValue < thresholdLow) {
    led1On = false;
  }
  digitalWrite(ledPin,  led1On ? HIGH : LOW);
  digitalWrite(ledPin2, led1On ? LOW  : HIGH);

  // Photoresistor
  int raw       = analogRead(photoPin);
  int mappedVal = map(raw, 0, 1023, 0, 255);
  Serial.println(String("LAMP:") + mappedVal);

  delay(50);
}

Serial messages:

• • MUSIC_ON / MUSIC_OFF (button)

  • TOUCH:<value> (capacitive)

  • LAMP:<0–255> (light)

P5js Code

let port;
const baudrate = 9600;
let connectButton;
let bgMusic;
let interactionStarted = false;
let isTouched = false;
let lampBrightness = 0;


let plankCount = 6;
let cam;
let myFont;
let waveAngle = 0;
let isWaving = false;
let waveStartTime = 0;



function preload() {
  myFont = loadFont('CalligraphyFLF.ttf');
  bgMusic = loadSound('musica.mp3');  // load  MP3
}
  // Works for clicks, touches, and fullscreen events
  const unlockAudio = () => {
    if (!audioUnlocked) {
      getAudioContext().resume().then(() => {
        console.log('AudioContext unlocked');
        audioUnlocked = true;
        if (musicOn && bgMusic.isLoaded()) bgMusic.loop();
      });
    }
  };
  // Mouse/touch unlock
  window.addEventListener('mousedown', unlockAudio);
  window.addEventListener('touchstart', unlockAudio);
  // Also unlock on fullscreen change
  document.addEventListener('fullscreenchange', unlockAudio);

function setup() {
  createCanvas(windowWidth, windowHeight, WEBGL);

  connectButton = createButton("Connect to Arduino");
  connectButton.position(20, 20);
  connectButton.mousePressed(() => {
    port.open(baudrate);
  });

  port = createSerial();
  const used = usedSerialPorts();
  if (used.length > 0) {
    port.open(used[0], baudrate);
  } else {
    console.warn("No previously used serial ports found.");
  }

  setInterval(onSerialData, 50);   

  textFont(myFont);
  textSize(36);
  textAlign(CENTER, CENTER);

  cam = createCamera();
  const fov = PI / 3;
  const cameraZ = (height / 2.0) / tan(fov / 2.0);
  cam.setPosition(0, 0, cameraZ);
  cam.lookAt(0, 0, 0);
  frameRate(60);
}
function onSerialData() { //serial data
  if (!port || !port.opened()) return;
  while (port.available() > 0) {
    const raw = port.readUntil('\n');
    if (!raw) break;
    const line = raw.trim();
    console.log('Received:', line);

    if (line.startsWith('LAMP:')) {
      lampBrightness = int(line.split(':')[1]);
    } else if (line.startsWith('TOUCH:')) {
      const t = int(line.split(':')[1]);
      // ignore baseline zero readings
      if (t > 0) {
        isTouched = true;
        isWaving = true;
        waveStartTime = millis();
      }
    } else if (line === 'START') {
      interactionStarted = true;
    } else if (line === 'MUSIC_ON') {
      musicOn = true;
      if (bgMusic.isLoaded()) {
        console.log('MUSIC_ON → play');
        bgMusic.play();
      }
    } else if (line === 'MUSIC_OFF') {
      musicOn = false;
      if (bgMusic.isPlaying()) {
        console.log('MUSIC_OFF → stop');
        bgMusic.stop();
      }
    }
  }
}

//Draw
function draw() {
  background(210, 180, 140);
  ambientLight(10);
  pointLight(255, 200, 150, 200, -300, 300);
  pointLight(255, 150, 100, 400, -300, 300);

  drawWarmGradientBackground();
  drawWarmLamp();
  drawWoodDeck();
  drawWoodenBase();
  drawShadow();
  drawPot();
  drawBody();
  drawFace();
  drawPetiole();
  drawInstructionFrame();

  if (!interactionStarted) {
    drawInstructionFrame();
  } else {
    drawScene();
    pointLight(lampBrightness, lampBrightness * 0.8, 100, 200, -300, 300);
  }

  if (isWaving) {
    waveAngle = sin((millis() - waveStartTime) / 2000) * 0.5;

    if (millis() - waveStartTime > 400) {
      isWaving = false;
      waveAngle = 0;
    }
  }

  push();
  rotateZ(waveAngle);
  drawLeaf(0, -140, 0, 1, 4);
  pop();

  push();
  rotateZ(-waveAngle);
  drawLeaf(0, -140, 0, -1, 4);
  pop();

}

function windowResized() {
  resizeCanvas(windowWidth, windowHeight);
  const fov = PI / 3;
  const cameraZ = (height / 2.0) / tan(fov / 2.0);
  cam.setPosition(0, 0, cameraZ);
}


function drawWarmLamp() {
  push();
  translate(250, -250, 0);

  // glow — modulate alpha by lampBrightness
  for (let r = 300; r > 50; r -= 20) {
    push();
    noStroke();
    // fade alpha between 0 and 150
    let a = map(lampBrightness, 0, 255, 20, 150);
    fill(255, 180, 90, a);
    ellipse(0, 10, r, r * 1.2);
    pop();
  }

  // stand
  fill(100, 70, 50);
  translate(0, 200, 0);
  cylinder(5, 400);

  // base
  translate(0, 200, 0);
  fill(80, 60, 40);
  ellipse(0, 0, 80, 16);

  // lampshade (inverted cone)
  push();
  translate(0, -400, 0);
  fill(225, 190, 150);
  cone(50, -100, 24);
  pop();

  pop();
}

function drawWoodDeck() {
  push();
    rotateX(HALF_PI);
    translate(0, -2, -170);
    // static plank color
    fill(160, 100, 60);
    stroke(100, 60, 40);
    const plankHeight = -50;
    for (let i = 0; i < plankCount; i++) {
      box(width, plankHeight, 10);
      translate(0, plankHeight, 0);
    }
  pop();
}

function drawWoodenBase() {
  push();
    rotateX(HALF_PI);
    translate(0, 150, -200);
    // static plank color
    fill(160, 100, 60);
    stroke(100, 60, 40);
    const baseCount = 8;
    const plankWidth = (width * 1.2) / baseCount;
    for (let i = 0; i < baseCount; i++) {
      push();
        translate(-width * 0.6 + i * plankWidth + plankWidth / 2, 24, 0);
        box(plankWidth, 400, 20);
      pop();
    }
  pop();
}


function drawShadow() {
  push();
  translate(0, 150, -10);
  rotateX(HALF_PI);
  noStroke();
  fill(80, 70, 60, 30);
  ellipse(0, 0, 190, 30);
  pop();
}

function drawPot() {
  push();
  translate(0, 100, 0);
  fill(170, 108, 57);
  stroke(120, 70, 40);
  strokeWeight(1);
  cylinder(60, 80);
  translate(0, -50, 0);
  fill(190, 120, 70);
  cylinder(80, 20);
  pop();
}

function drawBody() {
  push();
  translate(0, 20, 0);
  noStroke();
  fill(150, 255, 150);
  sphere(75);
  translate(0, -90, 0);
  sphere(70);

  // highlights
  fill(255, 255, 255, 90);
  translate(-30, -10, 50);
  sphere(10);

  pop();
}

function drawPetiole() {
  push();
  translate(0, -110, 20);  // start from top of head
  rotateX(radians(200));  // slight backward tilt
  fill(100, 200, 100);
  noStroke();
  cylinder(8, 100);       
  pop();
}


function drawLeaf(x, y, z, flip = 1, scaleFactor = 1) {
  push();
  translate(x, y, z);
  rotateZ(flip * QUARTER_PI); // tilt outwards
  rotateX(HALF_PI - QUARTER_PI * 0.5); // add a slight backward curve
  scale(flip * scaleFactor, scaleFactor); // flip + scale


  fill(100, 200, 100);
  stroke(60, 160, 80);
  strokeWeight(1);

  beginShape();
  vertex(0, 0);
  bezierVertex(-35, -90, -70, -10, 0, 0); // left curve
  endShape(CLOSE);

  // Center vein
  stroke(90, 150, 80);
  strokeWeight(2);
  line(0, 0, -40, -29);
  pop();
}

function drawFace() {
  push();
  translate(0, -100, 70);

  stroke(10);
  strokeWeight(2);
  noFill();
  arc(-20, 0, 20, 10, 0, PI);
  arc(20, 0, 20, 10, 0, PI);

  // blush
  noStroke();
  fill(255, 200, 200, 100);
  ellipse(-35, 10, 15, 8);
  ellipse(35, 10, 15, 8);

  // smile
  stroke(30);
  noFill();
  arc(0, 15, 30, 10, 0, PI);

  pop();
}
function drawWarmGradientBackground() {
  push();
  translate(0, 0, -500); // move far behind the scene
  noStroke();
  beginShape();
  fill(250, 230, 200); // top (warm cream)
  vertex(-width, -height);
  vertex(width, -height);
  fill(210, 170, 130); // bottom (warm brownish-orange)
  vertex(width, height);
  vertex(-width, height);
  endShape(CLOSE);
  pop();
}


function drawInstructionFrame() {
  push();
    // move to a spot on the back wall
    translate(-width * 0.25, -height * 0.25, -490);

    // outer frame (landscape)
    fill(120, 80, 40);
    box(430, 300, 10);

    // inner “paper” canvas inset
    push();
      translate(0, 0, 7);
      fill(255, 245, 220);
      box(390, 260, 2);
    pop();

    // text
    push();
      translate(0, 0, 12);
      fill(60, 40, 30);
      textSize(40);
      textAlign(CENTER, CENTER);
      text("Make the Plant Happy\n- Press to play Music\n- Control the Lighting\n- Pet the plant", 0, 0);
    pop();
  pop();
}

Functionality Flow

1. 1. Startup

      • Arduino sends MUSIC_OFF. p5.js opens port, waits for START (button press on avatar).

   2. Interaction

      • Touch: Plant touch → TOUCH:<value> → leaf animation.

      • Light: Ambient changes → LAMP:<0–255> → lamp glow intensity.

      • Music: Physical push-button → MUSIC_ON/MUSIC_OFF → background music.

Challenges & Solutions

• • Serial Fragmentation: Split messages were mis-parsed → switched to single println("TOUCH:"+value) calls.

  • Sensor Hysteresis: Capacitive values vary → implemented high/low thresholds to avoid flicker.

  • Full-Screen Behavior: Music wouldn’t play in full screen → tied audio unlock to fullscreenchange event.

User Testing:

During user testing, the visuals weren’t clear in indicating what the project was for. The user mentioned that instructions to interact weren’t needed, but also that the purpose of the project wasn’t directly clear.

For technical improvements I adjusted the interval averaging to 8 samples, which improved stability but slightly delayed responsiveness. I also tested different tempoMul ranges (originally 0.2–3) and settled on 0.5–2 to keep the music and visuals within a comfortable range.

User Testing Video

For my final project, I developed an interactive audiovisual experience that transforms a user’s heart rate into dynamic music and visuals, creating a biofeedback-driven art piece. Built using p5.js, the project integrates Web Serial API to read pulse data from an Arduino-based heart rate sensor, generating a musical chord progression (Cmaj7-Am7-Fmaj7-G7) and WebGL visuals (swirling ellipses and a point field) that respond to the calculated BPM (beats per minute). Initially, I proposed a STEM-focused feature to educate users about heart rate, but I pivoted to make the music and visuals adjustable via mouse movements, allowing users to fine-tune the tempo and visual intensity interactively.

The heart rate sensor sends pulse data to p5.js, which calculates BPM to adjust the music’s tempo (chord changes, bass, kick drum) and visual animation speed. Mouse X position controls a tempo multiplier (tempoMul), scaling both music and visuals, while BPM directly influences animation speed and audio effects. The visuals were inspired by p5.js data visualization examples from class, particularly those using WebGL to create dynamic, responsive patterns. The project aims to create a meditative, immersive experience where users see and hear their heartbeat in real-time, with interactive control over the output.

Hardware:

• Heart Rate Sensor: I used a PulseSensor connected to an Arduino Uno, wired directly to analog pin A0 without a resistor to simplify the circuit. The sensor is not attached and can freely be interacted with.
• Fabrication: I fit the arduino cable and heart rate sensor through the cardboard sparkfun box. I avoided a finger strap due to wire fragility and inconsistent pressure, opting for a loose finger placement method.
• Challenges: Direct wiring without a resistor may have increased noise in the pulse signal, requiring software filtering in the Arduino code. Loose finger contact sometimes caused erratic readings, so I adjusted the threshold and added a refractory period to stabilize detection.

The p5.js sketch reads serial data from the Arduino, calculates BPM, and updates music and visuals. Initially, I tried processing raw analog values in p5.js, but noise made it unreliable. After extensive debugging (around 10 hours), I modified the Arduino code to send pre-processed BPM estimates as integers (30–180 range), which streamlined p5.js logic. The mouse-driven tempoMul (mapped from mouse X) scales chord timing, note durations, and visual motion, replacing the STEM feature with an interactive control mechanism.

A significant challenge was balancing real-time BPM updates with smooth visualization. The visuals use the latest BPM, which can make animations appear jumpy if BPM changes rapidly. I averaged BPM over 8 intervals to smooth transitions, but this introduced a slight lag, requiring careful tuning. Serial communication also posed issues: the Web Serial API occasionally dropped connections, so I added robust error handling and a “Connect & Fullscreen” button for reconnection.

Arduino Code (Heart Rate Sensor):

#define PULSE_PIN A0
void setup() {
  Serial.begin(9600);
  pinMode(PULSE_PIN, INPUT);
}
void loop() {
  int pulseValue = analogRead(PULSE_PIN);
  if (pulseValue > 400 && pulseValue < 800) { // Basic threshold
    Serial.println(pulseValue);
  } else {
    Serial.println("0");
  }
  delay(10);
}

Images

The seamless integration of heart rate data into both music and visuals is incredibly rewarding. Seeing the ellipses swirl faster and hearing the chords change in sync with my heartbeat feels like a direct connection between my body and the art. I’m also proud of overcoming the Arduino noise issues by implementing software filtering and averaging, which made the BPM calculation robust despite the direct wiring.

Challenges Faced:

• Arduino Code: The biggest hurdle was getting reliable pulse detection without a resistor. The direct wiring caused noisy signals, so I spent hours tuning the THRESHOLD and REFRACTORY values in the Arduino code to filter out false positives.
• BPM Calculation: Calculating BPM in p5.js required averaging intervals (intervals array) to smooth out fluctuations, but this introduced a trade-off between responsiveness and stability. I used a rolling average of 8 intervals, but rapid BPM changes still caused slight visual jumps.
• Visualization Balance: The visuals update based on the latest BPM, which can make animations feel abrupt if the heart rate spikes. I tried interpolating BPM changes, but this slowed responsiveness, so I stuck with averaging to balance real-time accuracy and smooth motion.
• p5.js Visualization: Adapting WebGL examples from class to respond to BPM was tricky. The math for scaling ellipse and point field motion (visualSpeed = (bpm / 60) * tempoMul) required experimentation to avoid jittery animations while staying synchronized with the music.
• Serial Stability: The Web Serial API occasionally dropped connections, especially if the Arduino was disconnected mid-session. Robust error handling and the reconnect button mitigated this, but it required significant testing.

Possible Improvements:

• Smoother BPM Transitions: Implement linear interpolation to gradually transition between BPM values, reducing visual jumps while maintaining real-time accuracy.
• Dynamic Color Mapping: Map BPM to the hue of the ellipses or points (e.g., blue for low BPM, red for high), enhancing the data visualization aspect.
• Audio Feedback: Add a subtle pitch shift to the pad or bass based on BPM to make tempo changes more audible.
• Sensor Stability: Introduce a clip-on sensor design to replace loose finger placement, improving contact consistency without fragile wires.

Reflection + Larger Picture:

This project explores the intersection of biofeedback, art, and interactivity, turning an invisible biological signal (heart rate) into a tangible audiovisual experience. It highlights the potential of wearable sensors to create personalized, immersive art that responds to the user’s physical state. The data visualization component, inspired by p5.js examples from class (e.g., particle systems and dynamic patterns), emphasizes how abstract data can be made expressive and engaging. Beyond art, the project has applications in mindfulness, where users can regulate their heart rate by observing its impact on music and visuals, fostering a deeper connection between the body and mind.

Finally, we reached the end of the semester, and with it came the submission of the final project. I had decided to make a Simon-Says style game, using lights and buzzers to interact with the user and test their recall skills. An interesting thing I’ve found throughout this course that I seem to really enjoy memory-style games, as with my midterm, and now this.

Like with my midterm, I began by creating a prototype to make sure I got the basic features down before I integrated any fancy features or graphics. Looking back on it now, the initial version worked, but it just looked, in simple terms, boring.

The initial gameplay didn’t feel like anything I would be excited to play at all.

‘

The initial wiring setup didn’t inspire much confidence either.

But that’s the great part of a prototype. It didn’t need to look good, so long as it functioned well. I was able to nail down the game states with the prototype, then I began working on graphics.

I wanted to go for a retro style with my game, and so I tried to make the backgrounds and the board match a cohesive neon-arcade-esque vibe.

In the end, we arrived at the final submission. I ended up soldering many of the wires inside my box to hide them as much as possible, because attractive things work better (reading reference!).

After carrying out user-testing, I ended up integrating more features within my code, including an instructions screen, and more interactivity between the buttons (a shorter debounce delay, the buttons lighting up, etc).

And here we are. With the final submission! I had  both  a great  and  frustrating  experience  making  it,  but I’m  really  glad  with the  final result.

Schematic of my circuit (though I used arcade buttons)

// establishing variables
const int buttonPins[] = {2, 3, 4, 5}; // Yellow, green, blue, and red
const int ledPins[] = {8, 9, 10, 11}; // Yellow, green, blue, and red
const int buzzerPin = 6; // Buzzer set on pin 6

bool ledBlinking = false; // Checks whether the LEDs are blinking or not
unsigned long lastBlinkTime = 0; // Tracks when the LEDs were last blinked
bool blinkState = false; // Toggles between on and off when the LEDs are blinking

void setup() {
  // Setting up serial communication
  Serial.begin(9600); // Buad rate of 9600
  for (int i = 0; i < 4; i++) {
    // Setting the pin modes for the buttons, LEDs, and buzzer
    pinMode(buttonPins[i], INPUT_PULLUP);
    pinMode(ledPins[i], OUTPUT);
  }
  pinMode(buzzerPin, OUTPUT);
}

void loop() {
  // Handle blinking mode
  if (ledBlinking && millis() - lastBlinkTime > 500) { 
    blinkState = !blinkState; // Alternates between the LED being on and off every 500ms
    for (int i = 0; i < 4; i++) {
      if (blinkState) {
        digitalWrite(ledPins[i], HIGH); // Turn the LED on if blinkState is true
        } 
      else {
        digitalWrite(ledPins[i], LOW); // Turn the LED off if blinkState is false
        }
       }
    lastBlinkTime = millis();
  }

  // Check button presses
  for (int i = 0; i < 4; i++) {
    if (digitalRead(buttonPins[i]) == LOW) {
      Serial.println(buttonPins[i]); // Send button pin number to p5
      delay(100); // Debounce delay
    }
  }

  // Handle serial input from p5
  if (Serial.available()) {
    String command = Serial.readStringUntil('\n');
    command.trim();

    if (command.startsWith("ALL")) { // if the p5 command sends "ALL", all LEDs must be on
      int mode = command.substring(3).toInt();
      handleAllLEDs(mode);
    } 
    else if (command == "WRONG") { // if the p5 command sends "WRONG", play the sound
      tone(buzzerPin, 100, 500); // Wrong answer sound
    } 
    else {
      int pin = command.toInt(); // lights up the corresponding LED and plays the sound
      if (pin >= 8 && pin <= 11) {
        playColorFeedback(pin);
      }
    }
  }
}

// Turns on the LED corresponding to the button, and plays the sound
void playColorFeedback(int pin) {
  digitalWrite(pin, HIGH);
  playToneForPin(pin);
  delay(300);
  digitalWrite(pin, LOW);
  noTone(buzzerPin);
}

// Plays a specific tone based on the button pressed
void playToneForPin(int pin) {
  switch (pin) {
    case 8: tone(buzzerPin, 262); break; // Yellow is C4
    case 9: tone(buzzerPin, 330); break; // Green is E4
    case 10: tone(buzzerPin, 392); break; // Blue is G4
    case 11: tone(buzzerPin, 523); break; // Red is C5
  }
}

void handleAllLEDs(int mode) {
  ledBlinking = false;
  for (int i = 0; i < 4; i++) {
    digitalWrite(ledPins[i], LOW); // the LEDs are off
  }

  if (mode == 1) {
    for (int i = 0; i < 4; i++) {
      digitalWrite(ledPins[i], HIGH); // if the mode is 1, it turns on all the LEDs
    }
  } else if (mode == 2) { // if the mode is 2, it blinks the LEDs
    ledBlinking = true;
    lastBlinkTime = millis();
  }
}

My Arduino Code

Link to the full screen version

Thanks for a great semester!

Concept

The Smart House System is an interactive physical computing project that simulates features of an intelligent home using Arduino UNO and p5.js. The system includes:

• A smart parking assistant that detects cars entering and exiting, and updates available parking slots automatically.

• A light automation system that turns on indoor lights when it gets dark, based on ambient light readings.

• A real-time dashboard and voice announcer, implemented in p5.js, that visualizes the system state and speaks updates aloud using p5.speech.

This system provides a fun and engaging way to demonstrate real-world home automation, combining sensors, outputs, and visual/voice feedback for user interaction.

Interaction Demo

IMG_9078

How the Implementation Works

The system uses ultrasonic distance sensors to detect when a vehicle is near the entry or exit of the parking area. A servo motor simulates the gate that opens when a car arrives and parking is available.

A photoresistor (LDR) detects light levels to automatically turn on five LEDs that simulate indoor lighting when it gets dark.

All event messages from Arduino are sent to a p5.js sketch over web serial. The browser-based sketch then:

• Displays the parking status

• Shows light status

• Uses p5.speech to speak real-time messages like “Parking is full!” or “Lights are now on!”

Interaction Design

The project is designed for simple, touchless interaction using real-world analog sensors:

• Bringing your hand or an object close to the entry sensor simulates a car arriving. If space is available, the gate opens, the slot count is reduced, and a voice announces the update.

• Moving your hand in front of the exit sensor simulates a car leaving, increasing the parking availability.

• Covering the LDR sensor simulates nighttime — lights automatically turn on, and the system announces it.

• The p5.js dashboard shows real-time status and acts as an interactive voice feedback system.

Arduino Code

The Arduino UNO is responsible for:

• Reading two ultrasonic sensors for car entry/exit

• Reading the photoresistor (LDR) for light level

• Controlling a servo motor for the gate

• Controlling 5 indoor LEDs

• Sending status messages to the p5.js sketch over serial

Code Overview:

• Starts with 3 available parking slots

• Gate opens and slot count decreases when a car is detected at entry

• Slot count increases when a car exits

• Indoor lights turn on when light level drops below a threshold

• Sends messages like car_entry, car_exit, parking_full, lights_on, lights_off, and parking_spots:X

#include <Servo.h>

// Ultrasonic sensor pins
#define trigEntry 2
#define echoEntry 3
#define trigExit 4
#define echoExit 5

// Servo motor pin
#define servoPin 6

// LED pins
int ledPins[] = {7, 8, 9, 10, 11};

// Light sensor pin
#define lightSensor A0

Servo gateServo;
int Slot = 3; // Initial parking spots

void setup() {
  Serial.begin(9600);

  // Ultrasonic sensors
  pinMode(trigEntry, OUTPUT);
  pinMode(echoEntry, INPUT);
  pinMode(trigExit, OUTPUT);
  pinMode(echoExit, INPUT);

  // LED pins
  for (int i = 0; i < 5; i++) {
    pinMode(ledPins[i], OUTPUT);
  }

  // LDR analog input
  pinMode(lightSensor, INPUT);

  // Servo
  gateServo.attach(servoPin);
  gateServo.write(100); // Gate closed
}

void loop() {
  int entryDistance = getDistance(trigEntry, echoEntry);
  int exitDistance  = getDistance(trigExit, echoExit);
  int lightValue    = analogRead(lightSensor); // 0 (dark) to 1023 (bright)

  Serial.print("Entry: "); Serial.print(entryDistance);
  Serial.print(" | Exit: "); Serial.print(exitDistance);
  Serial.print(" | Light: "); Serial.print(lightValue);
  Serial.print(" | Slots: "); Serial.println(Slot);

  // ===== Car Entry Logic =====
  if (entryDistance < 10 && Slot > 0) {
    openGate();
    Slot--;
    Serial.println("car_entry");
    Serial.print("parking_spots:");
    Serial.println(Slot);
    delay(2000);
    closeGate();
  }
  // ===== Parking Full Logic =====
  else if (entryDistance < 10 && Slot == 0) {
    Serial.println("parking_full");
    delay(1000); // Prevent spamming the message
  }

  // ===== Car Exit Logic =====
  if (exitDistance < 10 && Slot < 3) {
    openGate();
    Slot++;
    Serial.println("car_exit");
    Serial.print("parking_spots:");
    Serial.println(Slot);
    delay(2000);
    closeGate();
  }

  // ===== Light Control (5 LEDs) =====
  if (lightValue < 900) { // It's dark
    for (int i = 0; i < 5; i++) {
      digitalWrite(ledPins[i], HIGH);
    }
    Serial.println("lights_on");
  } else {
    for (int i = 0; i < 5; i++) {
      digitalWrite(ledPins[i], LOW);
    }
    Serial.println("lights_off");
  }

  delay(500);
}

// ===== Gate Functions =====
void openGate() {
  gateServo.write(0);
  delay(1000);
}

void closeGate() {
  gateServo.write(100);
  delay(1000);
}

// ===== Distance Sensor Function =====
int getDistance(int trigPin, int echoPin) {
  digitalWrite(trigPin, LOW);
  delayMicroseconds(2);
  digitalWrite(trigPin, HIGH);
  delayMicroseconds(10);
  digitalWrite(trigPin, LOW);
  long duration = pulseIn(echoPin, HIGH);
  int distance = duration * 0.034 / 2;
  return distance;
}

Circuit Schematic

p5.js Code and Dashboard

The p5.js sketch:

• Uses the p5.webserial library to connect to Arduino

• Uses p5.speech for voice announcements

• Displays a dashboard showing the number of available parking slots

• Shows the indoor light status using a colored circle

The voice announcements are fun and slightly humorous, e.g.:

“A wild car appears!”
“Uh-oh! Parking is full.”
“It’s getting spooky in here… turning the lights on!”

The sketch uses a say() wrapper function to safely trigger voice output in Chrome after the user clicks once.

Code Highlights:

• Automatically resumes Chrome’s audio context

• Waits for user interaction before enabling speech

• Processes serial messages one line at a time

• Provides a Connect/Disconnect button for user control

Arduino and p5.js Communication

The communication uses Web Serial API via p5.webserial:

• Arduino sends messages like "car_entry\n", "lights_on\n", etc.

• p5.js reads each line, processes it, updates the dashboard, and speaks it out loud

• A connect button in the sketch allows users to select their Arduino port manually

• All communication is unidirectional: Arduino → p5.js

What I’m Proud Of

• Fully working sensor-triggered voice feedback via p5.js — makes the system feel alive

• Smooth parking logic with entry and exit detection

• Integration of multiple Arduino components (servo, LDR, LEDs, ultrasonic)

• An intuitive UI that works both visually and with voice

• Reliable browser-based connection using modern Web Serial

Areas for Future Improvement

• Add a screen-based parking spot display (e.g., 7-segment or OLED)

• Use non-blocking code in Arduino with millis() instead of delay()

• Make a mobile-responsive version of the dashboard UI

• Add a security camera feed or face detection in p5.js

• Improve the servo animation to be smoother and time-synced

• Add a buzzer or alert when parking is full