Concept

For my final project, I created an interactive art that aims to celebrate friendship and togetherness. The main idea of the art is that a tree branch appears from the 2 sides of the screen and grows towards each other; once they connect, a flower blooms. Consistent with the theme of togetherness, the project requires 2 people to hold hands together and the strength of their hold influences how fast the branches grow – force sensitive resistor (FSR) is sewed into a glove, which one person puts on and holds the other person’s hand with. With the free hands, each person puts a finger on the temperature sensor and the resulting reading from each sensor is mapped to a color (darkish pink = warm, light yellow = cool) and lerped together to generate a live color gradient for the background.

Video demonstration

IM showcase user interaction

Implementation

I wanted the user interface to be minimal but pretty, so the homepage, the instructions page and the main sketch are all designed to have minimal text and soft color palette.

homepage

instructions page

main sketch (after the branches have connected together)

At the end of the sketch after the branches connect, a quote about friendship and togetherness is randomly chosen from a text file and appears at the bottom of the screen.

initial hand sketches

Interaction design

As the project is about friendship and togetherness, the interaction is designed to involve 2 people’s participation. I was initially using alligator clips for connecting the temperature sensors, but they felt very frail to the hands and got disconnected easily, so I soldered wires into both of the temperature sensors and taped them on top of a cardboard to make sure they are in place. As for the FSR, my very first idea was to place it inside a soft thin object so that the users can put it in between their palms and then hold hands. I could not think of a suitable and meaningful object to place the FSR, not to mention realized it would be strange to ask people to put a random object in between their holding hands. The best option seemed to be using a glove and attaching the FSR inside it. I tested different ways of holding hands and figured that the location where the contact point is highest when holding hands with another person is the purple area in the picture below. This is where I placed the FSR and sewed it on with a thread. During the IM showcase, some people held hands very very faintly, so I had to jump in and demonstrate how they should hold hands to ensure that force was being applied to the FSR. It made me realize that it may have been helpful if there was an image depiction of proper hand holds in the instructions page.

• Arduino code

• int fsrPin = A0;
  int tempSensorPin = A1;
  int tempSensor2Pin = A2;
  int fsrReading;    // variable to store the FSR reading
  float fsrVoltage;     // variable to store the voltage value
  float tempSensorVoltage;
  float tempSensorReading;
  float temperature;
  float tempSensor2Voltage;
  float tempSensor2Reading;
  float temperature2;
  
  void setup() {
    // Start serial communication so we can send data
    // over the USB connection to our p5js sketch
    Serial.begin(9600);
  
    // We'll use the builtin LED as a status output.
    // We can't use the serial monitor since the serial connection is
    // used to communicate to p5js and only one application on the computer
    // can use a serial port at once.
    pinMode(LED_BUILTIN, OUTPUT);
  }
  
  void loop() {
    // read the FSR value (0–1023)
    fsrReading = analogRead(fsrPin); 
    fsrVoltage = fsrReading * (5.0 / 1023.0);
  
    tempSensorReading = analogRead(tempSensorPin);
    // convert the analog reading to a voltage (0–5V)
    tempSensorVoltage = tempSensorReading * (5.0 / 1023.0);
    temperature = (tempSensorVoltage - 0.5) * 100.0;
  
    tempSensor2Reading = analogRead(tempSensor2Pin);
    // convert the analog reading to a voltage (0–5V)
    tempSensor2Voltage = tempSensor2Reading * (5.0 / 1023.0);
    temperature2 = (tempSensor2Voltage - 0.5) * 100.0;
  
    // send the FSR reading and 2 temperatures to p5.js
    Serial.print(fsrReading);
    Serial.print(","); 
    Serial.print(temperature); 
    Serial.print(","); 
    Serial.println(temperature2); 
  
    // blink the built-in LED to show activity
    digitalWrite(LED_BUILTIN, HIGH);
    delay(100);
    digitalWrite(LED_BUILTIN, LOW);
  
    delay(200);
  }

  Circuit

•  p5.js code

Link to fullscreen sketch

// interface variables
let homepageImg, backButtonImg, infoButtonImg, infoButtonWhiteImg, gloveImg, sensorImg;
let ambienceSound, endingSound;
let backButtonX = 20;
let backButtonY = 20;
let backButtonW = 20;
let backButtonH = 20;
let infoButtonX = 920;
let infoButtonY = 20;
let infoButtonW = 20;
let infoButtonH = 20;
let fontNunito, fontNunitoLight;
// transparency variable for the fading start text
let alphaValue = 0;
// variable used for the fade effect of the start text
let fadeDirection = 1;
let c;
let startMsgColor = "#c27e85";
let titleColor = "#745248";
let instructionsBoxColor = "#738059";
let backgroundColor = "#F0EBE5";
let vizOver = true;
let vizStarted = false;
let instructionsOn = false;
let endingSoundPlayed = false;
let ambienceSoundPlaying = false;
let homePage;
let viz;

// illustration variables
let branchGrowth = 0.1;
let maxGrowth;
let leafGrowth;
// for tracking if the branches have connected at the center
let connected = false;
// for storing the coordinates of left branch points
let leftPoints = [];
// for storing the coordinates of right branch points
let rightPoints = [];
let leafImg;
let leadImgFlipped;
let temperature_1;
let temperature_2;
let quotes = [];
let chosenQuote = "";

// arduino variables
let fsrValue = 0;
let temperature = 0;
let temperature2 = 0;
let arduinoConnected = false;

// map the sensor readings to a different range
function mapVals(sensorType, val) {
  // map temperature to colors
  if (sensorType == "temperature") {
    // lowest reading on the temp sensor was 18
    // highest reading on the temp sensor was 29
    // so these are mapped to a number on a wider scale
    // to ensure that the background color changes are more visible/stronger
    return map(val, 18, 30, 1, 38);
  // map fsr reading to branchGrowth
  } else if (sensorType == "fsr") {
    // bound the branch growth to maximum 0.5 to make sure that
    // the visualization doesn't end too quickly
    return map(val, 0, 1023, 0.1, 0.5);
  }
}

function preload() {
  homepageImg = loadImage("/assets/homepage.png");
  backButtonImg = loadImage("/assets/left.png");
  infoButtonImg = loadImage("/assets/information.png");
  infoButtonWhiteImg = loadImage("/assets/information_white.png");
  fontNunito = loadFont("/fonts/Nunito-Medium.ttf");
  fontNunitoLight = loadFont("/fonts/Nunito-Light.ttf");
  leafImg = loadImage("/assets/nature.png");
  leafImgFlipped = loadImage("/assets/nature_flipped.png");
  flowerImg = loadImage("/assets/flower.png");
  gloveImg = loadImage("/assets/glove.png");
  sensorImg = loadImage("/assets/sensor.png");
  ambienceSound = loadSound("/sounds/ambience_long.mp3");
  endingSound = loadSound("/sounds/ending.mp3");
  quotes = loadStrings("/assets/quotes.txt");
  heartImg = loadImage("/assets/heart.png");
}

function setup() {
  createCanvas(960, 540);
  background(0);

  homePage = new HomePage();
  viz = new Visualization();

  // each branch can only grow for a length that's half of the screen
  maxGrowth = width / 2;
  chosenQuote = random(quotes);
}

class HomePage {
  constructor() {
  }

  display() {
    image(homepageImg, 0, 0, width, height);
    image(infoButtonImg, infoButtonX, infoButtonY, infoButtonW, infoButtonH);
    // fade effect on the "Press Enter to start" text
    // by varying the transparency
    alphaValue += fadeDirection * 3;
    if (alphaValue >= 255 || alphaValue <= 0) {
      fadeDirection *= -1;
    }

    push();
    textAlign(CENTER);
    textFont(fontNunito);
    textSize(16);
    c = color(startMsgColor);
    c.setAlpha(alphaValue);
    fill(c);
    // ask user to select serial port first
    if (!arduinoConnected) {
        text("press SPACE to select serial port", width / 2, 500);
    // once serial port selected, show a different text
    } else {
        text("press ENTER to start", width / 2, 500);
    }
    pop();
  }

  showInstructions() {
    this.display();
    let c = color(instructionsBoxColor);
    c.setAlpha(245);
    fill(c);
    noStroke();
    rect(0, 0, width, height);
    image(
      infoButtonWhiteImg,
      infoButtonX,
      infoButtonY,
      infoButtonW,
      infoButtonH
    );
    
    // instructions text
    push();
    textAlign(CENTER);
    textFont(fontNunito);
    textSize(16);
    fill(color(backgroundColor));
    text("h o w    i t    w o r k s", width / 2, 100);
    text(
      "1. have a friend next to you and stand facing each other",
      width / 2,
      200
    );
    text("2. put a finger each on the temperature sensor", width / 2, 250);
    text(
      "3. with the free hands: one person wears the glove and holds the friend's hand",
      width / 2,
      300
    );
    text("4. watch the tree branches bloom!", width / 2, 350);
    pop();
  }
}

class Visualization {
  constructor() {
  }
  
  start() {
    // play the ambience sound if it's not playing
    if (!ambienceSoundPlaying) {
      ambienceSound.play();
      // use a boolean variable to ensure it's only played once
      ambienceSoundPlaying = true;
    }
    
    // stop the sound if visualization is over
    if (connected || vizOver) {
      ambienceSound.stop();
    }
        
    noStroke();
    
    // map the temp sensor readings
    temperature_1 = mapVals("temperature", temperature);
    temperature_2 = mapVals("temperature", temperature2);
    // map each person's temperature to colors
    let color1 = mapTempToColor(temperature_1);  // color for person 1
    let color2 = mapTempToColor(temperature_2);  // color for person 2

    // smooth gradient blending between the two temperatures
    for (let x = 0; x < width; x++) {
      // lerp factor based on x position (left to right transition)
      let lerpFactor = map(x, 0, width, 0, 0.5);

      // blend between color1 and color2
      let col = lerpColor(color1, color2, lerpFactor);

      // apply the color vertically to the canvas
      stroke(col);
      line(x, 0, x, height);  // vertical lines for smooth gradient
    }

    // go back to homepage button
    image(backButtonImg, backButtonX, backButtonY, backButtonW, backButtonH);
    
    // map fsr reading
    let growthSpeed = mapVals("fsr", fsrValue);
    // use the mapped fsr value to grow the branch
    branchGrowth += growthSpeed;
    
    if (branchGrowth < maxGrowth) {
      // introduce noise to left branch
      let noiseOffsetLeftBranch = branchGrowth * 0.01;
      // introduce noise to right branch
      // slightly change the noise offset so that branches don't grow symmetrically and instead looks different
      let noiseOffsetRightBranch = branchGrowth * 0.01 + 1000;
      
      // generate x, y coordinates for points for both branches
      let yLeft = height / 2 + map(noise(noiseOffsetLeftBranch), 0, 1, -40, 40);
      let yRight =
        height / 2 + map(noise(noiseOffsetRightBranch), 0, 1, -40, 40);
      let xLeft = branchGrowth;
      let xRight = width - branchGrowth;
      
      // once the branches are nearing the center, reduce the noise and make them remain near the horizontal middle of the canvas to ensure they connect everytime
      let easingZone = 30;
      if (branchGrowth > maxGrowth - easingZone) {
        let amt = map(branchGrowth, maxGrowth - easingZone, maxGrowth, 0, 1);
        yLeft = lerp(yLeft, height / 2, amt);
        yRight = lerp(yRight, height / 2, amt);
      }
      
      // add the points to the array
      leftPoints.push({
        pos: createVector(xLeft, yLeft),
        // randomly decide if the leaf will be growing on the top or on the bottom
        flip: int(random(2))
});      
      rightPoints.push({
        pos: createVector(xRight, yRight),
        flip: int(random(2))
});     
    } else if (!connected) {
      connected = true;
      
      // play the ending sound if the branches have connected
      if (!endingSoundPlayed) {
        endingSound.play();
        endingSoundPlayed = true;
      }
    }

    // draw branches
    push();
    strokeWeight(3);
    stroke(110, 70, 40); // brown
    noFill();

    beginShape();
    // draw the left branch
    for (let ptObj of leftPoints) {
      let pt = ptObj.pos;
      vertex(pt.x, pt.y);
      
      // for every 75th x coordinate, draw an image of a leaf
      // position of leaf is stored in the flip attribute
      if (int(pt.x) % 75 == 0) {
        if (ptObj.flip === 0) {
          image(leafImgFlipped, pt.x, pt.y - 2, 40, 40);
        } else {
          image(leafImg, pt.x, pt.y - 37, 40, 40);
        }
      }
    }
    endShape();
    
    // draw the right branch
    beginShape();
    for (let ptObj of rightPoints) {
      let pt = ptObj.pos;
      vertex(pt.x, pt.y);

      if (int(pt.x) % 75 == 0) {
        if (ptObj.flip === 0) {
          image(leafImgFlipped, pt.x, pt.y - 2, 40, 40);
        } else {
          image(leafImg, pt.x, pt.y - 37, 40, 40);
        }
      }
    }
    endShape();
    pop();
    
    let leftEnd = leftPoints[leftPoints.length - 1].pos;
    let rightEnd = rightPoints[rightPoints.length - 1].pos;

    let d = dist(leftEnd.x, leftEnd.y, rightEnd.x, rightEnd.y);
    
    // determine if the branches have connected by finding the distance between the 2 branch end points and checking if it's less than 5
    if (d < 5) {
      push();
      rectMode(CENTER);
      image(flowerImg, leftEnd.x - 15, (leftEnd.y - 30), 80, 80);
      
      // console.log(chosenQuote);
      
      // show the quote
      if (chosenQuote !== "") {
        textAlign(CENTER, CENTER);
        textFont(fontNunito);
        textSize(20);
        fill(titleColor);
        text(chosenQuote, width / 2, height - 80);
      }
      
      // heart image at the bottom of the quote
      image(heartImg, width / 2, height - 60, 40, 40);
      pop();
    }
  }
}

function draw() {
  if (instructionsOn) {
    homePage.showInstructions();
  } else if (!vizStarted || vizOver) {
    homePage.display();
  } else {
    viz.start();
  }
  // print(mouseX + "," + mouseY);
}

// map temperature to color which will be used to control the color of the background
function mapTempToColor(temp) {
  let coolColor = color("#F0EBE5");
  let warmColor = color("#dea0a6"); 
  
  // lerp the light yellow color with the dark pink color based on the temperature
  return lerpColor(coolColor, warmColor, map(temp, 1, 38, 0, 1));
}

function readSerial(data) {
  ////////////////////////////////////
  //READ FROM ARDUINO HERE
  ////////////////////////////////////

  if (data != null) {
    let fromArduino = split(trim(data), ",");
    // if the right length, then proceed
    if (fromArduino.length == 3) {
      fsrValue = int(fromArduino[0]);
      temperature = int(fromArduino[1]);
      temperature2 = int(fromArduino[2]);
    }
    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    // let sendToArduino = left + "," + right + "\n";
    // writeSerial(sendToArduino);
  }
}

function keyPressed() {
  // if the enter key is pressed, reset state and start the visualization
  if (keyCode === ENTER) {
    if (vizOver && !instructionsOn) {
      // reset visualization state
      branchGrowth = 20;
      leftPoints = [];
      rightPoints = [];
      connected = false;
      ambienceSoundPlaying = false;
      endingSoundPlayed = false;
      chosenQuote = random(quotes);
      vizStarted = true;
      vizOver = false;
    }
  }
  
  // if space is pressed, setup serial communication
  if (key == " ") {
    setUpSerial();
    arduinoConnected = true;
  }
}

function mousePressed() {
  // if mouse is pressed on the < button, go back to the homepage
  if (
    mouseX >= backButtonX &&
    mouseX <= backButtonX + backButtonW &&
    mouseY >= backButtonY &&
    mouseY <= backButtonY + backButtonH
  ) {
    vizOver = true;
    ambienceSound.stop();
    vizStarted = false; // Reset visualization state when returning to homepage
    console.log("back button pressed");
  // if mouse if pressed on the information button, show the instructions page
  } else if (
    mouseX >= infoButtonX &&
    mouseX <= infoButtonX + backButtonW &&
    mouseY >= infoButtonY &&
    mouseY <= infoButtonY + backButtonH
  ) {
    instructionsOn = !instructionsOn;
    console.log("info button pressed");
  }
}

• Communication between Arduino and p5.js

3 readings from Arduino (2 temperature sensors and 1 FSR) are sent in a X,Y,Z format with commas separating them to p5.js using serial communication. The p5.js code then maps these values to their respective values for color and speed. I could not think of meaningful data to send from p5.js to Arduino that doesn’t just send data for the sake of having a two-way communication. I considered using the LCD to show text (names of the users, reading values etc), buzzer to play music, but as professor mentioned in our chat on the initial evaluation day, these things can be done on the screen as well and does specifically require to be implemented on the hardware. Thus, I decided to continue using Arduino as input only and I think it worked well at the end.

Reflection and Proud Moments

I like how the p5.js interface and the hardware interface ended up looking from a visual standpoint. I put an effort into choosing the visual assets, the fonts and the different colors I used across the sketch to create a coherent visual experience that is simple yet visually pleasing to look at, and I was very happy when people complimented on it! I decorated the cardboard casing covering the Arduino to match the artwork by drawing leaves and flowers, and I think that turned out successfully as well.

For the p5.js artwork, it was tricky to connect the 2 branches every time. The way the drawing of the branches work is that a variable, called branchGrowth, increases every frame, and its speed is determined by the FSR reading. For each new value of branchGrowth, the code calculates a new point for the left and right branches. To ensure that the branches look organic and not like a straight line, Perlin noise (noise(…)) is introduced to create smooth vertical variation in the branches. As each point of the branch is generated randomly (to a certain degree), unless the last segment of each branch was somehow smoothed out and maintained near the horizontal middle, the 2 branches always ended up at different heights. To solve this issue, I eased the y values of both branches toward the center as they approached close each other. This helped to make the branch growth to appear organic but also controlled so that they connected every time.

Overall, I had a lot of fun working on this project and it was a genuinely rewarding experience watching people’s reactions and seeing how they interact with the piece (including finding some loopholes!). I am glad I stuck with the interactive artwork idea and the theme of togetherness, because I got to see some cute interactions of friends who tried out my project <3

Future Improvements

If I were to continue with the project and develop it further, I think I would explore ways of adding more physical components (other than sensors) that people could interact with. One example could be a flower that blooms when the branches meet by using a servo motor. I would also make improvements to the instructions page because it seemed like people didn’t easily see that there was an info button. Even when they did see the button and press on it, there was a tendency for people to take a quick glance at it and head straight into starting the sketch.

Final Project Documentation

Concept:

“Olive Tree Stories” is an interactive storytelling project inspired by the deep cultural roots of olive tree’s in Palestinian heritage. I wanted my project to be meaningful and leave a strong impact on the user, so i drew inspiration from my own culture. The installation uses a plastic olive tree with three symbolic leaves. Each leaf has a hidden touch sensor made of aluminum foil and sponge, which activates when pressed gently on a red mark. When the leaf is pressed, the corresponding olive lights up on the tree, and a short audio (poem or song) starts playing, followed by a visual enlargement of a related image on screen. I was looking for the overall experience to evoke the act of picking an olive and hearing a story passed down through generations. My main goal for this project is to preserve memory and heritage through touch, sound, and visuals.

Interaction Design:
– The user interacts with the leaves of an olive tree.
– Each leaf has a red marking –> made using two layers of aluminum separated by sponge.
– When red marking is pressed:
– A green olive lights up on the tree (via LED).
– One of two audio stories tied to that olive is randomly selected and begins playing.
– A corresponding visual (image) on screen zooms in smoothly.
– After the story ends:
– The screen returns to its main menu.
– The olive light turns off.

I wanted this natural interaction design to draw people in gently, as they literally touch a memory through the leaves of the olive tree.

Link to process photos + Link to extra designs made

Arduino Code:

The Arduino reads input from 3 capacitive touch pins and lights up corresponding LEDs. It also sends a serial message (“1”, “2”, or “3”) to indicate which olive was touched.

#include <Adafruit_NeoPixel.h>

const int touchPin = 6;
const int touchLedPin = 3;

const int touchPin2 = 10;
const int touchLedPin2 = 11;

const int touchPin3 = 7;
const int touchLedPin3 = 12;

const int trigPin = 8;
const int echoPin = 9;

#define STRIP_PIN 5
#define NUMPIXELS 50
Adafruit_NeoPixel strip(NUMPIXELS, STRIP_PIN, NEO_GRB + NEO_KHZ800);

void setup() {
  pinMode(touchPin, INPUT_PULLUP);
  pinMode(touchLedPin, OUTPUT);

  pinMode(touchPin2, INPUT_PULLUP);
  pinMode(touchLedPin2, OUTPUT);

  pinMode(touchPin3, INPUT_PULLUP);
  pinMode(touchLedPin3, OUTPUT);

  pinMode(trigPin, OUTPUT);
  pinMode(echoPin, INPUT);

  strip.begin();
  strip.show();
  Serial.begin(9600);
}

void loop() {
  int touchState = digitalRead(touchPin);
  digitalWrite(touchLedPin, touchState == LOW ? HIGH : LOW);
  if (touchState == LOW) {
    Serial.println("1");
    delay(300);
  }

  int touchState2 = digitalRead(touchPin2);
  digitalWrite(touchLedPin2, touchState2 == LOW ? HIGH : LOW);
  if (touchState2 == LOW) {
    Serial.println("2");
    delay(300);
  }

  int touchState3 = digitalRead(touchPin3);
  digitalWrite(touchLedPin3, touchState3 == LOW ? HIGH : LOW);
  if (touchState3 == LOW) {
    Serial.println("3");
    delay(300);
  }

  digitalWrite(trigPin, LOW);
  delayMicroseconds(2);
  digitalWrite(trigPin, HIGH);
  delayMicroseconds(10);
  digitalWrite(trigPin, LOW);

  long duration = pulseIn(echoPin, HIGH, 30000);
  int distance = (duration == 0) ? 999 : duration * 0.034 / 2;

  if (distance < 100) {
    for (int i = 0; i < NUMPIXELS; i++) {
      strip.setPixelColor(i, strip.Color(60, 90, 30));
    }
  } else {
    for (int i = 0; i < NUMPIXELS; i++) {
      strip.setPixelColor(i, 0);
    }
  }

  strip.show();
  delay(50);
}

p5.js Code: 

– Loads 6 images and 5 audio files
– Maps each box to one audio
– Randomizes which box is triggered per olive
– Communicates with Arduino using Web Serial
– Displays interactive visuals with zoom animation and audio syncing

Arduino – p5.js Communication:
– Serial connection via Web Serial API
– Arduino sends “1”, “2”, or “3” when a sensor is triggered
– p5.js receives the value, randomly selects one of two mapped boxes, and plays audio/image
– Serial connection is opened through a “Connect to Arduino” button

Link to video demonstration 

Highlights / Proud Moments:

I was very proud of my use of leaves as soft sensors because I believe it adds a layer of tactile beauty while blending the installation with poetic significance. i also liked my use of the red markings on each leaf as they serve as subtle visual guide, naturally blending into the overall aesthetic of the tree while inviting interaction. When a user presses on a marked leaf, the seamless synchronisation between light and audio brings the experience to life, creating an emotionally resonant response. I knew that my goal was met when a lot of users commented and told me that they really appreciated the meaningful aspect of my project. Overall, I think that my project connects technology and tradition and turns a symbolic object into a responsive storytelling medium.

Challenges/Future Improvements: 

One of the main challenges I faced was my initial idea to make the olives themselves the touchpoint that triggered the audio.  I struggled to add a reliable sensor inside the small and spherical shape of the olives. However, to overcome this, I problem-solved by shifting the interaction to the leaves, which offered more surface area and flexibility. Also, I learned the importance of saving your code very often and debugging. Looking ahead, future improvements could include integrating haptic feedback into the olives, adding more stories with replay options, and even including narration subtitles for accessibility. Overall, I am extremely proud of how my project turned out and the responses it got from many users.

Concept 

My concept is a semi-thriller game that features two key sensors — the joystick, and the pulse sensor. It was inspired by my siblinghood. Growing up, I would often eat my sister’s snacks, of course, without her permission. This would give me a thrill that was akin to what someone would feel playing hide-and-seek with a murderer (I’m exaggerating here; I actually really love my sister haha). I settle on making it “slime-themed” since I wanted it to be more fun and gender-neutral to be inclusive to all players (possibly people that had older brothers, cousins, etc.). I mean certainly a slime character is non-specific enough I think.

How it works

Well, when you first start the game, you will be greeted with an eery background music and screen image with two options: “how to play” and “start game”. If you press on “how to play” with the joystick, you’ll be shown a very comprehensive manual on how to play the game. Then, if you press start, it’ll take you to the game dialog, and you can click through the dialog through the joystick clicks. After that, the game begins! When it starts, on the top right of the screen there is a icon of slime chips with a number on top, these are the number of chips you need to eat without getting caught and losing all your hearts. The number of hearts you have is on the top left. When the sister turns around and you’re eating your chips (i.e. your hands are touching; you can move the right hand with the chips left and right with the joystick), she gets mad and you lose a heart. Here are the game p5.js visuals:

(startscreen) (how to play) (dialog screen) (game started and sister turned around) (game started and sister caught you!!)(you won!!)

Implementation

1. Interaction Design:

I wanted it to have a flex sensor initially as the pseudo “joystick” but I realized that that wouldn’t really be intuitive or even enjoyable for the average user likely due to its foreign nature in games like these. So I decided to borrow a joystick from the IM lab, alongside a pulse sensor. I wanted to make the game take real-time input from the user and imitate the scenario of my heart racing as a kid when stealing my sister’s food. So, every 1.5 to 2.5 seconds, the game checks if the user’s heartbeat ‘jumps’ by more than 70 (imitating nervousness). If so, or with a 25% random chance, the sister turns around. I did this to simulate the tension of being caught, as the player’s nervousness (heartbeat jump) makes the slime sibling more likely to notice you with a bit of randomness. I added chip-eating sound effects among some other sounds to reflect what is happening on the screen.

2. Arduino Code:

My Arduino code controls three pairs of colored LEDs (red, yellow, green) based on commands received from a computer (via Serial). It also reads a joystick’s X-axis, a pulse sensor, and a joystick button (I put print statements for their values for debugging and monitoring). Only one color is lit at a time, and the code ignores repeated color commands, to make sure a state is reflected clearly for the player. I also added some helper functions to keep all LEDs off at the beginning and another one so that the same color LED is on at both sides of the box. Here is the code:

const int joyXPin   = A0;    // joystick a-axis
const int pulsePin  = A1;    // pulse sensor
const int selPin    = 8;     // joystick “select” button (digital)
const int baudRate  = 9600;

// LED pins
const int RED_L  = 7,  RED_R  = 5;
const int YEL_L  = 10, YEL_R  = 4;
const int GRN_L  = 12, GRN_R  = 2;

const int threshold = 50;    // dead-zone around center (512)

// remembering last LED state
String currentCmd = "";

void setup() {
  Serial.begin(baudRate);
  pinMode(selPin, INPUT_PULLUP);
  
  // LED pins
  pinMode(RED_L, OUTPUT);
  pinMode(RED_R, OUTPUT);
  pinMode(YEL_L, OUTPUT);
  pinMode(YEL_R, OUTPUT);
  pinMode(GRN_L, OUTPUT);
  pinMode(GRN_R, OUTPUT);

  // start with all off
  clearAllLeds();
}

void loop() {
  // checking for new command from p5.js
  if (Serial.available()) {
    String cmd = Serial.readStringUntil('\n');
    cmd.trim();

    // only accept valid colors
    if (cmd == "RED" || cmd == "YELLOW" || cmd == "GREEN") {
      // if different than last, update
      if (cmd != currentCmd) {
        currentCmd = cmd;
        applyLeds(currentCmd);
      }
    }
    // echo back for debugging
    Serial.print("Received: ");
    Serial.println(cmd);
  }

  // existing joystick/pulse prints
  int xRaw    = analogRead(joyXPin);
  int pulseRaw = analogRead(pulsePin);
  bool selPressed = (digitalRead(selPin) == LOW);

  int dirX;
  if (xRaw < 512 - threshold)      dirX = 0;
  else if (xRaw > 512 + threshold) dirX = 1;
  else                              dirX = -1;

  Serial.print("DIRX:");   Serial.print(dirX);
  Serial.print("  PULSE:");Serial.print(pulseRaw);
  Serial.print("  SEL:");  Serial.println(selPressed ? 1 : 0);
}

// helper to turn all LEDs off
void clearAllLeds() {
  digitalWrite(RED_L, LOW); digitalWrite(RED_R, LOW);
  digitalWrite(YEL_L, LOW); digitalWrite(YEL_R, LOW);
  digitalWrite(GRN_L, LOW); digitalWrite(GRN_R, LOW);
}

// helper to set exactly one color on both sides
void applyLeds(const String &cmd) {
  clearAllLeds();
  if (cmd == "RED") {
    digitalWrite(RED_L, HIGH);
    digitalWrite(RED_R, HIGH);
  } 
  else if (cmd == "YELLOW") {
    digitalWrite(YEL_L, HIGH);
    digitalWrite(YEL_R, HIGH);
  } 
  else if (cmd == "GREEN") {
    digitalWrite(GRN_L, HIGH);
    digitalWrite(GRN_R, HIGH);
  }
}

3. Schematic of Project:

4. p5.js Code:

The game features multiple states (start menu, gameplay, how-to-play screen) managed through a state machine architecture. When playing, users control a hand to grab chips while avoiding detection from the sister character who randomly turns around. The sister’s turning behavior is influenced by both randomness and the player’s actual heartbeat measured through a pulse sensor.

Key features in the p5.js include:

• Dialog system for narrative progression
• Character animation with multiple emotional states
• Collision detection between hands
• Health and scoring systems (hearts and chip counter)
• Two endings (happy and bad) based on gameplay outcomeThis is the code:

  let gameState = 'start'; // 'start', 'game', or 'howToPlay'
  let startImg;
  let bgImg;
  let howToPlayImg; 
  let bgMusic;
  let angry;
  
  // serial communication variables
  let rightX = 0;
  let pulseValue = 0;  
  let connectButton; // button to initiate serial connection
  
  // hoystick control variables for menu navigation
  let currentSelection = 0; // 0 = start game, 1 = how to play
  let selPressed = false;
  let prevSelPressed = false; // to detect rising edge
  let backButtonSelected = false; // tracking if back button is selected in how to play screen
  
  // hand objects
  let leftHand = {
    img: null,
    x: 0,
    y: 0,
    width: 150,
    height: 150,
    visible: false
  };
  
  let rightHand = {
    img: null,
    x: 0,
    y: 0,
    width: 150,
    height: 150,
    visible: false,
    currentImg: 1, // tracking current hand image (1, 2, or 3)
    animationStarted: false, // tracking if animation has started
    animationTimer: 0, // timer for animation
    lastLedState: "",
    sisterTurned: false, // tracking if sister is turned around
    sisterTurnStart: 0,      // ← when she last flipped to “turned”
    maxTurnDuration: 1700,   // ← max ms she can remain turned (1.7 s)
    lastTurnCheck: 0, // timer for checking sister's state
    heartDeducted: false, // flag to track if heart was deducted in current overlap
    lastSisterState: false, // tracking previous sister state to detect changes
    sisterAngry: false, // flag to track if sister is angry
    lastPulseValue: 0 // tracking last pulse value for heartbeat detection
  };
  
  let chipCounter = {
    img: null,
    x: 0,
    y: 0,
    width: 80,
    height: 80,
    count: 10
  };
  
  let hearts = {
    img: null,
    count: 5, 
    width: 30,
    height: 30,
    spacing: 10,
    x: 20, 
    y: 20 
  };
  
  let gameOver = false;
  let gameEndingType = ''; // 'good' or 'bad' to track which ending to show
  
  let dialogSounds = {};
  
  let startButton = {
    x: 0,
    y: 0,
    width: 150, 
    height: 30,
    text: 'start game',
    isHovered: false
  };
  
  let howToPlayButton = {
    x: 0,
    y: 0,
    width: 180, 
    height: 30,
    text: 'how to play',
    isHovered: false
  };
  
  let backButton = { 
    x: 0,
    y: 0,
    width: 100,
    height: 40,
    text: 'Back',
    isHovered: false
  };
  
  let instrumentSerifFont;
  let currentDialog = null; 
  
  let characters = {};
  
  function preload() {
    startImg = loadImage('Images/start.png');
    bgImg = loadImage('Images/bg.png');
    howToPlayImg = loadImage('Images/howtoplay.png'); 
    instrumentSerifFont = loadFont('InstrumentSerif-Regular.ttf');
    bgMusic = loadSound('Audios/bg.mp3');
    leftHand.img = loadImage('Images/left_hand.png');
    rightHand.img = loadImage('Images/right_hand1.png');
    rightHand.img2 = loadImage('Images/right_hand2.png');
    rightHand.img3 = loadImage('Images/right_hand3.png');
    chipCounter.img = loadImage('Images/slimechips.png');
    hearts.img = loadImage('Images/heart.png');
    happyEndingImg = loadImage('Images/happy_ending.png');
    badEndingImg = loadImage('Images/bad_ending.png');
  
    dialogSounds['chipbag.mp3'] = loadSound('Audios/chipbag.mp3');
    dialogSounds['chip_crunch.mp3'] = loadSound('Audios/chip_crunch.mp3');
    dialogSounds['angry.mp3'] = loadSound('Audios/angry.mp3');
  
    characters['sister_normal'] = new Character('sister_normal', 'Sister', 'Images/sister_normal.png');
    characters['sister_speaking'] = new Character('sister_speaking', 'Sister', 'Images/sister_speaking.png');
    characters['sister_turned'] = new Character('sister_turned', 'Sister', 'Images/sister_turned.png');
    characters['sister_upset'] = new Character('sister_upset', 'Sister', 'Images/sister_upset.png');
    characters['sister_angry'] = new Character('sister_angry', 'Sister', 'Images/sister_angry.png');
    characters['slime'] = new Character('slime', 'me', 'Images/slime.png');
    characters['slime_speaking'] = new Character('slime_speaking', 'me', 'Images/slime_speaking.png');
    for (let key in characters) {
      characters[key].preload();
    }
  }
  
  async function setUpSerialWithErrorHandling() { // handling errors with serial
    try {
      await setUpSerial();
    } catch (error) {
      console.log("Serial connection failed: " + error.message);
    }
  }
  
  function setup() {
    createCanvas(627, 447);
    noCursor(); 
  
    // playing background music
    bgMusic.setVolume(0.5); 
    bgMusic.loop(); 
  
    textFont(instrumentSerifFont);
  
    connectButton = createButton('connect serial');
    connectButton.position(25, 25);
    connectButton.style('font-size', '18px');
    connectButton.style('font-weight', 'bold');
    connectButton.style('color', '#000');
    connectButton.style('background-color', '#fff');
    connectButton.style('border', '2px solid #000');
    connectButton.style('border-radius', '8px');
    connectButton.style('padding', '8px 16px');
    connectButton.style('cursor', 'pointer');
  
      connectButton.mouseOver(() => {
        connectButton.style('background-color', '#000');
        connectButton.style('color', '#fff');
      });
      connectButton.mouseOut(() => {
        connectButton.style('background-color', '#fff');
        connectButton.style('color', '#000');
      });
  
    connectButton.mousePressed(setUpSerialWithErrorHandling); // using setUpSerial from p5.web-serial.js
  
  
    positionStartScreenButtons();
  
    positionBackButton();
  
    rightHand.x = width - 10;
    rightHand.y = height/2 - rightHand.height/2;
  
    chipCounter.x = width - chipCounter.width - 20;
    chipCounter.y = 20;
  
    const sampleDialog = [
      { speaker: 'sister_speaking', line: "Hey! Have you seen my chips?" },
      { speaker: 'slime', line: "Nope, I haven't seen them." , sound: "chipbag.mp3"},
      { speaker: 'sister_upset', line: "Hmmm... are you sure?" },
      { speaker: 'slime_speaking', line: "Yep...! Haven't seen them! I'll let you know if I see them!" },
      { speaker: 'sister_speaking', line: "Grrr! If I catch you with them..I'm going to kill you!" }
    ];
    currentDialog = new Dialog(sampleDialog); // passing the new structure to the Dialog constructor
  }
  
  function readSerial(data) {
    let val = trim(data);
    if (!val) return;
  
    let parts = val.split(" ");
  
    // parsing DIRX for both game movement and menu navigation
    let dirPart = parts.find(p => p.startsWith("DIRX:"));
    if (dirPart) {
      let dir = int(dirPart.split(":")[1]);
  
      // setting rightX for in-game hand movement
      if (dir === 0) {
        rightX = width * 0.25;  // move left
        
        // for menu navigation, select the left button (start game)
        if (gameState === 'start') {
          currentSelection = 0; // selecting start game button
        } 
        // for how to play screen, select the back button when moving left
        else if (gameState === 'howToPlay') {
          backButtonSelected = true;
        }
      } else if (dir === 1) {
        rightX = width * 0.75;  // move right
        
        // for menu navigation, select the right button (how to play)
        if (gameState === 'start') {
          currentSelection = 1; // selecting how to play button
        }
        // for how to play screen, deselect the back button when moving right
        else if (gameState === 'howToPlay') {
          backButtonSelected = false;
        }
      } else if (dir === -1) {
        rightX = width / 2;     // center (neutral)
      }
  
      console.log("Joystick Direction:", dir);
    }
  
    // parsing SEL for button selection
    let selPart = parts.find(p => p.startsWith("SEL:"));
    if (selPart) {
      let sel = int(selPart.split(":")[1]);
      prevSelPressed = selPressed;
      selPressed = (sel === 1);
      
      // if SEL button was just pressed (rising edge detection)
      if (selPressed && !prevSelPressed) {
        // handling button selection based on current screen
        if (gameState === 'start') {
          if (currentSelection === 0) {
            startGame();
          } else if (currentSelection === 1) {
            showHowToPlay();
          }
        } else if (gameState === 'howToPlay') {
          // only going back if the back button is selected
          if (backButtonSelected) {
            goBackToStart();
          }
        } else if (gameState === 'game' && currentDialog && !currentDialog.isComplete) {
          currentDialog.advanceDialog();
        }
      }
    }
  
    // parsing PULSE
    let pulsePart = parts.find(p => p.startsWith("PULSE:"));
    if (pulsePart) {
      pulseValue = int(pulsePart.split(":")[1]);
      console.log("Pulse Reading:", pulseValue);
    }
  }
  
  
  // helper that mirrors mouseClicked() logic but uses joystick coords
  function joystickClick(x,y){
    if(gameState==='start'){
      // is it on the “start” button?
      if(x > startButton.x && x < startButton.x+startButton.width &&
         y > startButton.y && y < startButton.y+startButton.height){
        startGame();
      }
      // how-to-play button?
      else if(x > howToPlayButton.x && x < howToPlayButton.x+howToPlayButton.width &&
              y > howToPlayButton.y && y < howToPlayButton.y+howToPlayButton.height){
        showHowToPlay();
      }
    }
  }
  
  
  function draw() {
    background(255); 
    
    switch (gameState) {
      case 'start':
        drawStartScreen();
        updateButtonHover(startButton);
        updateButtonHover(howToPlayButton);
        break;
      case 'game':
        drawGameScreen();
        updateButtonHover(backButton);
        break;
      case 'howToPlay':
        drawHowToPlayScreen();
        updateButtonHover(backButton);
        break;
    }
  
    if (!gameOver) {
      noCursor();
    }
  }
  
  function drawStartScreen() {
    image(startImg, 0, 0, width, height);
    
    // updating button hover state based on joystick selection instead of mouse
    if (gameState === 'start') {
      // hilighting the currently selected button based on joystick input
      startButton.isHovered = (currentSelection === 0);
      howToPlayButton.isHovered = (currentSelection === 1);
    }
    
    drawButton(startButton);
    drawButton(howToPlayButton);
    
    // hiding back button on start screen
    backButton.x = -backButton.width; // positioning off-screen
  }
  
  function drawGameScreen() {
    image(bgImg, 0, 0, width, height);
    
    // drawing dialog if it exists and isn’t complete
    if (currentDialog && !currentDialog.isComplete) {
      currentDialog.display();
      return;
    }
    
    // if dialog is complete but gameover, show ending
    if (gameOver) {
      drawGameOverScreen();
      return;
    }
  
    // --- sister character ---
    let sisterWidth  = 180;
    let sisterHeight = 200;
    let sisterX = width/2 - sisterWidth/2;
    let sisterY = height/2 - sisterHeight/2 - 30;
    
    let sisterImg = characters['sister_turned'].img;
    if (rightHand.sisterAngry)        sisterImg = characters['sister_angry'].img;
    else if (rightHand.sisterTurned)  sisterImg = characters['sister_upset'].img;
    
    // possibly flip her based on time+random+heartbeat
    if (!rightHand.lastTurnCheck 
        || millis() - rightHand.lastTurnCheck > random(1500, 2500)) {
          
      rightHand.lastTurnCheck = millis();
      let heartbeatTrigger = (rightHand.lastPulseValue 
        && Math.abs(pulseValue - rightHand.lastPulseValue) > 70);
      rightHand.lastPulseValue = pulseValue;
      
      if (random() < 0.25 || heartbeatTrigger) {
        rightHand.sisterTurned = !rightHand.sisterTurned;
        if (rightHand.sisterTurned) {
          rightHand.sisterTurnStart = millis();
        }
        rightHand.sisterAngry   = false;
        rightHand.heartDeducted = false;
      }
    }
    
    image(sisterImg, sisterX, sisterY, sisterWidth, sisterHeight);
  
    // enforcing max “turned” time
    if (rightHand.sisterTurned 
        && millis() - rightHand.sisterTurnStart > rightHand.maxTurnDuration) {
      rightHand.sisterTurned   = false;
      rightHand.sisterAngry    = false;
      rightHand.heartDeducted  = false;
    }
    
    // --- Collision ---
    leftHand.x = width/2 - leftHand.width - 50;
    leftHand.y = height/2 + leftHand.height/2;
    
    // compute & smooth right hand
    let targetX = constrain(rightX, width*0.1, width*0.9);
    rightHand.x += (targetX - rightHand.x)*0.2;
    rightHand.y  = height/2 + rightHand.height/2;
    
    let handsOverlap = checkHandCollision(leftHand, rightHand);
    let ledState = "";
    if (handsOverlap && rightHand.sisterTurned && !rightHand.heartDeducted) {
      // CAUGHT → angry → RED
      hearts.count--;
      rightHand.heartDeducted = true;
      rightHand.sisterAngry   = true;
      ledState = "RED";
      dialogSounds['angry.mp3'].play();
  
      if (hearts.count <= 0) {
        gameOver = true;
        gameEndingType = 'bad';
      }
    }
  
    else if (handsOverlap && rightHand.sisterTurned && rightHand.heartDeducted) {
      ledState = "RED";
    }
    else if (!rightHand.sisterTurned) { // sister is not turned
      // SAFE while she’s turned → GREEN
      ledState = "GREEN";
    }
    else if (rightHand.sisterTurned && !handsOverlap) { // sister is turned but hands are not overlapping
      // NORMAL (not turned) → YELLOW
      ledState = "YELLOW";
    }
  
    if (ledState && ledState !== rightHand.lastLedState) {
      writeSerial(ledState + "\n");
      rightHand.lastLedState = ledState;
    }
    
    // --- Eating Animation & Chips ---
    if (handsOverlap && !rightHand.animationStarted) {
      rightHand.animationStarted = true;
      rightHand.animationTimer   = millis();
      if (rightHand.currentImg === 3 && chipCounter.count > 0) {
        chipCounter.count--;
        if (chipCounter.count <= 0) {
          gameOver = true;
          gameEndingType = 'good';
        }
      }
      rightHand.currentImg = 1;
      dialogSounds['chip_crunch.mp3'].play();
    }
    else if (handsOverlap && rightHand.animationStarted) {
      let t = millis() - rightHand.animationTimer;
      if (rightHand.currentImg === 1 && t > 300) rightHand.currentImg = 2;
      if (rightHand.currentImg === 2 && t > 500) rightHand.currentImg = 3;
      if (rightHand.currentImg === 3 && t > 800) {
        rightHand.animationTimer = millis();
        rightHand.currentImg = 1;
        if (chipCounter.count > 0) {
          chipCounter.count--;
          if (chipCounter.count <= 0) {
            gameOver = true;
            gameEndingType = 'good';
          }
        }
        dialogSounds['chip_crunch.mp3'].play();
      }
    }
    else if (!handsOverlap && rightHand.animationStarted) {
      rightHand.animationStarted = false;
      dialogSounds['chip_crunch.mp3'].stop();
    }
    
    // drawing hands
    let handImg = rightHand.currentImg===1 ? rightHand.img
                : rightHand.currentImg===2 ? rightHand.img2
                : rightHand.img3;
    image(handImg, rightHand.x, rightHand.y, rightHand.width, rightHand.height);
    image(leftHand.img, leftHand.x, leftHand.y, leftHand.width, leftHand.height);
  
    image(chipCounter.img, chipCounter.x, chipCounter.y, chipCounter.width, chipCounter.height);
    push();
      fill(255); stroke(0); strokeWeight(2);
      textSize(24); textAlign(CENTER, CENTER);
      text(chipCounter.count, chipCounter.x+chipCounter.width/2, chipCounter.y+chipCounter.height/2);
    pop();
    for (let i=0; i<hearts.count; i++) {
      image(hearts.img, hearts.x+(hearts.width+hearts.spacing)*i, hearts.y, hearts.width, hearts.height);
    }
  }
  
  // function to check collision between hand objects
  function checkHandCollision(hand1, hand2) {
    // rectangular collision detection
    return (
      hand1.x + 10 < hand2.x + hand2.width &&
      hand1.x + hand1.width > hand2.x &&
      hand1.y < hand2.y + hand2.height &&
      hand1.y + hand1.height > hand2.y
    );
  }
  
  function drawHowToPlayScreen() {
      background(50); 
      let imgWidth = width * 0.9; // Use 90% of screen width
      let imgHeight = height * 0.7; // Use 70% of screen height
      image(howToPlayImg, width/2 - imgWidth/2, height/2 - imgHeight/2 + 30, imgWidth, imgHeight);
      
      fill(255);
  
      textSize(35);
      textAlign(CENTER, CENTER);
      text("How to Play", width/2, height/2 - imgHeight/2 - 15);
  
      backButton.isHovered = backButtonSelected;
      
      drawButton(backButton);
      
      startButton.x = -startButton.width; // Position off-screen
      howToPlayButton.x = -howToPlayButton.width; // Position off-screen
  }
  
  
  function startGame() {
    gameState = 'game';
    console.log("Game Started!");
  
    if (bgMusic.isPlaying()) {
      bgMusic.stop();
    }
  
    if (currentDialog) {
        currentDialog.currentLineIndex = 0;
        currentDialog.isComplete = false;
    }
  
    positionBackButton();
  }
  
  function showHowToPlay() {
    gameState = 'howToPlay';
    console.log("Showing How to Play");
     positionBackButton();
  }
  
  function goBackToStart() {
      gameState = 'start';
      console.log("Going back to Start Screen");
      positionStartScreenButtons();
  }
  
  
  // helper function to position buttons on the start screen
  function positionStartScreenButtons() {
      textSize(20); 
      startButton.width = textWidth(startButton.text) + 40; 
      howToPlayButton.width = textWidth(howToPlayButton.text) + 40; 
      startButton.height = 40; 
      howToPlayButton.height = 40; 
  
  
      let buttonY = height * 0.65; // Moved buttons higher (was 0.75)
  
      let buttonSpacing = 20;
      let totalButtonWidth = startButton.width + howToPlayButton.width + buttonSpacing;
      let leftShift = 15; 
      startButton.x = width / 2 - totalButtonWidth / 2 - leftShift;
      howToPlayButton.x = startButton.x + startButton.width + buttonSpacing;
  
      startButton.y = buttonY;
      howToPlayButton.y = buttonY;
  }
  
  function positionBackButton() {
      textSize(20); 
      backButton.width = textWidth(backButton.text) + 40;
      backButton.height = 40; 
      backButton.x = 20; 
      backButton.y = 20; 
  }
  
  
  // function to draw a button
  function drawButton(button) {
       if (button.x + button.width > 0 && button.x < width) {
           push(); // Save current drawing settings
  
          if (button.isHovered) {
              fill(255, 0, 0); 
          } else {
              fill(0); 
          }
  
          rect(button.x, button.y, button.width, button.height, 10); // 10 is corner radius
  
          fill(255);
          textSize(22);
          textAlign(CENTER, CENTER);
  
          text(button.text, button.x + button.width / 2, button.y + button.height / 2 - 5);
  
          pop(); // restoring drawing settings
       }
  }
  
  // helper function to update button hover state
  function updateButtonHover(button) {
      // inly update hover if the button is likely visible on screen
      if (button.x + button.width > 0 && button.x < width) {
          if (mouseX > button.x && mouseX < button.x + button.width &&
              mouseY > button.y && mouseY < button.y + button.height) {
              button.isHovered = true;
              cursor(HAND); // Change cursor on hover
          } else {
              button.isHovered = false;
              // only set cursor back to ARROW if NO button is hovered
               let anyButtonHovered = false;
               if (gameState === 'start') {
                   if (startButton.isHovered || howToPlayButton.isHovered) anyButtonHovered = true;
               } else if (gameState === 'howToPlay' || gameState === 'game') {
                   if (backButton.isHovered) anyButtonHovered = true;
               }
               if (!anyButtonHovered) {
                   cursor(ARROW);
               }
          }
      }
  }
  
  // helper function to update button hover state with joystick
  function updateButtonHoverWithJoystick(button, joyX, joyY) {
    if (button.x + button.width > 0 && button.x < width) {
      if (joyX > button.x && joyX < button.x + button.width &&
          joyY > button.y && joyY < button.y + button.height) {
        button.isHovered = true;
      } else {
        button.isHovered = false;
      }
    }
  }
  
  function mouseClicked() {
    switch (gameState) {
      case 'start':
        // Check for button clicks on the start screen
        if (mouseX > startButton.x && mouseX < startButton.x + startButton.width &&
            mouseY > startButton.y && mouseY < startButton.y + startButton.height) {
          startGame();
        } else if (mouseX > howToPlayButton.x && mouseX < howToPlayButton.x + howToPlayButton.width &&
                   mouseY > howToPlayButton.y && mouseY < howToPlayButton.y + howToPlayButton.height) {
          showHowToPlay();
        }
        break;
      case 'game':
        // handling dialog clicks in the game screen
        if (currentDialog && !currentDialog.isComplete) {
          if (mouseX > currentDialog.boxX && mouseX < currentDialog.boxX + currentDialog.boxWidth &&
              mouseY > currentDialog.boxY && mouseY < currentDialog.boxY + currentDialog.boxHeight) {
            currentDialog.advanceDialog();
          } else if (mouseX > backButton.x && mouseX < backButton.x + backButton.width && 
                     mouseY > backButton.y && mouseY < backButton.y + backButton.height) {
              goBackToStart(); // Go back if back button is clicked
          }
        } else { 
             if (mouseX > backButton.x && mouseX < backButton.x + backButton.width &&
                     mouseY > backButton.y && mouseY < backButton.y + backButton.height) {
              goBackToStart(); // Go back if back button is clicked
             }
        }
        break;
      case 'howToPlay':
          if (mouseX > backButton.x && mouseX < backButton.x + backButton.width &&
              mouseY > backButton.y && mouseY < backButton.y + backButton.height) {
              goBackToStart(); // Go back to start screen
          }
          break;
    }
  }
  
  
  class Character {
    constructor(key, displayName, imagePath) {
      this.key = key;
      this.displayName = displayName;
      this.imagePath = imagePath;
      this.img = null;
    }
    preload() {
      this.img = loadImage(this.imagePath);
    }
  }
  
  class Dialog {
    constructor(dialogLines) {
      // dialogLines: array of strings like 'sister_normal: "Hey!"' or just dialog text
      this.lines = dialogLines.map(lineObj => {
        if (typeof lineObj === 'string') {
          let match = lineObj.match(/^(\w+):\s*"([\s\S]*)"$/);
          if (match) {
            return { speaker: match[1], line: match[2] };
          } else {
            return { speaker: 'sister_normal', line: lineObj };
          }
        } else if (typeof lineObj === 'object') {
          return {
            speaker: lineObj.speaker || 'sister_normal',
            line: lineObj.line || '',
            sound: lineObj.sound || null 
          };
        }
      });
      this.currentLineIndex = 0;
      this.isComplete = false;
      this.boxWidth = width * 0.8;
      this.boxHeight = 100;
      this.boxX = (width - this.boxWidth) / 2;
      this.boxY = height - this.boxHeight - 20;
      this.textSize = 18;
      this.textMargin = 15;
      this.lineHeight = this.textSize * 1.2;
    }
  
    display() {
      if (this.isComplete) {
        return;
      }
      fill(0, 0, 0, 200);
      rect(this.boxX, this.boxY, this.boxWidth, this.boxHeight, 10);
  
      let currentLineObj = this.lines[this.currentLineIndex];
      let charKey = currentLineObj.speaker || 'sister_normal';
      
      let displayedCharKey = charKey; // Either 'me' or one of the sister variants
      
      let charObj = characters[displayedCharKey] || characters['sister_normal'];
      
      let charImgSize = (displayedCharKey === 'me') ? 150 : 220; 
      
      if (charObj && charObj.img) {
        let centerX = width / 2 - charImgSize / 2;
        
        let centerY = (displayedCharKey === 'me') 
          ? height / 2 - charImgSize / 2 - 20  
          : height / 2 - charImgSize / 2 - 45; 
        
        image(charObj.img, centerX, centerY, charImgSize, charImgSize);
        
        fill(255);
        textSize(18);
        textAlign(CENTER, TOP);
        text(charObj.displayName, width / 2, centerY + charImgSize + 5);
      }
  
      fill(255);
      textSize(this.textSize);
      textAlign(LEFT, TOP);
      let textY = this.boxY + this.textMargin;
      text(currentLineObj.line, this.boxX + this.textMargin, textY, this.boxWidth - this.textMargin * 2);
  
      if (this.currentLineIndex < this.lines.length - 1) {
        fill(255, 150);
        textSize(14);
        textAlign(RIGHT, BOTTOM);
        text("Click to continue...", this.boxX + this.boxWidth - this.textMargin, this.boxY + this.boxHeight - 5);
      }
    }
  
    advanceDialog() {
      if (this.isComplete) {
        return;
      }
      
      this.currentLineIndex++;
      
      if (this.currentLineIndex >= this.lines.length) {
        this.isComplete = true;
        console.log("Dialog complete!");
        return;
      }
      
      let currentLineObj = this.lines[this.currentLineIndex];
      if (currentLineObj.sound && dialogSounds[currentLineObj.sound]) {
        dialogSounds[currentLineObj.sound].play();
      }
    }
  }
  
  function drawGameOverScreen() {
    push();
    fill(0, 0, 0, 180); // dark overlay with 70% opacity
    rect(0, 0, width, height);
    
    let endingImg;
    if (gameEndingType === 'good') {
      endingImg = happyEndingImg;
    } else {
      endingImg = badEndingImg;
    }
    
    let imgWidth = 300;
    let imgHeight = 200;
    image(endingImg, width/2 - imgWidth/2, height/2 - 150, imgWidth, imgHeight);
    
    fill(255);
    textSize(40);
    textAlign(CENTER, CENTER);
    if (gameEndingType === 'good') {
      text("YOU WON!", width/2, height/2 - 30);
    }
    else {
      text("YOU LOST!", width/2, height/2 - 30);
    }
    textSize(24);
    text("Press R to restart", width/2, height/2 + 80);
    pop();
  }
  
  function keyPressed() {
    // Check if 'R' key is pressed to restart the game when game over
    if (key === 'r' || key === 'R') {
      if (gameState === 'game' && gameOver) {
        resetGame();
      }
    }
  }
  
  function resetGame() {
    hearts.count = 5;
    chipCounter.count = 10;
    gameOver = false;
    
    handsOverlap = false;
    rightHand.sisterTurned = false;
    rightHand.animationStarted = false;
    rightHand.currentImg = 1;
    rightHand.heartDeducted = false;
  }

   

5. Communication between Arduino and p5.js:

The Arduino and the p5.js sketch exchange data over a 9600 baud serial link, as we did in the sample homework 2 weeks back. On each loop, the Arduino reads the joystick X‐axis (A0), the pulse sensor (A1) and the select button (pin 8), then emits a line of the form “DIRX:<–1|0|1> PULSE:<0–1023> SEL:<0|1>
” (these were the specific format that I chose as it helped me debug as well). The p5.js sketch listens for these lines, parses out the numeric values for direction, heart rate, and button state, and uses them to drive menu selection, character motion, and game logic. When the sketch determines that the physical LED indicators should change color (based on the game state), it sends right back a single‐word command: “RED\n”, “YELLOW\n” or “GREEN\n” via writeSerial(), and the Arduino responds by calling applyLeds() to set the appropriate digital outputs. I’m glad that I was able to implement a bidirectional serial communication where not only the Arduino sends information to the p5 sketch but also vice versa.

Reflection

I was really proud of how seamlessly the physical box integrated into the game environment; it didn’t feel tacked on, but like a natural extension of the game world I created .  I’m also proud that I soldered the LEDs to the wires because I was really afraid of soldering in the beginning. However, after a couple of tries on a sample, I was able to solder all 12 legs . In the future, I’d look to add a custom cursor that follows the joystick movements and actually wire up the Y-axis input so users can move that cursor freely around the screen. During playtesting, at least two people instinctively pushed the stick up and down—and, of course, nothing happened (since I hadn’t implemented Y at all)—which led to a couple of awkward pauses. Enabling full two-axis control would make the interaction more intuitive and keep the game flow smooth.

Images from Showcase:

THANK YOU PROFESSOR & EVERYONE FOR AN AMAZING CLASS FOR MY LAST SEMESTER!

User Testing

So, I tried letting people play the game. I realized that most of them didn’t really look at the instructions. At first, I let them figure it out on their own, and after the first round, they usually managed to understand how to play, which was nice. However, starting with the third user, I started making more of an effort to explain things minimally, answer questions when they came up, or suggest they check the “How to Play” manual.

People really liked the design I was going for, so I kept that aesthetic and focused on making it pop and align with the theme I envisioned. One thing I noticed is that the wires could be better hidden with blue tape (to match the box), so I’ll likely add that along with other cosmetic improvements. I also want to tweak the win conditions, like changing how hearts are deducted or how chips are counted, because I felt it was a little too easy to win. I’d also like to add an LED that reflects the state of the game for clearer feedback.

One issue I noticed was that players didn’t fully understand the joystick controls, especially since there’s no visible cursor. I had to explain to at least two people that the joystick only moves left and right, not up and down. To help with this, I plan to make the “How to Play” manual more detailed so users can understand the controls better from the start.

Video 

(I know the video is after I added the LEDs..imagine them without because those were actually last minute additions + also for some reason I can’t attach the video as media so I’ll add a google drive link :>)

https://drive.google.com/file/d/1lNbEfx_gUZIfRPWL0JcJXUbqzv9c68sE/view?usp=sharing

Source

The full source code, and accompanying documentation, is available at on Github.

Concept

This project is an interactive media piece that combines audio, visuals, and interactive elements to share the personal stories of students who faced repercussions for their politics. Using both digital and physical interfaces, users can explore these narratives at their own pace and in thier own manner.

My motivation behind choosing this cause was to highlight the similarity and elicit the empathy of the audience, as fellow international students.

How It Works

The implementation consists of three main components working together:

1. Client Interface: A browser-based application built with p5.js that handles the visualization, audio playback, and user interaction.
2. Server Backend: A FastAPI backend that serves content and manages WebSocket connections for real-time control.
3. Physical Controls: An Arduino-based controller that provides tangible ways to interact with the digital content.

When a user accesses the application, they’re presented with a multimedia canvas that displays images related to a student’s story while playing their audio narrative. As the audio plays, transcribed text appears dynamically on screen, visualizing the spoken words. Users can control the experience either through on-screen interactions or using the physical Arduino controller.

Interaction Design

The interaction design prioritizes intuitive exploration and meaningful engagement:

– Starting the Experience: Users simply click anywhere on the screen to begin
– Content Navigation: Pressing buttons or clicking on screen transitions to the next story
– Playback Controls:
– A physical switch pauses/resumes the audio
– Potentiometers adjust volume and playback speed
– On-screen interactions mirror these controls for users without the physical interface

The design deliberately minimizes explicit instructions, encouraging exploration and discovery. As found during user testing, providing just a small amount of context greatly improved the experience while still allowing for personal discovery. Users who spent more time interacting with the piece often discovered deeper layers of meaning as they engaged with multiple stories.

Arduino Implementation

The Arduino controller serves as a physical interface to the digital content, creating a more tangible connection to the stories. The circuit includes:

– A toggle switch for pausing/resuming content
– Two push buttons for transitioning between stories
– Three potentiometers for adjusting various playback parameters

The Arduino code continuously monitors these inputs and sends structured data to the server whenever a change is detected. The Arduino communicates with the server over a serial connection, sending a compact binary structure containing the state of all controls.

Future Improvements

Based on user testing and reflection, several areas could be enhanced in future iterations:

1. Introductory Context: Adding a brief introduction screen would help orient users.
2. More Intuitive Transitions: Some users were initially uncertain about how to navigate between stories.
3. Additional Narratives: Expanding the collection of stories would create a more comprehensive experience.

Concept

I created this project to introduce the viewers to 4 regions in the Northern Caucasus: Adyghea, Dagestan, Ingushetia, Checnya.

As a physical object, i pivoted and decided to build with four distinct doors, each representing one of the regions. Each door is embedded with a color-coded button, inviting the user to press and explore a region.

• Red Button – Ingusheita

• Green Button – Dagestan

• Blue Button – Chechnya

• Yellow Button – Adyghea

This interaction was designed to make the user feel like a traveler stepping into a new cultural experience. Upon pressing a door button, users are “teleported” to a selected region. Each region features:

• Must-See Spots: Highlighted landscapes, canyons and mountains

• Traditions: Rituals, dances, or cultural practices—some of which are interactive

• Taste of Region: Food visuals

Code

P5: Link to no arduino version

Arduino code:

const int buttonPin1 = 4;  // Dagestan button
const int buttonPin2 = 2;  // Adyghea button
const int buttonPin3 = 5;  // Checnya button
const int buttonPin4 = 3;  // Ingushetia button

int buttonState1 = 0;
int buttonState2 = 0;
int buttonState3 = 0;
int buttonState4 = 0;

int lastButtonState1 = HIGH;
int lastButtonState2 = HIGH;
int lastButtonState3 = HIGH;
int lastButtonState4 = HIGH;

void setup() {
  Serial.begin(9600);
  pinMode(buttonPin1, INPUT_PULLUP);
  pinMode(buttonPin2, INPUT_PULLUP);
  pinMode(buttonPin3, INPUT_PULLUP);
  pinMode(buttonPin4, INPUT_PULLUP);
}

void loop() {
  // Read button states
  buttonState1 = digitalRead(buttonPin1);
  buttonState2 = digitalRead(buttonPin2);
  buttonState3 = digitalRead(buttonPin3);
  buttonState4 = digitalRead(buttonPin4);

  // Check for button presses and only send once on press
  if (buttonState1 == LOW && lastButtonState1 == HIGH) {
    Serial.println("DAGESTAN");
  }    
  
  if (buttonState2 == LOW && lastButtonState2 == HIGH) {
    Serial.println("ADYGHEA");
  }
  
  if (buttonState3 == LOW && lastButtonState3 == HIGH) {
    Serial.println("CHECNYA");
  }
  
  if (buttonState4 == LOW && lastButtonState4 == HIGH) {
    Serial.println("INGUSHETIA");
  }
  
  // Save current button states for next comparison
  lastButtonState1 = buttonState1;
  lastButtonState2 = buttonState2;
  lastButtonState3 = buttonState3;
  lastButtonState4 = buttonState4;
  
  delay(50); // Debounce delay
}

Reflections:

This project cost me a lot of all nighters, confusion and frustration. At the same time, it taught me so much both in technical and non-technical aspects and became one of the most rewarding learning experiences I’ve had so far.

Originally, my idea was to create a compass-based interaction, but I struggled to get the p5.js and Arduino integration to work properly. After a whole night night of troubleshooting and brainstorming, I pivoted the concept and fully committed to building a physical house. To bring that idea to life, I had to learn Adobe Illustrator from scratch to design the house structure, prepare it for laser cutting, and assemble it manually. I also learned how to solder and wire hardware components, connecting each button to the Arduino and ensuring smooth communication with the p5.js interface.

Even though my project wasn’t a game or something instantly engaging for everyone, I was quite surprised by the positive reactions at the showcase. Many visitors had never heard of the Caucasus before, yet they showed genuine curiosity to learn about its culture through an interactive format. Several people commented on the effort it must have taken to build the physical installation and to research and create the digital content for each region.

Overall, this project and the entire class had its highs and lows, but I truly loved the creative process—from just having an idea to actual execution of it. Even when it seemed impossible at times, the process pulled me into a state of creative flow and i enjoyed it.

User Testing

For my project, I created an interactive media piece that tells the stories of students who faced consequences due to political protests. The user testing aimed to see how intuitive and engaging the experience was without any instructions.

I found that providing a small amount of context greatly improved the user experience. Users could scan a QR code to access additional multimedia content, which helped them engage more deeply. While some users were initially uncertain, a brief introduction to the controls and background information made them feel more comfortable exploring the project.

I also noticed that users who spent more time interacting with the piece took away more meaningful insights. They often stood and engaged with it for several minutes, discovering layers of meaning for them as they went along.

To improve the work in the future, I may decide to add an introduction screen with some helpful guidance, as well as provide a bit more context to what I was attempting to convey.