Concept 

For my final project, I will create an Emotion Communicator Keyboard; a simple interactive system designed mainly for babies and people with speech difficulties. The device will include several large, colorful buttons, each representing a basic emotion or need (such as happy, sad, hungry, sleepy, or scared). When a user presses a button, the system will communicate that emotion through both physical sound (from Arduino) and visual + digital sound feedback (from P5). This is a preliminary concept and it is still subject to change as I continue developing and testing the project.

Purpose & Audience

Babies and people who have speech difficulties often struggle to communicate how they feel. This device gives them a clear, physical way to express emotions using big, easy-to-press buttons. I got the idea after seeing videos on social media of mothers with children who have speech difficulties. Many of them talked about how challenging it can be to understand what their kids want or how they feel. I noticed that some families use simple communication apps that display basic words or emotions, and even though the tools look very simple, they make a huge difference. I would notice that when the children learned how to express their feelings better through this tool, you could literally see their entire face change. Their eyes would brighten, their expression would relax, and they looked relieved and proud that someone understood them. It made me realize how powerful and meaningful even a very simple communication tool can be. It’s not just about sending a message; it’s about giving the child a way to feel seen, heard, and understood.
The current version is intentionally simple for accessibility, but for adults it would be too basic. In the future, I could expand this into a more advanced assistive communication tool with more emotions, more complex sensors, or personalized options.

Arduino Components 

• 10-12 large push buttons (each representing one emotion)
• LEDs
• Piezo buzzer (to play emotion-specific tones)
• 10kΩ resistors (pull-downs for button circuits)
• USB connection (for serial communication with P5)
• I can build the keyboard with cardboard and cutting holes for the buttons and designing it to look like a keyboard.

Arduino will:

• sense which button is pressed
• send messages like “happy” or “sad” to P5
• play an immediate tone or sound pattern on the buzzer depending on the emotion

P5 receives the emotion label from Arduino and triggers:

• an animated visual (color changes, shapes, movement)
• a matching digital sound or short audio clip
• Maybe text labels for clarity

Interaction Flow

1. User presses a button
2. Arduino detects it instantly
3. Arduino plays a corresponding tone on the buzzer
4. Arduino sends the emotion name to P5
5. P5 displays a visual animation + plays a sound + maybe led lights up

Ideas I can add:
1. Users can press two or more buttons in sequence to express combined emotions (e.g., Happy + Hungry → “I’m happy and hungry.”)

2. As positive reinforcement is something that plays a big role in this, an extra large button can trigger a celebratory animation and sound when the user successfully expresses an emotion or sequence. to motivate them to continue using this tool as a way to express their feelings better.

Future Expansion

Later, this system could be expanded with:

• more advanced sensors
• a larger emotion vocabulary
• more meaningful sound design for adults
• different modes of communication (LEDs, vibration feedback)

Before reading Graham Pullin’s Design Meets Disability, I thought about assistive technology in a pretty simple way: the better the design, the more it should be invisible. Like a lot of people, I assumed that the goal for things like hearing aids or prosthetic limbs was for them to blend in, so users could just “look normal” and not stand out.

Pullin completely changed the way I think. He points out that trying to hide a device can actually make people feel like disability is something to be ashamed of. Instead, designers should think about fashion, creativity, and self-expression.

The example he gives with eyeglasses really hit me. Glasses used to be seen as a weakness, but now they’re a fashion statement – people even wear frames without lenses! That shift happened because designers stopped trying to hide them and started making them bold and stylish. I realized it’s not visibility that’s the problem, but the lack of thoughtful, positive design.

I really agree with Pullin that we should stop seeing users as “patients” who need fixing, and start seeing them as “wearers” who can express themselves. Athletes like Aimee Mullins show this perfectly, treating her prosthetic legs as works of art or fashion. She’s not just adapting to a missing limb-she’s making a confident personal choice through design. This made me see that style can empower, rather than hide, disability.

Exercise 1: Arduino to p5.js  – Potentiometer

Concept

This project shows how a physical sensor can control a digital object. A potentiometer on the Arduino sends its values to p5.js . In the sketch, these values move an ellipse left or right on the screen. Turning the knob changes the ellipse’s position instantly.

Arduino Code

The Arduino reads the potentiometer value and sends it over serial:

int interval = 100; // milliseconds between readings

void setup() {
  Serial.begin(9600); // start serial communication
}

void loop() {
  int potValue = analogRead(A1); // read potentiometer
  int mappedValue = map(potValue, 0, 1023, 0, 255); // scale value
  Serial.println(mappedValue); // send value to p5.js
  delay(interval); // avoid flooding serial
}

p5.js Code

The p5.js sketch reads the serial data and moves the ellipse:

let positionX = 0;
let port;
let sensorValue = 0;

function setup() {
  createCanvas(400, 400);
  positionX = width / 2;

  port = createSerial();
  let used = usedSerialPorts();
  if (used.length > 0) port.open(used[0], 9600);
}

function draw() {
  background(220);

  let line = port.readUntil("\n");
  if (line.length > 0) {
    sensorValue = int(line.trim());
  }

  positionX = map(sensorValue, 0, 255, 0, width);

  ellipse(positionX, height / 2, 50, 50);
}

Video of the final result: Video

Exercise 2: p5.js-to-Arduino LED Control

Concept

This project demonstrates how a digital signal from p5.js can control hardware on Arduino. Using a p5.js sketch, we can send commands over serial to turn an LED on or off.

Arduino Code

The Arduino listens for serial commands from p5.js and controls the LED accordingly:

The Arduino listens for serial commands from p5.js and controls the LED accordingly:
int ledPin = 13; // onboard LED

void setup() {
  pinMode(ledPin, OUTPUT);
  Serial.begin(9600); // start serial communication
}

void loop() {
  if (Serial.available() > 0) {
    String command = Serial.readStringUntil('\n'); // read command
    command.trim(); // remove whitespace

    if (command == "ON") {
      digitalWrite(ledPin, HIGH); // turn LED on
    } 
    else if (command == "OFF") {
      digitalWrite(ledPin, LOW); // turn LED off
    }
  }
}

p5.js Code

The p5.js sketch sends commands to the Arduino based on events:

let port;
let connectBtn;

function setup() {
  createCanvas(400, 400);
  background(220);

  // initialize serial connection
  port = createSerial();
  let used = usedSerialPorts();
  if (used.length > 0) port.open(used[0], 9600);

  connectBtn = createButton("Connect");
  connectBtn.position(10, 10);
  connectBtn.mousePressed(toggleConnection);
}

function draw() {
  background(220);
}

function keyPressed() {
  if (port.opened()) {
    if (key === 'L') {         // press L to turn LED on
      port.write("ON\n");
    } else if (key === 'K') {  // press K to turn LED off
      port.write("OFF\n");
    }
  }
}

Video of the final result: Video

Exercise 3: Bidirectional: Controlling LED with Bouncing Ball and Controlling Wind with Potentiometer

Concept

This project demonstrates bidirectional communication between Arduino and p5.js, combining physics simulation with hardware interaction. In this, A ball bounces on the p5.js canvas, following simple gravity physics. The horizontal wind affecting the ball is controlled by a potentiometer connected to the Arduino. Turning the knob changes the wind force in real-time. Every time the ball hits the floor, p5.js sends a “BOUNCE” signal to Arduino, lighting up an LED. When the ball stops bouncing, p5.js sends a “STOP” signal, turning the LED off.

Arduino Code

Arduino reads the potentiometer and controls the LED based on serial commands:

int ledPin = 13;   // LED pin
int potPin = A1;   // potentiometer pin
int potValue = 0;

void setup() {
  pinMode(ledPin, OUTPUT);
  Serial.begin(9600);
}

void loop() {
  // Read potentiometer value
  potValue = analogRead(potPin);
  Serial.println(potValue);  // send to p5.js

  // Check for incoming serial commands
  if (Serial.available() > 0) {
    String command = Serial.readStringUntil('\n');
    command.trim();

    if (command == "BOUNCE") {
      digitalWrite(ledPin, HIGH);   // light LED on bounce
    } else if (command == "STOP") {
      digitalWrite(ledPin, LOW);    // turn LED off when ball stops
    }
  }

  delay(50); // small delay for stability
}

p5.js Code

The p5.js sketch simulates a bouncing ball and sends commands to Arduino:

let velocity, gravity, position, acceleration, wind;
let mass = 50;
let drag = 0.99;

// SERIAL
let port;
let sensorValue = 0;

function setup() {
  createCanvas(640, 360);
  noFill();

  // physics
  position = createVector(width/2, 0);
  velocity = createVector(0, 0);
  acceleration = createVector(0, 0);
  gravity = createVector(0, 0.5 * mass);
  wind = createVector(0, 0);

  // serial
  port = createSerial();
  let used = usedSerialPorts();
  if (used.length > 0) port.open(used[0], 9600);
}

function draw() {
  background(255);

  // read potentiometer from Arduino
  let line = port.readUntil("\n");
  if (line.length > 0) sensorValue = int(line.trim());
  wind.x = map(sensorValue, 0, 1023, -1, 1);

  // apply forces
  applyForce(wind);
  applyForce(gravity);

  velocity.add(acceleration);
  velocity.mult(drag);
  position.add(velocity);
  acceleration.mult(0);

  ellipse(position.x, position.y, mass, mass);

  // bounce detection
  if (position.y > height - mass/2) {
    position.y = height - mass/2;

    if (abs(velocity.y) > 1) {
      velocity.y *= -0.9;
      if (port.opened()) port.write("BOUNCE\n"); // LED on
    } else {
      velocity.y = 0;
      if (port.opened()) port.write("STOP\n");   // LED off
    }
  }
}

function applyForce(force) {
  let f = p5.Vector.div(force, mass);
  acceleration.add(f);
}

Video of the final result: Video

Challenges and Improvements

In all three projects, the hardest part was making sure the Arduino and p5.js talked to each other correctly. Sometimes the data from the potentiometer didn’t come through clearly, which made the ellipse or bouncing ball move strangely or the LED turn on and off at the wrong time. It was also tricky to scale the sensor values so they controlled the visuals in a smooth way. In the bouncing ball project, making the LED light only when the ball bounced was challenging because the ball slowed down naturally. To improve, I could smooth the sensor readings to make movement less jumpy, make the ball bounce more realistically, or add more sensors or LEDs for extra interaction. These changes would make the projects work better and feel more interactive.

Ref:  In the last exercise, ChatGPT helped us by explaining and guiding the use of p5.Vector, physics calculations, and serial communication, which made Implementation easier.

Concept  : Voice-Controlled Car

The main idea is to build a car that responds to spoken words like “forward,” “left,” “right,” and “stop.”
Instead of a remote control or buttons, the user interacts only through speech, making the experience feel natural and intuitive.

Interaction (How it Works)

On the laptop, a p5.js sketch uses the SpeechRec library to listen through the microphone.
When the user says a direction:

• “forward” → the car should drive straight

• “left” → the car turns left

• “right” → the car turns right

• “stop” → the car stops moving

p5.js identifies the spoken word and sends the corresponding command through serial communication to the Arduino.

The Arduino receives these simple messages and activates the correct motors to move the car.
The entire system becomes a loop of:

You speak → p5 listens → p5 processes → Arduino moves the car

The goal is to make it feel almost like the car is “listening” and responding to the user in real time.

Arduino Components

• 1 Arduino Uno

• 2 DC motors (for movement and turning)

• 1 motor driver (L298N or similar)

• Wheels + chassis

• External battery pack for the motors

• USB cable for serial communication

I might later add optional components like LED indicators or a small buzzer that reacts to commands, depending on how the project develops.

Why I Like This Idea

I like this concept because it feels fun, intuitive, and a bit magical.
Anyone can walk up and immediately understand how it works since people naturally speak commands like “go” or “stop.”

Exercise 1: Arduino to p5.js  – Potentiometer

Concept

This project shows how a physical sensor can control a digital object. A potentiometer on the Arduino sends its values to p5.js . In the sketch, these values move an ellipse left or right on the screen. Turning the knob changes the ellipse’s position instantly. 

Arduino Code

The Arduino reads the potentiometer value and sends it over serial:

int interval = 100; // milliseconds between readings

void setup() {
  Serial.begin(9600); // start serial communication
}

void loop() {
  int potValue = analogRead(A1); // read potentiometer
  int mappedValue = map(potValue, 0, 1023, 0, 255); // scale value
  Serial.println(mappedValue); // send value to p5.js
  delay(interval); // avoid flooding serial
}

p5.js Code

The p5.js sketch reads the serial data and moves the ellipse:

let positionX = 0;
let port;
let sensorValue = 0;

function setup() {
  createCanvas(400, 400);
  positionX = width / 2;

  port = createSerial();
  let used = usedSerialPorts();
  if (used.length > 0) port.open(used[0], 9600);
}

function draw() {
  background(220);

  let line = port.readUntil("\n");
  if (line.length > 0) {
    sensorValue = int(line.trim());
  }

  positionX = map(sensorValue, 0, 255, 0, width);

  ellipse(positionX, height / 2, 50, 50);
}

Video of the final result : Video

Excercise 2 : p5.js-to-Arduino LED Control

Concept

This project demonstrates how a digital signal from p5.js can control hardware on Arduino. Using a p5.js sketch, we can send commands over serial to turn an LED on or off.

Arduino Code

The Arduino listens for serial commands from p5.js and controls the LED accordingly:

The Arduino listens for serial commands from p5.js and controls the LED accordingly:
int ledPin = 13; // onboard LED

void setup() {
  pinMode(ledPin, OUTPUT);
  Serial.begin(9600); // start serial communication
}

void loop() {
  if (Serial.available() > 0) {
    String command = Serial.readStringUntil('\n'); // read command
    command.trim(); // remove whitespace

    if (command == "ON") {
      digitalWrite(ledPin, HIGH); // turn LED on
    } 
    else if (command == "OFF") {
      digitalWrite(ledPin, LOW); // turn LED off
    }
  }
}

p5.js Code

The p5.js sketch sends commands to the Arduino based on events:

let port;
let connectBtn;

function setup() {
  createCanvas(400, 400);
  background(220);

  // initialize serial connection
  port = createSerial();
  let used = usedSerialPorts();
  if (used.length > 0) port.open(used[0], 9600);

  connectBtn = createButton("Connect");
  connectBtn.position(10, 10);
  connectBtn.mousePressed(toggleConnection);
}

function draw() {
  background(220);
}

function keyPressed() {
  if (port.opened()) {
    if (key === 'L') {         // press L to turn LED on
      port.write("ON\n");
    } else if (key === 'K') {  // press K to turn LED off
      port.write("OFF\n");
    }
  }
}

Video of the final result : Video

Excercise 3 : Bidirectional : Bouncing Ball with LED and Potentiometer

Concept

This project demonstrates bidirectional communication between Arduino and p5.js, combining physics simulation with hardware interaction. In this : A ball bounces on the p5.js canvas, following simple gravity physics. The horizontal wind affecting the ball is controlled by a potentiometer connected to Arduino. Turning the knob changes the wind force in real-time. Every time the ball hits the floor, p5.js sends a “BOUNCE” signal to Arduino, lighting up an LED. When the ball stops bouncing, p5.js sends a “STOP” signal, turning the LED off.

Arduino Code

Arduino reads the potentiometer and controls the LED based on serial commands:

int ledPin = 13;   // LED pin
int potPin = A1;   // potentiometer pin
int potValue = 0;

void setup() {
  pinMode(ledPin, OUTPUT);
  Serial.begin(9600);
}

void loop() {
  // Read potentiometer value
  potValue = analogRead(potPin);
  Serial.println(potValue);  // send to p5.js

  // Check for incoming serial commands
  if (Serial.available() > 0) {
    String command = Serial.readStringUntil('\n');
    command.trim();

    if (command == "BOUNCE") {
      digitalWrite(ledPin, HIGH);   // light LED on bounce
    } else if (command == "STOP") {
      digitalWrite(ledPin, LOW);    // turn LED off when ball stops
    }
  }

  delay(50); // small delay for stability
}

p5.js Code

The p5.js sketch simulates a bouncing ball and sends commands to Arduino:

let velocity, gravity, position, acceleration, wind;
let mass = 50;
let drag = 0.99;

// SERIAL
let port;
let sensorValue = 0;

function setup() {
  createCanvas(640, 360);
  noFill();

  // physics
  position = createVector(width/2, 0);
  velocity = createVector(0, 0);
  acceleration = createVector(0, 0);
  gravity = createVector(0, 0.5 * mass);
  wind = createVector(0, 0);

  // serial
  port = createSerial();
  let used = usedSerialPorts();
  if (used.length > 0) port.open(used[0], 9600);
}

function draw() {
  background(255);

  // read potentiometer from Arduino
  let line = port.readUntil("\n");
  if (line.length > 0) sensorValue = int(line.trim());
  wind.x = map(sensorValue, 0, 1023, -1, 1);

  // apply forces
  applyForce(wind);
  applyForce(gravity);

  velocity.add(acceleration);
  velocity.mult(drag);
  position.add(velocity);
  acceleration.mult(0);

  ellipse(position.x, position.y, mass, mass);

  // bounce detection
  if (position.y > height - mass/2) {
    position.y = height - mass/2;

    if (abs(velocity.y) > 1) {
      velocity.y *= -0.9;
      if (port.opened()) port.write("BOUNCE\n"); // LED on
    } else {
      velocity.y = 0;
      if (port.opened()) port.write("STOP\n");   // LED off
    }
  }
}

function applyForce(force) {
  let f = p5.Vector.div(force, mass);
  acceleration.add(f);
}

Video of the final result : Video

Challenges and Improvements

In all three projects, the hardest part was making sure the Arduino and p5.js talked to each other correctly. Sometimes the data from the potentiometer didn’t come through clearly, which made the ellipse or bouncing ball move strangely or the LED turn on and off at the wrong time. It was also tricky to scale the sensor values so they controlled the visuals in a smooth way. In the bouncing ball project, making the LED light only when the ball bounced was challenging because the ball slowed down naturally. To improve, I could smooth the sensor readings to make movement less jumpy, make the ball bounce more realistically, or add more sensors or LEDs for extra interaction. These changes would make the projects work better and feel more interactive.

Ref:  In the last exercise, ChatGPT helped us by explaining and guiding the use of p5.Vector, physics calculations, and serial communication, which made Implementation easier.

For my final project, I’m currently deciding between two game concepts. I’m still brainstorming and working on the overall ideas, so since I haven’t fully decided what to go with yet, I’m documenting both possible directions.

Concept 1 — Rhythm Game

This concept is based on rhythm games like Dance Dance Revolution, but turned into a tabletop physical controller. Instead of tapping keys on a keyboard, players interact with physical buttons connected to an Arduino that trigger actions in p5.js.

Interaction (How it Works)

On the p5.js screen, colored notes fall from the top to the bottom. The player has 4–6 physical push buttons arranged in a row, each one corresponding to a lane on the screen. When a falling note reaches the bottom marker, the player must press the matching physical button at the right moment.

The game gives immediate feedback, such as:

• Perfect hits respond with a sound and/or visual flash
• A miss is indicated with a red flash
• The score updates in real time

Arduino Components

• 4–6 push buttons
• 1 potentiometer

The potentiometer will control the difficulty level by adjusting the speed of the falling notes (higher difficulty = faster notes).

I like this idea because I think people will naturally want to try it, and it’s generally easy to understand without needing much explanation.

Concept 2 — Wizard Duel (Mashing Game)

This concept is a 2-player competitive game inspired by Harry Potter wand duels and by games like “Wizard” on 1-2-Switch. Each player has a physical button, and the faster they mash their button, the more they push their spell beam toward the opponent.

Interaction (How it Works)

The p5.js screen shows two wizards on opposite sides. Player 1 has one button, Player 2 has another. When the game starts (using a separate start/reset button):

• Both players mash their button as fast as possible
• The faster player pushes the spell beam toward the opposite side
• If the beam reaches Player 2’s side, Player 1 wins
• If it reaches Player 1’s side, Player 2 wins

Arduino Components

• 1 button to start/reset
• 2 buttons (one for each player to mash)

I might also add LEDs for each player that flash with every mash (not 100% sure yet, I’ll see what works best). I’m also open to adding more components if it makes the game more interesting.

I like this idea because I love competitive games, especially mashing games. I used to play them all the time on game consoles with my sisters.

Conclusion

I still need to decide which concept I’ll fully commit to (and who knows, I might even change my entire idea by next week). Over the next few days, I’ll prototype small parts of each idea to see which one feels more engaging and fun for me to complete.

Exercise 1

For this exercise, we decided to use Ultrasonic sensor because we thought it will interesting to control the ellipse with our hand kind of like in video games Wii. Wherein based on your physical hand movement the objects move on the screen. To proceed with this idea, we first made a connection to the Arduino on P5.js, similar to how we did it in class. Then to control the x-axis we first read the serial write in P5.js and then mapped the value read from 0 to windowWidth. Another thing we noticed while performing this assignment is that the movement for ellipse was very abrupt and so we used a function called ‘lerp’. I had previously used this function for one of the P5 assignments and so I recalled that to smooth distance we can use this function. This function basically generates a value between my last x position to the current measured distance at certain interval (0.1 in this case). This makes sure that the circle moves in a smooth movement. Below I have attached my code to P5 and Arduino along with the demonstration video.

P5.js

Arduino

assignment1-demo

Exercise 2

For the second exercise, we chose to create a visual representation of the conversion from one side to another. Inspired by the chess clock, where time shifts between players while playing. We designed the LEDs so that as one grows brighter, the other dims, in turn symbolizing the passing of control from one side to the other. We started by first establishing the serial connection between Arduino and p5.js, similar to the previous exercise and what we did in class. In the p5 we created a slider ranging from 0 to 255, which we used to determine the brightness of LED 1, then for LED 2 we then set the inverse of that value so that as one increased, the other decreased. We continuously sent these two mapped values through serial in the format “value1,value2” and read them on the Arduino side to update the LED brightness using analogWrite. This setup allowed us to control both LEDs simultaneously from the browser and visually see the transition between them. Below are the p5.js and Arduino code along with the demonstration video.

P5.js

Arduino

Arduinoexercise2-demo

Exercise 3

To complete this exercise we made changes to the gravity ball code provided. We first made sure that the LED lights up everytime the ball bounces to do so we maintained a state variable so everytime the ball touches ground we switched the state to HIGH else we set the state to LOW. This helped us achieve the goal. For the second module of this exercise, we decided to use our concept in exercise 1 i.e. ultrasonic sensor to control the wind movement of the ball. We looked at our distance from ultrasonic sensor and then decided to set a threshold of 50. So if the distance is >50, we set the wind speed to -3, else 3. This helped to move the ball in different directions and control it with our hand. We have provided the P5.js and Arduino code below for your references.

P5.js

Arduino

exercise3-demo

Concept (1)

For my final project, I am creating Doom Drive, an interactive driving game where the player navigates a chaotic road using a physical steering wheel while the environment itself actively sabotages the world. The idea behind Doom Drive is to turn sensory interruption into a playable mechanic. Instead of relying solely on player input, the game listens to real-time changes in the environment, particularly light, and transforms those changes into unpredictable events on the road.

The goal is to explore how external stimuli can impact focus, control, and experience. By letting the IM Fair audience unintentionally (or intentionally) alter the driving conditions, the installation becomes social, performative, and slightly overwhelming in a comedic way. Doom Drive exaggerates the feeling of “everything is happening all at once” and turns overstimulation into an entertaining challenge.

Arduino Inputs (Listening)

The physical interaction relies on two sensors connected to the Arduino:

Potentiometer (Mounted Inside the Steering Wheel)
Controls the player’s steering in real time.
Turning the wheel left or right rotates the potentiometer.
Arduino sends smooth analog values to p5.js to control car movement.

Light Sensor (Photoresistor)
Reads ambient light levels in the environment.
Flashes, shadows, or sudden brightness changes trigger in-game chaos.
Different light thresholds activate different game modes, such as:
Glare Mode (screen washout and reduced visibility)
Night Mode (darkness and narrow road)
Tunnel Mode (sharp directional constraints)
Panic Mode (screen shake and heavy obstacles)

Arduino continuously sends both sensor readings to the computer through serial communication. Together, these inputs give the game a mixture of user control and environmental unpredictability.

p5.js Output (Thinking and Speaking)

The p5.js sketch interprets the incoming sensor data and constructs the game world dynamically.

Steering values from the potentiometer directly move the player’s car left and right.
Light sensor fluctuations trigger visual effects, obstacle frequency, color palette shifts, and difficulty spikes.
p5.js uses thresholds and cooldowns so the game remains challenging but still playable in a busy environment like the IM Fair.

The screen speaks back to the player through animation, color changes, distortion, and motion. For example, covering the light sensor might plunge the game into sudden darkness, while a bright flash could create a blinding glare effect.

What I Want to Explore

I chose this project because it blends physical and digital interaction in a way that feels both intuitive and chaotic. Driving is a familiar, easy action for players to understand, but the environment’s influence transforms it into something playful and unpredictable. By allowing the room, its lighting, its people, and its energy to directly affect the game world, Doom Drive becomes a shared experience rather than an isolated one.

This project also lets me explore sensory overload as a mechanic rather than a narrative. Instead of explaining it, the game shows it: the road changes, the screen shifts, and the player has to adapt to external forces beyond their control.

Concept (2)

For my final project, I am creating an interactive dance game that combines body tracking with physical floor pads. The player follows on-screen prompts and uses both their feet and upper body to complete dance moves. The game uses p5.js with a pose detection library (similar to what we saw in class) to track body gestures through the webcam, while Arduino handles input from floor pads and other physical controls.

The goal of the project is to explore how digital choreography can be shaped by both full-body movement and tactile interaction. Instead of limiting the experience to either camera tracking or button pressing, this project blends both. The player has to move their body in space and also step on specific pads at the right time, which makes the game feel more physical and performance-like. I want the experience to be playful and encouraging, with visual feedback such as “Good,” “Great,” or “Miss” appearing on the screen depending on timing and accuracy.

Arduino Inputs (Listening)

The Arduino focuses on physical interaction through the feet and optional controls:

Floor Pads

I will create two or three simple dance pads on the floor, each acting as a digital input to the Arduino. These can be built using large buttons or DIY pressure pads. Each pad corresponds to a direction or beat:

• Left pad (for left-step moves)

• Right pad (for right-step moves)

• Optional center pad (for jumps or special moves)

When the player steps on a pad, the Arduino registers a press and sends a message to p5.js over serial communication indicating which pad was activated.

Additional Controls (not sure yet)

• Potentiometer: used as a physical knob to select difficulty level or song choice before the game starts.

• Light Sensor: can be used to trigger “disco mode” or visual effects when the environment changes, for example when someone shines a light or covers the sensor.

Arduino continuously sends the state of the pads (and any extra sensors) to the computer. This gives the system a clear sense of when and where the player is stepping.

p5.js Output (Thinking and Speaking)

The p5.js sketch is responsible for three main elements:

1. Dance Prompts and Timing
   p5 shows prompts on the screen, such as arrows, circles, or icons that represent foot placement and upper body moves. These prompts appear in time with a rhythm or song. For example, a left arrow might indicate “step on left pad,” while an icon near the top of the screen might indicate “raise your hands.”

2. Body Tracking
   Using a body tracking library with p5 (such as PoseNet), the program detects key points on the player’s body, like hands, arms, and head. It checks whether the player’s pose matches the required gesture at the correct moment. This allows the game to recognize upper body moves, such as raising both hands, leaning to one side, or performing a simple arm gesture.

3. Feedback and Scoring
   When Arduino sends a pad press and p5 confirms that the player also matched the required pose (or at least the timing), the game displays feedback like “Good,” “Great,” or “Miss.” It also keeps track of score or combo streaks. Visual feedback can include color changes, particle bursts, or short animations to make the experience feel rewarding and energetic.

The screen functions as the “instructor” and “judge,” guiding the player and showing how well they are doing.

What I Want to Explore

I chose this project because it brings together several aspects of interaction that I find interesting: full-body movement, tactile input, and real-time visual feedback. Dance games are usually either camera-based or pad-based, but rarely both at once. By combining floor pads (through Arduino) with body tracking (through p5.js), I want to design a system that feels more embodied and theatrical.

I am also interested in how feedback can encourage people rather than intimidate them. Instead of focusing on strict accuracy, I want the game to celebrate participation and motion, with clear but playful responses like short messages, bursts of color, and simple scores. At the IM Fair, the goal is for people to feel comfortable stepping up, moving around, and laughing at their attempts, while still experiencing a technically structured interactive system that listens, thinks, and responds across both hardware and software.