Final Project Proposal – Week 12

For my original concept, I planned to build an Emotion Communicator Keyboard designed mainly for babies and people with speech difficulties. Although meaningful, I realized the idea was too simple, too targeted to a specific group, and not interactive enough for a final project. After discussing the importance of two-way interaction and creating something that pulls reactions out of the user, I shifted my approach. I wanted something that anyone (children, teenagers, and adults) would find fun, challenging, and engaging. From that brainstorming, I developed my finalized concept: Emotion Sprint, a fast-paced reflex game that uses a custom Arduino emotion keyboard and a p5.js game interface.

Emotion Sprint is an interactive reaction-based game where emojis, expressions, or short emotional prompts appear on the p5 screen, and the user must press the matching physical emotion button at the right moment. The game mixes fast visuals, timing accuracy, audio feedback, and emotional recognition. As the game progresses, objects fall faster, the reactions get more complex, and the player must instantly choose the correct emotion.

This concept is accessible, replayable, competitive, and genuinely exciting for all ages. The physical interface (Arduino keyboard) and digital interface (p5.js animations and timing) depend on each other to create a fast, tight feedback loop.

Physical Sketch

Hardware Elements

  • 10–12 buttons for the Emotion Keyboard
  • Cardboard
  • LED Pixels (feedback indicators)
  • Piezo buzzer (tone feedback)
  • Arduino Uno
  • Resistors + wiring

Arduino Design and Description

Inputs (Reading User Reactions): Arduino acts as the physical senses + physical voice of the game

  • Continuously read digital pins for button presses
  • Debounce inputs to ensure clean, precise timing
  • Detect when a player presses multiple buttons quickly

Outputs (Immediate Physical Feedback)

  • Trigger a short tone on the buzzer based on how accurate the timing was
    • Perfect → Higher pitch
    • Good → Mid pitch
    • Miss → Low “error” tone
  • Light up the LED under the pressed emotion button

Communication With P5
Arduino → p5 (Serial):

  • Sends emotion labels (ex: “happy”, “angry”, “surprised”)
  • Sends timestamps of button presses
  • Sends “double reaction” when two buttons are pressed fast
  • Sends “applause button” activation (for positive reinforcement moments)

p5 → Arduino (Serial):

  • Sends feedback signals such as:
    • “correct” → Arduino lights LED green
    • “incorrect” → LED flashes red
    • “bonus” → special buzzer melody
  • Sends timing difficulty changes (game speed up/down)

P5 Design and Description

Game Engine

  • Generates falling emojis/emotions with target timing
  • Tracks the position and movement of each object
  • Calculates accuracy of each press using timestamps
  • Adjusts game difficulty (speed increases over time)

Visual Output

  • Display falling emojis
  • Flash screen colors based on success/failure
  • Show score, multiplier, streak
  • Show special video/image/meme prompts during reaction rounds
  • End-of-game summary screen

Audio Output

  • Play digital sound effects for:
    • Correct timing
    • Misses
    • Combos
    • Bonus rounds

Communication With Arduino
p5 → Arduino:

  • Sends green/red flashing commands
  • Sends buzzer melody for bonus streak
  • Sends “new round” or “speed increase” signals

Arduino → p5:

  • Sends emotion pressed
  • Sends timestamp of press
  • Sends sequence or “combo input”
  • Sends “applause button” activation

I’ll be developing the p5 game first, focusing on the visuals, falling emojis, scoring, and special reaction rounds. Once the game mechanics are solid, I will integrate the Arduino keyboard to connect the physical inputs, buzzer, and LED feedback. As I continue working on the project, I will keep modifying and improving the system if I come up with new ideas or ways to make the gameplay more interactive and engaging. I will document all progress here on the blog, showing updates, refinements, and the evolving design of both the software and hardware components.

Week 12: Final Project Proposal and Progress

Concept

For my final project, I am building a Voice-Controlled Car.
The main idea is to create a small robotic vehicle that moves based on simple spoken commands such as “forward,” “left,” “right,” “backward,” and “stop.”

Instead of using a remote controller, buttons, or joysticks, the user interacts with the car entirely through speech. The goal is to make the interaction feel natural and intuitive, almost like the car is “listening” and responding in real time.

The project uses bi-directional communication between Arduino and p5.js:

  • p5.js listens to the user’s voice, recognizes the command, and sends it to the Arduino.

  • Arduino moves the motors and sends back status messages to p5.js.

This creates a complete loop where the user, p5.js, and the car are constantly communicating.

Arduino and P5 Communication

These are the main components I’m using in the project:

  • Arduino Uno

  • Adafruit Motor/Stepper/Servo Shield

  • 4 DC motors + wheels

  • Wooden board chassis

  • Battery pack (to power the motors)

  • USB cable or Bluetooth module (Adafruit Bluefruit EZ-Link)

  • Optional: LEDs or buzzer for feedback

Here is how the system is organized:

  • Microphone is connected to  p5.js to detect speech.

  • p5.js sends simple movement commands to the Arduino.

  • The Arduino controls the motors through the motor shield.

  • Arduino also sends responses back to p5.js to display status.

This makes the communication two-way, not just one-directional.

Arduino → P5 Communication

Every time the Arduino receives a command and performs the movement, it sends back a message so p5.js can update the interface.

For example: car_forward, car_left, car_right, car_backward, car_stopped

These messages allow the p5.js program to show real-time feedback about what the car is doing.
This also helps with testing and makes the experience feel more responsive.

Later on, this system can be expanded to include things like speed

P5 → Arduino Communication

The p5.js sketch uses the p5.SpeechRec() library to listen for specific keywords.
When a valid direction is heard, p5.js sends a short command to the Arduino through serial.

Spoken Word Code Sent to Arduino
forward F
left L
right R
backward B
stop S

Using one-letter codes keeps communication fast and reduces errors.

Arduino then reads the code and moves the motors accordingly using the Adafruit motor shield

Project Progress So Far

I began by building the physical structure of the car before working on the code.

What I have completed:

  • Attached four DC motors to the corners of my wooden chassis

  • Mounted the Adafruit Motor Shield onto the Arduino Uno

  • Fitted the wheels onto the motors

  • Tested spacing and placement for the battery pack

  • Ensured that the chassis is stable and balanced

  • Confirmed that all electronic components fit properly

Week 12: Final Project Proposal (New Idea) and Progress

Concept

Soo I ended up not doing any of my initial ideas (oops) because I came up with a more exciting one today while talking to my sister. Basically, this project will be an interactive time machine (of some sort) that lets the user travel to different years using physical controls, and choose to either play real music or news from that selected time period/year. My project is inspired by the computer game Club Penguin, specifically the game feature where the player completes missions as a spy for the PSA (Penguin Secret Agency) and uses gadgets to solve tasks. Similarly, my concept is that the player/user is assigned a mission to test out a new prototype time machine. (I’m also making the main screen kind of look like the start screen of the minigame Aqua Grabber in Club Penguin.)

Arduino and P5 Communication

These are the components I’m going to use in my project:

  • 2 buttons, where one button selects the music, and the other selects the news broadcast.
  • A potentiometer, chooses the user’s preferred year/time period that shows up on the screen.
  • One more button, kind of acts like a start button, which will play the sounds of music of news after either one is selected by the user.
  • 5 Green LEDs, each LED lights up just like the ones on the screen will.

Just to be clear, the potentiometer is the time period/year selector, and the buttons select either music or news to be played.

The main interface of this project will look like the start screen of one of the minigames in Club Penguin called Aqua Grabber, which looks like this:

Arduino to P5 Communication

Every time the potentiometer’s value changes by 100, the sketch on P5 will change the green lights (as shown in the picture) to the next one. So the first light is initially lit up since the potentiometer’s value is 0, and once it reaches 100, it changes to the next one (goes from left to right).

P5 to Arduino Communication

The specific green LED that’s lit up on the sketch on P5 will also trigger the corresponding green LED on the breadboard to light up as well.

How it Works
  1. User turns the potentiometer and chooses a time period/year
  2. P5 lights up the corresponding green light on the sketch
  3. Arduino lights the matching green LED on the breadboard
  4. User presses either the “music” or “news” button
  5. User presses the “start” button
  6. P5 plays music/news from the selected year/time period
Project Progress So Far

I worked on the Arduino part of it first. I attached 5 green LEDs and a potentiometer. As I explained before, the LEDs light up when the potentiometer’s value is within that specific LED’s range.

Here’s the code I created on Arduino IDE: FullCode

Here’s a video of it:

Final Project Progress

Finalised Concept for the Project

My final project is a dual-player wheel controlled dance survival game that uses both Arduino and p5.js to create a physically interactive digital experience. A physical wheel attached to a rotary encoder or potentiometer acts as the primary input device. When the player rotates the wheel, the Arduino reads real-time angle data and sends it to p5, which controls the horizontal position of two characters represented by circles inside a circular arena.

Random obstacles fall from the top of the p5 canvas, and the player must rotate the wheel to move both characters and avoid collisions. If either character touches a falling object, the game ends

Arduino Program:

Inputs:

  1. Potentiometer: measures the angle of the physical wheel. Maps the analogue input to degree values.
  2. This value is sent to p5 via Serial.

Outputs:

  1. Red / Green LED- for the disco effect of lights. Activates according to the song beats

P5 Program:

    1. Draw two player circles on the arena perimeter or surface.
    2. Spawn falling objects at random intervals and positions.
    3. Detect collisions between objects and characters.
    4. Read wheel angle data from Arduino
    5. Smoothly rotate the two characters horizontally based on the mapped value.
    6. Track the best score
    7. Play a song in the background and send instructions to Arduino to light up LEDs according to the beats

week 11 – in class exercises

For this week’s assignment, Zeina and I worked on three different exercises that focused on serial communication.

1- Make something that uses only one sensor  on Arduino and makes the ellipse in p5 move on the horizontal axis, in the middle of the screen, and nothing on arduino is controlled by p5

ARDUINO CODE

void setup() {
  Serial.begin(9600);
}

void loop() {
  int sensorValue = analogRead(A1); // 0–1023
  Serial.println(sensorValue);             // send to p5.js
  delay(50);
}

P5 CODE

let port;
let connectBtn;
let baudrate = 9600;
let lastMessage = "";
let sensorValue = 0;

function setup() {
  createCanvas(400, 400);
  background(220);

  //Setting the global variable port to a new serial port instance inside setup:
  port = createSerial();

  // we can open ports we have used previously without user interaction
  let usedPorts = usedSerialPorts(); //array of used ports
  if (usedPorts.length > 0) {  
    port.open(usedPorts[0], baudrate); //if any used port is in the array, open that port with 9600 baudrate
  }

  // any other ports (new ones) can be opened via a dialog after user interaction (see connectBtnClick below)
  connectBtn = createButton("Connect to Arduino");
  connectBtn.position(width/2, 270);
  connectBtn.mousePressed(connectBtnClick);

}

function draw() {
  background("white");
 
  // Read from the serial port. This is a non-blocking function. If a full line has come in (ending in \n), it returns that text. If the full line is not yet complete, it returns an empty string "" instead.
  let str = port.readUntil("\n");
  if (str.length > 0) {   // if str -a string- has any characters
    // print(str);
    lastMessage = str;
    sensorValue = int(lastMessage);  
  }
 
  //draw ellipse mapped to horizontal axis
  let x = map(sensorValue, 0, 1023, 0, width);  
  ellipse(x, height / 2, 40, 40);
 
  // Display the most recent message
  text("Last message: " + lastMessage, 10, height - 20);

  // change button label based on connection status
  if (!port.opened()) {
    connectBtn.html("Connect to Arduino");
  } else {
    connectBtn.html("Disconnect");
  }
}

function connectBtnClick() {
  if (!port.opened()) {
    port.open("Arduino", baudrate);
  } else {
    port.close();
  }
}

 

 

2-Make something that controls the LED brightness from p5

DEMO

IMG_8392 (2)

ARDUINO CODE

// Week 12 Example of bidirectional serial communication

int leftLedPin = 2;
int rightLedPin = 5;

void setup() {
  Serial.begin(9600);

  pinMode(LED_BUILTIN, OUTPUT);
  pinMode(leftLedPin, OUTPUT);
  pinMode(rightLedPin, OUTPUT);

  digitalWrite(leftLedPin, HIGH);
  digitalWrite(rightLedPin, HIGH);
  delay(200);
  digitalWrite(leftLedPin, LOW);
  digitalWrite(rightLedPin, LOW);

  while (Serial.available() <= 0) {
    digitalWrite(LED_BUILTIN, HIGH);
    Serial.println("0,0");
    delay(300);
    digitalWrite(LED_BUILTIN, LOW);
    delay(50);
  }
}

void loop() {
  while (Serial.available()) {
    digitalWrite(LED_BUILTIN, HIGH);

    int left = Serial.parseInt();
    int right = Serial.parseInt();

    if (Serial.read() == '\n') {

      // -----------------------
      // ONLY CHANGE IS HERE:
      // -----------------------
      digitalWrite(leftLedPin, left);     // left stays ON/OFF
      analogWrite(rightLedPin, right);    // right is now BRIGHTNESS (0–255)
      // -----------------------

      int sensor = analogRead(A0);
      delay(5);
      int sensor2 = analogRead(A1);
      delay(5);
      Serial.print(sensor);
      Serial.print(',');
      Serial.println(sensor2);
    }
  }
  digitalWrite(LED_BUILTIN, LOW);
}

p5 CODE

let port; // making a var to hold the serial port
let baudrate = 9600; // speed for talking to arduino
let brightnessSlider; // slider to pick brightness
let smoothBrightness = 0; //transition into the brightness instead of jumping

function setup() {
  createCanvas(400, 200); // just making a small canvas for ui
  textSize(18); // bigger test

  brightnessSlider = createSlider(0, 255, 0); // slider from 0 to full bright
  brightnessSlider.position(20, 80); // where it shows up on screen
  brightnessSlider.style('width', '200px'); // make it a bit wider

  port = createSerial(); // create a serial object so we can connect to arduino

  let used = usedSerialPorts(); // check if we already used a port before
  if (used.length > 0) {
    port.open(used[0], baudrate); // auto connect to the last used port
  }
}

function setupSerial() {
  if (!port.opened()) { // if no connection yet
    port.open("Arduino", baudrate); // try to open one
  } else {
    port.close(); // if already open then close it (toggle)
  }
}

function draw() {
  background(240); // light grey

  if (!port.opened()) {
    text("Press SPACE to connect", 20, 30); // tell the user what to do
  } else {
    text("Connected!", 20, 30); // connection message
  }

  let target = brightnessSlider.value(); // get the slider value

  // do transitional brightness
  smoothBrightness = lerp(smoothBrightness, target, 0.07);
 

  text("Brightness: " + int(smoothBrightness), 20, 70); // show the number

  // actually send the brightness to the arduino
  if (port.opened()) {
    let sendString = "0," + int(smoothBrightness) + "\n"; // left=0 right=smooth
    port.write(sendString); // send it over serial
  }
}

function keyPressed() {
  if (key === " ") {
    setupSerial(); // hitting space toggles the port connection
  }
}

3-Take the gravity wind example (https://editor.p5js.org/aaronsherwood/sketches/I7iQrNCul) and make it so every time the ball bounces one led lights up and then turns off, and you can control the wind from one analog sensor

ARDUINO CODE

void setup() {
  pinMode(2, OUTPUT);
  Serial.begin(9600);
}

void loop() {
  int sensorValue = analogRead(A1);
  Serial.println(sensorValue);

  //check for bounce
  if (Serial.available() > 0) {
    //if 1, light up led
    if (Serial.parseInt() == 1) {            
      digitalWrite(2, HIGH);
      delay(100);                          
      digitalWrite(2, LOW);
    }
  }
  delay(10);
}

P5.JS CODE

let velocity;
let gravity;
let position;
let acceleration;
let wind;
let drag = 0.99;
let mass = 50;
let port;
let connectBtn;
let baudrate = 9600;
let lastMessage = "";
let sensorValue = 0;

function setup() {
  createCanvas(640, 360);
  noFill();
  position = createVector(width / 2, 0);
  velocity = createVector(0, 0);
  acceleration = createVector(0, 0);
  gravity = createVector(0, 0.5 * mass);
  wind = createVector(0, 0);

  //Setting the global variable port to a new serial port instance inside setup:
  port = createSerial();

 
  // we can open ports we have used previously without user interaction
  let usedPorts = usedSerialPorts(); //array of used ports
  if (usedPorts.length > 0) {  
    port.open(usedPorts[0], baudrate); //if any used port is in the array, open that port with 9600 baudrate
  }
// any other ports (new ones) can be opened via a dialog after user interaction (see connectBtnClick below)
  connectBtn = createButton("Connect to Arduino");
  connectBtn.position(width/2, 270);
  connectBtn.mousePressed(connectBtnClick);
}

function draw() {
  background(255);
 
  // Read from the serial port. This is a non-blocking function. If a full line has come in (ending in \n), it returns that text. If the full line is not yet complete, it returns an empty string "" instead.
  let str = port.readUntil("\n");
  if (str.length > 0) {   // if str -a string- has any characters
    // print(str);
    lastMessage = str.trim();
    sensorValue = int(lastMessage);    
  }
 

  //wind controlled by analog sensor
  wind.x = map(sensorValue, 0, 1023, -1, 1);
  console.log("Sensor value: " + sensorValue);
 

  applyForce(wind);
  applyForce(gravity);

  velocity.add(acceleration);
  velocity.mult(drag);
  position.add(velocity);
  acceleration.mult(0);

  ellipse(position.x, position.y, mass, mass);

  // Bounce detection
  if (position.y > height - mass / 2) {
    velocity.y *= -0.9; // Dampening
    position.y = height - mass / 2;

    //send bounce signal to Arduino
    port.write("1\n");
  }
// Display the most recent message
  text("Last message: " + lastMessage, 10, height - 20);

  // change button label based on connection status
  if (!port.opened()) {
    connectBtn.html("Connect to Arduino");
  } else {
    connectBtn.html("Disconnect");
  }
}

function applyForce(force){
  // Newton's 2nd law: F = M * A
  // or A = F / M
  let f = p5.Vector.div(force, mass);
  acceleration.add(f);
}

function keyPressed(){
  if (keyCode==LEFT_ARROW){
    wind.x=-1;
  }
  if (keyCode==RIGHT_ARROW){
    wind.x=1;
  }
  if (key==' '){
    mass=random(15,80);
    position.y=-mass;
    velocity.mult(0);
  }
}

function connectBtnClick() {
  if (!port.opened()) {
    port.openDialog(); // user selects port
  } else {
    port.close();
  }
}

Final Project Proposal: Plant Care Station

Final Project Proposal: Plant Care Station

The final project idea I have in mind is to make a connection between the Arduino and p5.js to take care of plants. Through buttons the user can fill the meter for water (through a blue buttons), then add fertilizer (yellow button), to trim (red button), to move sunlight closeness(yellow buttons one for arrow up one for arrow down).

You can even get a score or a rating based on how well you took care of the plant. Checking each metric. The Arduino part would be multiple buttons and a box that covers these buttons with larger buttons on the wooden box (design is inspired by the beat hoven project displayed in class).

ARDUINO

p5

Final Project Idea

My project idea is to make a game or maybe some kind of art tool that uses a tilt controller I’m going to build myself. I might use an Adafruit Circuit Playground to be the controller, since it has an accelerometer inside to sense which way you’re tilting it. I plan to use the Arduino IDE to program the Circuit Playground and get the accelerometer data. Then, I want to send that data to my computer and use p5.js to make the actual game or art program. I’m not totally sure what the final thing will be, but it could be a maze game where you tilt to move the character, a simple pilot game, or a cool art program where the tilt moves the brush around instead of using arrow keys.

Week 11: Final Project Concept

Main Concept

The main concept for my final project is the “Lock In Buddy.” It basically never lets you stop studying. Your laptop acts as the monitor, and the system uses Arduino sensors and p5 to track your behavior while you are studying. If it detects your phone, it triggers a super loud alarm. If it senses that you’re leaving the table using a distance sensor, it activates the speaker and says, “Come back to the desk. You gotta lock in!” I also want to incorporate the Pomodoro technique, where you study for 25 minutes and take a 5-minute break. A student can press a physical button to start the timer, and press it again to stop it. At the end, the system will give you a score based on how distracted you were by environmental variables, like looking at people walking by, eating, or any other interruptions. Essentially, it becomes your study buddy until you graduate.

 

Lock In Buddy will have these features: 

    • Phone detection alarm 

It will detect when the user picks up a phone, uses, or brings their phone to a nearby desk. The monitor is always watching the user to make sure that they don’t use their phone. 

    • Leaving the desk detection 

The distance sensor detects when the user moves away from the study area. It plays a warning voice message saying “you gotta lock in buddy. Or you will become homeless”. 

    • Real time distraction detection 

The system detects whether you’re being distracted by environmental conditions. Those include looking at people passing by, eating, chair movement, or noise.

    • Pomodoro timer with physical buttons 

Users can press the button to start, pause and stop the Pomodoro session. Also, it will record how long you have been studying for, which is going to be recorded for the scoring system as well. 

    • Focus score generation 

P5 will calculate the number of distractions, phone usage count, time away from desk, noise levels, and the number of successful Pomodoro sessions.

 

How to integrate p5 and Arduino

Phone detection + alarm: 

    • p5.js + Webcam + ML5 COCO-SSD 

Or 

    • Arduino light sensor + phone jail box
    • Start playing the warning alarm saying “Lock in” in p5

 

Eating or talking: 

    • Sound sensor

 

Pomodoro Timer with physical buttons: 

    • Buttons with arduino

 

Leaving desk: 

    • Ultrasonic distance sensor 

 

Focus Score generation: 

    • Arduino continuously reads 
      • Light sensor → phone removed 
      • Ultrasonic sensor → left desk 
      • Sound sensor → eating/talking 
      • Button → start/stop Pomodoro
    • Send those data to p5 and calculate the score 

Week 11: Final Project Preliminary Concept

For the final project, I will create an interactive virtual gallery that will respond to the user input through a variety of sensors to guide them to a direction in the gallery. The works themselve are interactive as well, on a smaller scale, where each explore a different output, such as sound and visuals. The concept behind it is exploring the intersection between interactivity and the different themes we explored through class and the reading, to provide a viewer of the work with an engaging experience where they get to see a reflection of their actions, giving them control over the work.

Final Project Preliminary Concept: Motion-Responsive Interactive Art Installation

For my final project, I want to create a motion-responsive interactive installation. The idea is that a user’s movements will be sensed by Arduino sensors, like an ultrasonic sensor, and sent to p5.js, which will process the input and generate dynamic visuals. I haven’t yet decided on the exact type of art to display; it could be abstract particle systems, animated images, or other digital creations. However, the key is that the visuals will respond in real-time to the user’s motion, making the experience immersive and engaging.

A recent trip I made to teamLab Phenomena, where I experienced installations that react to human gestures, is the inspiration behind this project. For example, moving through a space could cause flowers to bloom digitally or change the colors of projected visuals. I want to recreate that sense of wonder and interaction in my own project, where the user feels connected to the digital environment.

This project will be bidirectional.

  • Arduino → p5.js: The user’s gestures are sensed by Arduino and drive the visuals, controlling motion, shape, size, or color of the on-screen elements.

  • p5.js → Arduino: When the visuals reach certain thresholds, for instance, a cluster of particles grows large, or an animated image reaches a boundary, p5.js will send meaningful signals back to Arduino. These signals could trigger LEDs or small motors to give tactile feedback, making the interaction feel alive and responsive. This way, the user sees the effect on screen and feels it physically, creating a continuous loop of action and response.