Category: S2026 – Aya

I was inspired by Color Twister, a game where players follow color-based instructions by placing their hands and feet on matching spots. I wanted to take that idea further and make it less about quick reactions and more about thinking, memory, and control, so I came up with Echo Move.

In this game, players are shown a sequence of visual signals on screen for a few seconds. Once it disappears, they have to recall and perform the sequence using their body on the “Echo Move Board,” which is connected to an Arduino. The challenge is to remember the information and use it accurately on time.

Echo Move doesn’t just tell where to step, it also tells the player how to interact. When the pink color appears in the sequence, the player should step into the zone and hold it until the level ends. If it’s yellow, the player should step once and remove it afterward. Holding too long or stepping on it again results in a loss. If it’s black, the player should be aware that it is only for deceit. They should avoid it completely, even if it appears in the sequence. Because of this, players have to interpret each signal before acting. It’s not just about memorizing positions, it’s about understanding instructions, timing movements, and controlling the body.

The game has four levels. Each level gets harder. It adds more complex sequences, tighter timing, and trickier signals. These elements test the player’s memory and decision-making skills.

Another key aspect is that the physical board itself has no colors, only neutral zones. All cues come from the screen, which means players must mentally map what they see onto the board. This adds abstraction. It forces them to turn visual cues into actions without direct help.

P5 and Arduino

In this game, P5 handles the game logic, visuals, and state control, while Arduino handles the physical input.

Each zone on the Echo Move board uses force-sensitive resistors (FSRs) to detect when a player presses on it. Arduino constantly reads these sensors and converts them into simple signals, such as pressed or not pressed. In addition to the FSR sensors, the Arduino will also read inputs from physical buttons connected to the system, including on/off, play, and pause/stop. These buttons control the state of the game and send signals to P5 when pressed. Arduino then sends all input data to P5 through serial communication in real time.

The P5 program, on the other hand, controls the game logic and what the player sees on screen. It displays the sequence of colors and signals that the player needs to remember. After a few seconds, the sequence disappears, and the player must perform it on the board. As Arduino sends input data, P5 reads both the sensor inputs and button signals. The buttons are used to control the flow of the game, such as starting, pausing, or stopping. P5 then checks if the player is following the correct order, timing, and actions, such as holding, tapping, or avoiding certain zones. Based on this, P5 decides if the player wins or loses and can send a message back to Arduino to control the game state. 

Stipend Usage

I’ll be using my stipend to buy the main parts I need for the Echo Move board, especially the force-sensitive resistors (FSRs) and longer jumper wires. These will help me properly detect player input and make sure the connections across the board are stable and flexible.

Preliminary Concept: Pressure-Based Digital Instrument

For my final project, I plan to create a physically interactive musical instrument using Arduino and p5.js. The idea is to build a simple keyboard-like interface where users can play notes by touching pressure sensors.

Instead of using buttons, I will use Force Sensitive Resistors (FSRs) so that the interaction feels softer and more natural, similar to placing fingers on a piano. Each sensor will represent one note (C, D, E, F, G, A, B, C, including sharps/flats).   Below picture will have only 4 FSR406 and 4 FSR402, I will buy them more and also 16 channel analog multiplexer to complete my project soon. 

Arduino will read the sensor values and send the data to p5.js through serial communication. In p5, the input will be used to generate sound and simple visual feedback. I will use the computer’s speaker for audio output instead of a buzzer to improve sound quality.

Since I need to use multiple sensors, I will use a CD74HC4067 multiplexer to expand the number of inputs. This allows Arduino to read many FSRs efficiently.

The system will also include a feedback loop. For example, when a note is played, p5 can send a signal back to Arduino to turn on an LED. This creates a simple two-way communication between Arduino and p5.

The goal of this project is to explore how physical touch can be translated into digital sound and visual feedback in a clear and responsive way.

 

Inspiration

This project is inspired by different types of interactive musical interfaces:

* Digital MIDI controllers and touch-based instruments
* Piano keyboards, especially the idea of soft and responsive touch
* Interactive installations where physical actions directly control sound
* Simple expressive systems that focus on user interaction rather than complexity

I am particularly interested in how small physical gestures, like lightly touching a surface, can create meaningful digital outputs such as sound.

 

References / Tutorials

Here are some references and tutorials that informed my idea:

* p5.js Sound Library (official reference)

https://p5js.org/reference/#/libraries/p5.sound

* Basic FSR (Force Sensitive Resistor) guide
https://learn.adafruit.com/force-sensitive-resistor-fsr

* Multiplexer (CD74HC4067) explanation
https://learn.sparkfun.com/tutorials/multiplexer-breakout-hookup-guide

 

Sketch Description

The system will be arranged like a simple keyboard:

* FSR406 → white keys (main notes)
* FSR402 → black keys (sharps/flats)

For my final project, I am creating a physically interactive maze game that uses both Arduino and P5.js. So I decided to pick minnie mouse and mickey mouse as my two characters. The idea is to guide minnie mouse through a maze using two physical potentiometers instead of a keyboard or mouse. The player’s goal is to move minnie mouse from the start of the maze to the end, where mickey mouse is waiting.

The interaction works in a simple cycle of listening, thinking, and responding. The Arduino will be the “listener” because it reads the positions of the two potentiometers. One knob controls left andright movement, and the other controls up and down movement. When the player turns the knobs, Arduino senses the changes and sends those values to P5.js. P5. will be the “thinker,” because it takes those values and turns them into movement on the screen. The game then responds by moving the character in real time and showing whether the player is controlling minnie mouse in staying inside the maze or hitting the walls. I want to consider adding a celebration melody when minnie mouse reaches mickey mouse. I might as well make the LED blink when Minnie hits a wall to alert the player about it.

The maze itself will be drawn in P5 using simple shapes or a custom layout. I will have a start screen and instructions screen. The character will move smoothly through the pathways, and basic collision detection will prevent it from passing through walls. When the player reaches the goal area, the game will display a small celebration animation or visual cue to show that the objective has been completed. The experience is meant to feel playful, fun and slightly challenging, since the player must learn how to control movement using physical knobs instead of controlling it with the laptop. I will probably take photos of minnie mouse and mickey mouse of the internet and use them as my characters.

Reading Design Meets Disability made me rethink what I previously considered “good design.” I used to associate good design with aesthetics and general usability, but this reading challenged my assumption by showing how design is deeply tied to social attitudes toward disability. The section on “discretion” stood out to me, where assistive devices like hearing aids are designed to be hidden in order to avoid stigma. This made me question whether invisibility is actually a positive goal, or if it reinforces the idea that disability should be concealed. I found myself reflecting on how design choices are not neutral and that they reflect cultural values about what is considered “normal.” The tension between fashion and disability also left a strong impression on me. While making assistive devices more fashionable can empower users, it also risks emphasizing appearance over accessibility. This contradiction made me realize that design often operates between competing goals rather than clear solutions.

This reading also connected to my previous understanding of user-centered design but expanded it in a way I had not considered before. It made me realize that “the user” is often imagined as a narrow, able-bodied group, which doesn’t take into account many real experiences. The idea of “simple meets universal” raised an important question for me that is it actually possible to design something that works for everyone, or does inclusivity inevitably introduce complexity?   This challenges the assumption that simplicity is always better. Personally, this reading shifted my perspective from seeing design as purely functional problem-solving to understanding it as a more cultural and ethical practice that shapes how people are perceived and included. It also made me more aware of the limitations in the designs I encounter daily and how they really reflect broader biases rather than just technical decisions.

Github:

https://github.com/skyorachorn/Intro-to-IM/blob/5c35d686b16c8ad6965f1755d6dd0e0c786ab2c4/Ex03_Bi_Directional_Com.ino

 

Youtube:

https://youtu.be/9bqEiR_zGJM?si=Mm7S7kV4fjWl15sk

Circuit:

 

Hand Written Circuit:

In this exercise, we extended our work from the previous tasks by implementing bi-directional communication between Arduino and p5.js. The goal was to use an analog sensor to control the wind force in a physics simulation, and at the same time send data back from p5 to Arduino to control an LED.

We started from the gravity and wind example provided in class and kept the structure mostly unchanged. Instead of using keyboard input to control the wind direction, we replaced it with a joystick connected to Arduino. The horizontal axis of the joystick was read using analogRead() and sent to p5 through serial communication.

On the p5 side, the incoming value was read using port.readUntil(“\n”), then converted into an integer. We mapped this value to a range between -1 and 1, which was used as the horizontal wind force. As a result, moving the joystick left or right directly influenced the movement of the ball in the simulation.

To complete the bi-directional interaction, we added a simple condition in p5 to detect when the ball bounces on the ground. Whenever a bounce occurs, p5 sends a value back to Arduino. On the Arduino side, this value is read using Serial.parseInt(), and the LED is turned on or off accordingly.

This allowed us to create a system where Arduino sends sensor data to p5, and p5 responds by sending control signals back to Arduino.

⸻

What We Learned:

* How to implement bi-directional serial communication
* How to use an analog sensor (joystick) to control a physics simulation
* How to send control signals from p5 back to Arduino
* How physical and digital systems can interact in both directions

⸻

Code I’m Proud Of:

One part we are particularly satisfied with is how the system detects a meaningful bounce and avoids triggering unnecessary signals.

p5.js (bounce detection + sending data)

let bounced = 0;
  let floorY = height - mass / 2;

  // bottom bounce
  if (position.y > floorY) {
    position.y = floorY;

    if (velocity.y > 2) {
      velocity.y *= -0.9;
      bounced = 1;
    } else {
      velocity.y = 0;
    }
  }

  // send LED state back to Arduino
  if (port.opened()) {
    port.write(bounced + "\n");
  }

This section is important because it defines what counts as a “real” bounce. By using a velocity threshold, we ensure that only stronger impacts trigger the LED, while small movements at the ground are ignored.

⸻

Arduino (receiving and controlling LED)

int ledState = Serial.parseInt();

    if (Serial.read() == '\n') {
      digitalWrite(ledPin, ledState);
    }

This part connects the digital signal from p5 to a physical output. It shows how a simple value sent through serial communication can directly control hardware.

Challenges:

One issue we encountered was that the LED flickered rapidly when the ball reached the ground. Even though the ball appeared to be at rest, the system continued to detect small movements due to gravity and drag.

At first, this made the LED stay on or flicker continuously, which was not the intended behavior.

To solve this, we introduced a simple velocity threshold. Instead of triggering a bounce whenever the ball touched the ground, we only considered it a valid bounce when the downward velocity was greater than a certain value. After testing, we found that using a condition of velocity.y > 2 provided a stable and clear result.

This helped us better understand how physics simulations can produce small continuous movements, and why additional conditions are sometimes needed to define meaningful events.

⸻

Future Improvements:

If we were to continue developing this project, we would consider:

* Adding a dead zone to the joystick to reduce unwanted drift
* Using both axes of the joystick to control more complex motion
* Adding more LEDs or outputs to represent different events
* Improving the visual feedback to better match the physical response

 

Reference:

https://youtu.be/6LkqhQ1-vJc?si=E_jhX4XUAnMQBCrY

https://youtu.be/pD2JUNUWJGU?si=xgiGR7X3Bhdkyi6W

Github:

https://github.com/skyorachorn/Intro-to-IM/blob/0f3767a9cc4fe21946d8b84a76a020a09abacbbd/Ex02_P5_to_Arduino_Com.ino

 

Youtube:

https://youtu.be/BT8Y9AafPIk?si=77WJciHPNwpz8n2J

 

Picture of Circuit:

Hand Written Diagram:

In this exercise, we explored serial communication in the opposite direction, from p5.js to Arduino. The main objective was to control the brightness of an LED on Arduino using an interaction created in p5.

We again started from the class example and kept the overall structure close to what was introduced by professor Aya. However, for this exercise, we simplified the communication so that p5 sends only one value instead of multiple values. This made it easier to focus on the main idea of sending data from the browser to Arduino.

On the p5 side, we used the p5.webserial library and the same button-based connection method as in the previous exercise. Once the serial connection is opened, p5 continuously sends a brightness value based on the horizontal position of the mouse. This value is mapped into a range between 0 and 255, which matches the range used by analogWrite() on Arduino.

On the Arduino side, we used Serial.parseInt() to read the incoming integer sent from p5. After checking for the newline character \n, the value is applied to an LED connected to a PWM pin. This allows the LED to gradually brighten or dim instead of simply turning on and off.

This exercise helped us better understand how p5 can be used not only to receive data from Arduino, but also to actively control physical outputs on the hardware side. Compared to Exercise 01, this made the communication feel more interactive because the browser was no longer only displaying data, but also sending instructions back to Arduino.

Through this process, we gained a clearer understanding of how serial data can be used to control physical output in real time, and how important it is for both the code structure and the wiring to match the intended behavior.

⸻

What We Learned:

* How to send data from p5.js to Arduino through serial communication
* How to use Serial.parseInt() on the Arduino side
* How to control LED brightness with analogWrite()
* How browser-based interaction can directly affect hardware output

Code I’m Proud Of:

One part we are particularly satisfied with is how the p5 sketch and Arduino sketch work together to control LED brightness in real time. This section clearly shows how a visual interaction in the browser becomes a physical output on the Arduino side.

p5.js (sending brightness value)

let brightness = int(map(mouseX, 0, width, 0, 255));
  brightness = constrain(brightness, 0, 255);

  // show the current value on screen
  fill(brightness);
  rect(100, 150, 200, 100);

  fill(0);
  textSize(16);
  text("Brightness: " + brightness, 120, 130);

  if (port.opened()) {
    port.write(brightness + "\n");
  }

This part is meaningful because it translates a simple mouse movement into a numerical value that can be understood by Arduino. It also helped us see how p5 can act as the controlling side of the communication.

⸻

Arduino (receiving + controlling LED)

int brightness = Serial.parseInt();

    if (Serial.read() == '\n') {
      analogWrite(ledPin, brightness);
    }

We found this section especially important because it shows how Arduino receives a value from p5, processes it, and immediately applies it to a physical output. In this case, the output is the LED brightness controlled through PWM.

Problems We Encountered:

One issue we encountered was that the LED did not appear to brighten or dim smoothly at first. Even though the value on the p5 side was clearly changing, the LED response did not look correct. After checking both the code and the circuit, we realized that the LED needed to be connected to a PWM pin in order for analogWrite() to work as expected. Once we moved the LED to the correct pin, the brightness control became much more visible.

Another thing we paid attention to was keeping the communication format simple. Since this exercise only required brightness control, we kept the message to a single value followed by a newline character. This made the program easier to understand and reduced confusion while testing.

⸻

Future Improvements:

If we were to continue developing this exercise, we would consider:

* Replacing mouse control with a more interesting p5 interaction, such as dragging, key presses, or etc.
* Adding multiple LEDs and sending more than one value from p5
* Expanding the system into bi-directional communication so Arduino could also send sensor data back to p5
* Improving the visual design in p5 so that the screen feedback more closely matches the physical LED behavior

Github:

https://github.com/skyorachorn/Intro-to-IM/blob/38f5a920179feadd602c862571956198fbb9e0cf/Ex01_Arduino_to_P5_Com.ino

 

Youtube:

https://youtu.be/dvRseX6c4VU?si=mKZLcT-Lm9T1LGMQ

Handwritten Diagram:

Picture of Circuit:

In this exercise, we explored serial communication between Arduino and p5.js. The main objective was to use a single sensor on Arduino and translate its input into visual movement in p5, specifically controlling an ellipse moving horizontally across the screen.

We began by following the example demonstrated in class and gradually adapted it to better understand the data flow. On the Arduino side, we used a potentiometer as an analog input. The sensor value (0–1023) was mapped to a smaller range (0–255) before being sent through the serial port using Serial.println(). This ensured that the data could be easily interpreted on the p5 side.

For p5.js, we adopted the structure introduced by professor Aya using the p5.webserial library. The connection process is handled through a button (connectBtn), and the serial port is opened using port.open(baudrate). This approach helped us clearly understand how communication between Arduino and the browser is initiated manually rather than automatically.

Inside the draw() loop, we used port.readUntil(“\n”) to read incoming serial data line by line. The received string is then cleaned and converted into a number using trim() and int(). We then used map() to convert this value into a position across the canvas width. As a result, the ellipse moves smoothly from left to right, and when the input value decreases, it naturally moves back from right to left, creating a responsive and continuous motion.

Through this process, we gained a clearer understanding of how physical input can directly influence digital visuals in real time. It also highlighted the importance of consistent data formatting and timing in serial communication.

⸻

What We Learned
• How to send analog data from Arduino using Serial.println()
• How to receive and interpret serial data in p5.js
• How to map sensor values into visual output
• How hardware and software can interact in real time

⸻

Code I’m Proud Of

One part we are particularly satisfied with is how the sensor data is processed and translated into visual movement. This section demonstrates how raw data from Arduino becomes meaningful interaction in p5.

int potentiometer = analogRead(A1);

 // map the value from 0-1023 to 0-255
 int mappedPotValue = map(potentiometer, 0, 1023, 0, 255);

 // send value to p5 as a string with newline
 Serial.println(mappedPotValue);

This part is important because it simplifies the raw sensor data into a range that is easier to use on the p5 side.

⸻

let str = port.readUntil("\n");

if (str.length > 0) {
  let val = int(trim(str));   // convert string → number

  // map value to screen position
  x = map(val, 0, 255, 0, width);
}

We found this section especially meaningful because it clearly shows the full pipeline: sensor → serial → processing → visual output.

⸻

Future Improvements

If we were to continue developing this project, we would consider:
• Using different sensors such as FSR or ultrasonic sensors for more dynamic interaction
• Adding smoothing to reduce noise in sensor readings
• Expanding the visuals to control multiple elements instead of a single ellipse
• Exploring bi-directional communication between Arduino and p5

For my final project, I plan to create a physically interactive game where the player helps recover a corrupted digital signal of a story using their feet. The idea is inspired by games like Piano Tiles and Just Dance, but instead of simply playing for points, the user is actively uncovering a hidden message through their performance. By stepping on floor pads in time with visual cues, which I am thinking will be something similar to piano tiles and the Just Dance mat, the player will gradually restore a distorted audio/visual, which makes the experience into kind of like a mystery game:

 

I will use Arduino to capture physical input and P5 to handle the visual and audio output. On the floor, there will be 4 pads made from materials like cardboard or foam with conductive layers inside (or if there is a better sensor for this idea, I will use it). Each pad will act as a button connected to the Arduino, and when someone steps on a pad, the circuit is completed, and the Arduino sends that input data to p5 through the serial communication.

On the screen, p5 will display a rhythm like visuals with vertical lanes and falling tiles, like the piano tile game. The user must step on the pad at the correct time to match the falling tiles (which I am thinking will be color based, so it matches the floor tiles, and it is more obvious). For the visuals, it will initially appear distorted, with glitch effects, static, and broken text or images to represent a broken signal along with the tiles. As the player successfully hits notes on time, the system will respond immediately by reducing the distortion, sharpening visuals, and revealing fragments of audio or text of a story. I want it to gradually form a short narrative, such as a corrupted voicemail or like a partial conversation. But if the user misses notes, the distortion stays or I might back it to where it temporarily increases.

The game basically listens through the Arduino sensors, thinks by processing the timing and accuracy of the input in p5, and responds through changes in the visuals and sound. The player will then be paying attention and adjusting their movements based on the feedback they receive, especially when they start revealing more of the story.

I prompt Gemini AI to create a picture of the game, so you can get a vision of what I am trying to achieve:

 

Concept

For my final project I had an idea inspired by the musical/film The Phantom of the Opera minus the obviously really weird creepy part. The idea is that you are a composer at an opera, and you have to be able to compose at least a good amount of the music while jump scares happen in the game.

I will be using my Arduino to create and connect the composer’s batons to my code and will connect the batons’ position to certain notes or volume or pitches or tempo (still unsure about what I want to do with this since its very complex) so when you try to move the sticks the position will translate to musical notes.

I will use p5 to code the design and the UI of the game and translate motion of the batons to actual events in the game.

I’m still unsure if I want to keep the jump scare component or if I want to keep it so that you just try to compose the music,  but for now this is the main concept I have in mind.

Visuals

 

Reading Design Meets Disability made me think differently about how society treats disability devices. One part that stayed with me says that trying to hide these devices can show “a lack of self confidence that can communicate an implied shame.” This made sense to me because I believe disability devices should be shown proudly. There is nothing to hide, and people should feel free to embrace what makes them unique. The reading helped me understand that hiding something can sometimes send a negative message, even if people do not mean it that way.

Another idea that surprised me was how design for disability can inspire mainstream design. The text explains how the Eames leg splint influenced famous furniture. I never expected a medical object to shape everyday design. This showed me that disability centered design can be creative and important, not only functional.

The reading also changed how I think about the look of medical devices. When Aimee Mullins chooses “off the chart glamorous” prosthetics, it shows how fashion can help people feel confident. If I had to wear a hearing aid or prosthetic, I would want it to be unique and fashionable too. Something that stands out in a good way and can influence others to do the same. This is why I think fashion designers, especially designer brands, should help create these devices. It would make people feel included and inspire many other small businesses and brands globally.

Overall, the reading taught me that disability and design do not need to focus on hiding. They can focus on expression, creativity and how each person shows their own personality with something so unique.