Finalized Concept
For my final project, it will basically be my preliminary concept, where I will create a physically interactive rhythm based system called Recovered Signal, where the user helps reconstruct a corrupted digital story through timed physical interaction. The concept is inspired by rhythm games such as Piano Tiles, but instead of focusing only on scoring, the user will actively uncover fragments of a hidden/broken narrative.
The experience is designed as a mystery signal recovery system. On screen, the user sees falling colored rhythm tiles with music playing, while physically pressing with their hand the right buttons/pads in sync with them. For the visuals, it will initially appear distorted, with glitch effects and broken text to show a broken signal along with the tiles. As the player successfully hits notes on time, the system will respond immediately by reducing the distortion, sharpening the text, and revealing fragments of audio or text of a story. It will gradually form a short narrative, like a corrupted voicemail or like a partial conversation/story. But if the user misses notes, the distortion stays or I might make it so that it temporarily increases.
Arduino
So the Arduino will act as the physical input system for the project.
The inputs will be either an arcade style push buttons or pressure pads, and each pad represents a tile in the rhythm game. When the pad is pressed, it sends a high signal to the Arduino. Within the Arduino, it will continuously read the state of each button, detecting which button was pressed. The Arduino will not handle any game logic or timing evaluation, just detect the input and send that data to p5.
For output and communication, it will send a serial message to p5 in the format of a button id (which will show which pad was pressed), and a timestamp using millis to indicate when the press occurred. I will probably use the numbers 1,2,3,4 in correlation to each pad/button. For example 1 if the red button was pressed or 1, 10230, if it was pressed at a specific time.
UPDATE: the Arduino will send the button information, and then p5 checks if it was the right timing, and p5 sends back whether it was correct or wrong (0 or 1), and correct will turn on the green LED light, and wrong will turn on the red LED light. I am also thinking about making the LEDs flicker if they’re still on the weak signal stage, or have a steady glow when their half way there, and the brightest when it’s strong.
(If time allows, I will also set up an LED that lights up when a button is pressed for feedback/confirmation.)
P5
P5 will function as the main visual and audio system of the project.
It will generate a rhythm game with 4 vertical lanes that displays falling tiles synchronized to a beat. It will detect input from Arduino (which button was pressed) and it will match Arduino input with falling tile positions, evaluate timing accuracy (perfect/miss/late/early), and update the system state based on performance. Everything will be in real time as the button is pressed.
For the audio, it will be handled entirely in p5, so it’s clear. Each correct input will trigger musical elements, and the layered audio starts muffled or distorted, becomes clearer as player progresses, and eventually reveals a full short narrative (voicemail or conversation)
What it will communicate with the Arduino is that it will receive button press data through serial input and use that data to trigger game events and check accuracy, and send back to Arduino if the timing was right and the right button was pressed, in which it will light up a red or green LED light.
Current Progress
I have currently made all of the backgrounds and set up the skeleton code in p5, with the game states and made a signal level bar (as the player taps, a signal bar fills up on the screen. When the signal reaches a certain level, it triggers a story message to appear, like a transmission being decoded).
Here is the link to the sketch and some of the backgrounds:
https://editor.p5js.org/farahshaer/sketches/LJyoiS0O_
I will next make the tiles, stories (using oop), and sounds, glitch effects, and make my Arduino code and circuit…I still have a long way to go.
Stipend Breakdown
I plan to use the $50 stipend to build the pressure pads or buy arcade buttons (but I am not too sure if it will come in time).
Cardboard/foam board for mounting structure
Copper tape and sheets
Colored paper and materials for labeling the pads visually to match the p5 lanes/tiles.
Basic craft supplies (tape, glue, cutting materials)