Hand-Drawn Shapes Recognition System

Project Overview

Please find code on my github repository

The Hand-Drawn Shapes Recognition system is an innovative interactive desktop application that combines computer vision and machine learning to recognize shapes drawn by users in real-time. The project emerged from the recognition that while humans can easily identify simple hand-drawn shapes, creating a computer system to replicate this capability presents significant challenges due to variations in drawing styles, imprecisions, and the inherent ambiguity of hand-drawn content. The system addresses these challenges through a sophisticated hybrid approach that leverages both traditional computer vision techniques and modern machine learning methods.

At its core, the application provides users with an intuitive drawing canvas where they can create shapes using either a mouse/touchpad or a connected Arduino controller. Once a shape is drawn, the system processes the image using OpenCV for preliminary analysis and contour detection, then employs a dual-recognition strategy combining geometric feature analysis with an SVM classifier to identify the drawn shape with high accuracy. This hybrid approach enables the system to recognize various shapes even with limited training data, making it both powerful and adaptable to individual users’ drawing styles.

Beyond mere recognition, the system offers a conversational interface that provides dynamic feedback based on recognition confidence and established interaction patterns. The application continually improves its recognition capabilities through user feedback, saving labeled drawings to a training database and supporting incremental model training in the background, effectively “learning” from each interaction.

System Architecture

The system employs a modular architecture with clearly defined components that handle specific aspects of the application’s functionality. This approach enhances maintainability, supports extensibility, and simplifies the debugging process. The architecture consists of four main component categories: Core Components, UI Components, Data Management, and Hardware Integration.

The core components form the backbone of the application’s functionality. The Recognition System serves as the central element, implementing the hybrid approach to shape recognition. It orchestrates the entire recognition process from image preprocessing to final shape classification. This component contains several specialized classes including the DrawingRecognizer that coordinates the recognition workflow, the ShapeFeatureExtractor for deriving geometric and statistical features from contours, the ShapeClassifier for machine learning classification, and the GeometricAnalyzer for traditional computer vision approaches to shape identification.

Supporting the recognition system, the Drawing Manager bridges the UI and recognition system, managing drawing operations and history tracking. The Conversation Manager handles the AI assistant’s responses, providing dynamic, context-aware feedback based on recognition results and interaction history. The Text-to-Speech component adds an auditory dimension to the user experience, verbalizing the AI assistant’s responses through multiple TTS engine options.

The UI components provide the visual interface through which users interact with the system. The Main Window contains the primary application interface, housing the drawing canvas and AI response display. The Canvas component serves as the interactive drawing surface, handling mouse events and supporting features like undo/redo, zoom, and grid display. Complementing these elements, the Toolbar offers access to drawing tools such as color selection and pen size adjustments, while various dialog screens provide access to settings, training data management, and shape labeling.

Data management components ensure persistence and organized data handling. The Database Interface manages data storage using SQLite, maintaining records of user settings, labeled shapes, and drawing history. User Settings handles application preferences, while Drawing History tracks past drawings and recognition results, allowing users to review their progression over time.

Recognition Technology

The recognition technology represents the system’s most sophisticated aspect, implementing a dual-approach strategy that combines traditional computer vision techniques with machine learning. This hybrid methodology provides robust baseline performance through geometric analysis while continuously improving through machine learning from user interactions.

The recognition process begins with image preprocessing, where the drawn shape is converted to grayscale, Gaussian blur is applied to reduce noise, and adaptive thresholding creates a binary image. The system then performs contour detection using OpenCV to identify shapes within the image, extracting the largest contour as the primary shape of interest. This approach effectively isolates the intended shape even when the drawing contains imperfections or stray marks.

Feature extraction forms the next critical step in the process. The ShapeFeatureExtractor class derives a comprehensive set of geometric and statistical features from the identified contour. These features include basic metrics such as area, perimeter, and bounding box dimensions; shape properties including circularity, convexity, and solidity; moment-based features like Hu Moments that provide rotation, scale, and translation invariance; multiple levels of contour approximation; corner analysis examining count, angles, and distributions; symmetry analysis measuring vertical and horizontal symmetry; and enclosing shape analysis testing fit against geometric primitives.

With features extracted, the GeometricAnalyzer applies traditional computer vision approaches to classify the shape. This component implements specialized detectors for common shapes like rectangles, triangles, ellipses, and hexagons. Each detector analyzes the extracted features against known geometric patterns, calculating confidence scores that reflect how closely the drawing matches each potential shape type. This rule-based approach provides strong baseline recognition even before machine learning is applied.

The machine learning component, implemented in the ShapeClassifier class, adds another dimension to the recognition process. Using scikit-learn’s LinearSVC as the primary classifier, the system categorizes shapes based on their extracted features. The classification pipeline includes feature standardization to normalize values to zero mean and unit variance, feature selection using ANOVA F-value to focus on the most discriminative attributes, and finally, classification with LinearSVC including class balancing to handle imbalanced training data. This approach yields high accuracy even with limited training examples.

The final recognition decision combines results from both approaches. Geometric analysis provides baseline recognition scores, while machine learning classification results receive higher weighting when available. Confidence scores are normalized and ranked, with the system returning the top guesses along with their confidence levels. This dual-approach strategy leverages the strengths of both paradigms, producing recognition that is both accurate and continuously improving.

 

Here’s an example of debug images

User Interface

The user interface prioritizes intuitive interaction while providing access to the system’s advanced capabilities. Built with PyQt5, the interface combines simplicity with functionality to accommodate both novice and experienced users. The UI consists of several key elements designed to work together harmoniously while maintaining a clear separation of functions.

The drawing canvas serves as the primary interaction point, providing a responsive surface where users can create shapes. The canvas supports freehand drawing with customizable pen properties including color and thickness. Drawing operations benefit from features like undo/redo capability (supporting up to 50 steps), zoom and pan functionality for detailed work, optional grid display for alignment assistance, and pressure sensitivity support for hardware that offers this capability. An auto-save function ensures work is preserved even in the event of unexpected issues.

Complementing the canvas, the toolbar provides access to essential drawing tools and functions. Users can select pen color from a palette or using a color picker, adjust stroke thickness through a slider control, toggle between pen and eraser modes, clear the canvas with a single click, and access undo/redo functions for correcting mistakes. The toolbar’s layout prioritizes frequently used functions while maintaining a clean, uncluttered appearance that doesn’t distract from the drawing process.

The information panel displays the AI assistant’s responses and recognition results. After recognition, this area shows the top shape guesses along with their confidence percentages, presented in a clear, easy-to-understand format. The assistant’s conversational responses provide context-aware feedback, varying based on recognition confidence and previous interactions to avoid repetitive messaging. This panel also offers buttons for confirming or correcting the AI’s guesses, facilitating the training feedback loop that improves recognition over time.

Dialog screens provide access to less frequently used functions without cluttering the main interface. The Settings Dialog allows users to customize application behavior through categories including general settings, drawing tool properties, recognition parameters, and text-to-speech options. The Training Dialog displays statistics on the training data, showing labeled shapes in the database and allowing management of training examples. The Label Dialog facilitates correction of misrecognized shapes, capturing user feedback that enhances model performance.

The user experience flow has been carefully designed to feel natural and responsive. When drawing a shape, users experience immediate visual feedback as their strokes appear on the canvas. Upon requesting recognition (either through the UI button or Arduino controller), the system processes the drawing and promptly displays results in the information panel. The conversational AI response provides context to the recognition, often suggesting improvements or offering praise based on recognition confidence. If the system misidentifies a shape, users can easily correct it, with the application acknowledging this feedback and incorporating it into future recognition attempts.

Hardware Integration

The system extends beyond traditional mouse and keyboard input through its Arduino integration, offering a novel physical interaction method that enhances the drawing experience. This hardware component connects through a serial interface and enables users to draw shapes using physical controls rather than conventional computer input devices.

The Arduino controller serves as an alternative input method, allowing users to draw using a joystick and trigger various actions with physical buttons. Five buttons are mapped to specific functions: triggering shape recognition, clearing the canvas, changing drawing color, adjusting stroke thickness, and toggling drawing mode. These correspond to pins 2 through 6 on the Arduino board. Drawing control is achieved through a potentiometer connected to pins A0 and A1, offering analog control of cursor position similar to a joystick. This physical interface provides a more tactile drawing experience that some users may find more intuitive than mouse-based drawing.

The system implements robust connection management for the Arduino controller. At application startup, the program automatically scans available serial ports to detect connected Arduino devices. Once detected, a dedicated thread continuously reads input data, translating it into drawing actions within the application. The connection management system includes auto-reconnect capability, allowing the application to recover from temporary disconnections without requiring user intervention. This ensures reliable hardware communication even in environments where connections might be intermittent.

Data processing for Arduino input employs a buffering system to handle the potentially variable data rate from the serial connection. Incoming data follows a structured format that indicates the input type (button press or analog input) and its value, with the application parsing this information and triggering appropriate actions in response. Analog inputs are normalized and mapped to canvas coordinates, ensuring smooth and predictable cursor movement despite potential variations in potentiometer readings.

Error handling for hardware integration is particularly robust, accounting for common issues like connection loss, malformed data, or hardware failures. The system implements graceful degradation when hardware components are unavailable, automatically falling back to mouse/keyboard input if the Arduino connection cannot be established or is lost during operation. Users receive clear notifications about hardware status through the application’s status bar, ensuring they remain informed about available input methods.

Development Journey

The development of the Hand-Drawn Shapes Recognition system followed an iterative process with distinct phases that progressively enhanced functionality and performance. Each phase built upon previous achievements while addressing limitations and incorporating user feedback, though not without significant challenges along the way.

The project began with the foundation phase, establishing the basic architecture and developing core components. During this period, I implemented the PyQt5-based user interface with a functional drawing canvas and basic shape extraction using OpenCV. Initial recognition relied solely on geometric analysis using traditional computer vision techniques, providing reasonable accuracy for well-drawn shapes but struggling with imprecise or ambiguous drawings. This phase established the project’s technical foundation while highlighting the need for more sophisticated recognition approaches.

As development progressed, I introduced machine learning to enhance recognition capabilities. Initial experiments with various classifiers led to my selection of Support Vector Machines as the primary classification algorithm due to their effectiveness with limited training data. The first training dataset consisted of manually labeled examples across eight shape categories: cross, square, other, triangle, ellipse, rectangle, hexagon, and line. This initial training process demonstrated the potential of machine learning while revealing challenges in data collection and feature selection.

A significant milestone occurred when the training dataset expanded to over 10,000 samples. Console output from this period reveals the distribution across shape categories: ellipse with 2,970 samples, rectangle with 2,680 samples, triangle with 2,615 samples, and “other” with 1,950 samples represented the majority classes, while cross (36 samples), square (17 samples), hexagon (16 samples), and line (20 samples) constituted minority classes. Training the model with this imbalanced dataset required careful consideration of class weights to prevent bias toward majority classes. The training process extracted 36 numeric features from each sample, excluding non-numeric data that might compromise model performance.

The training process during this phase required several minutes of computation, highlighting performance considerations that would later be addressed through optimization. Despite these challenges, the model achieved impressive accuracy metrics with 0.99 training accuracy and 0.97 test accuracy. These results validated the machine learning approach while providing a baseline for future improvements. Upon completion, the model was saved to disk as “shape_classifier.pkl” for subsequent use by the application.

One of the most devastating challenges I faced during development was losing approximately 60% of the codebase due to a catastrophic data loss incident. Most critically, this included a highly refined model that I had trained on over 10 million shapes from Google’s Quick Draw dataset. This advanced model represented tens of hours of training time across multiple GPU instances and had achieved significantly higher accuracy rates than previous iterations, particularly for complex and ambiguous shapes. Rebuilding after this loss required considerable effort, recreating critical code components from memory and documentation while working to reconstruct a training dataset that could approach the quality of the lost model.

Hardware integration represented another significant development phase. The Arduino controller implementation expanded the application’s input options while introducing new technical challenges related to serial communication and input mapping. I worked through issues of connection reliability, data parsing, and input calibration to create a seamless experience across both traditional and hardware-based input methods. This integration demonstrated the system’s flexibility while providing a novel interaction method that some users preferred over mouse-based drawing.

Throughout the development journey, user feedback played a crucial role in refining the system. Early testing revealed usability issues in the drawing interface that I addressed through UI refinements. Recognition errors highlighted gaps in the training data, leading to targeted data collection for underrepresented shape categories. Performance concerns during recognition and training prompted the optimization efforts that would become a major focus in later development stages.

Performance Optimization

As the system evolved and the training dataset grew, performance optimization became increasingly important to maintain responsiveness and enable real-time recognition. I implemented several key optimizations that significantly improved both training speed and runtime performance, particularly critical after losing my previous highly-optimized model.

A fundamental enhancement involved replacing the original SVC (Support Vector Classifier) implementation with LinearSVC, dramatically reducing computational complexity from O(n³) to O(n). This change resulted in training times that scaled linearly rather than cubically with dataset size, making it practical to train with larger datasets and more features. For a dataset with over 10,000 samples, this optimization reduced training time from hours to minutes, enabling more frequent model updates and facilitating experimentation with different feature sets and hyperparameters.

The feature extraction process, initially a bottleneck during both training and recognition, benefited from several optimizations. I implemented parallel feature extraction using Python’s multiprocessing capabilities, distributing the CPU-intensive work of calculating geometric features across multiple processor cores. This approach achieved near-linear speedup on multi-core systems, significantly reducing processing time for large batches of training images. Additionally, vectorizing operations with NumPy replaced inefficient Python loops with optimized array operations, further accelerating the feature calculation process. These optimizations were essential not just for performance, but for helping me recover from the lost advanced model by making retraining more efficient.

Data management optimizations addressed I/O-related performance issues. The system implemented batch loading and preprocessing of training data, reducing disk access frequency and allowing more efficient memory utilization. Feature caching stored pre-computed features for training examples, eliminating redundant calculations when retraining the model or performing incremental updates. Database operations were optimized with appropriate indexing and query strategies, ensuring efficient retrieval of training examples and user settings even as the database grew in size.

The recognition pipeline itself underwent substantial optimization to support real-time feedback. The system implemented adaptive algorithm selection, applying simpler, faster recognition methods for clear, well-formed shapes while reserving more computationally intensive analysis for ambiguous cases. Feature selection using techniques like Principal Component Analysis (PCA) and SelectKBest reduced the dimensionality of the feature space without significantly impacting accuracy, accelerating both training and inference. Memory management techniques minimized allocations during recognition, reducing garbage collection overhead and preventing memory-related performance degradation.

Command-line options added during this phase provided further optimization capabilities. The --retry flag enabled an automatic retry mechanism for failed samples, improving training robustness. Users could configure the maximum number of retry attempts with --max-retry-attempts (defaulting to 3) and specify the minimum required samples per shape class with --min-samples (defaulting to 10). For situations where machine learning was unnecessary or unavailable, the --geometric-only option limited recognition to geometric template rendering, reducing computational requirements. The --output option allowed specifying a custom output path for the trained model, facilitating experimentation with different model configurations.

These optimization efforts transformed the application from a proof-of-concept demonstration to a practical tool suitable for regular use. Recognition response times decreased from seconds to sub-second levels, providing the immediate feedback essential for a satisfying user experience. Training times reduced dramatically, enabling more frequent model updates and supporting the incremental learning approach that helped the system adapt to individual users’ drawing styles.

Future Enhancements

The Hand-Drawn Shapes Recognition system establishes a solid foundation that can be extended in numerous directions to enhance functionality, improve performance, and expand applicability. While the current implementation successfully addresses the core recognition challenge, several potential enhancements have been identified for future development iterations.

Advanced machine learning represents a promising direction for further development. Integrating deep learning approaches, particularly convolutional neural networks (CNNs), could improve recognition accuracy for complex shapes without requiring explicit feature engineering. Transfer learning from pre-trained models would enable leveraging existing visual recognition capabilities while reducing the required training data volume. Implementing ensemble methods combining multiple classifiers could enhance recognition robustness, especially for ambiguous cases where different approaches might yield complementary insights.

User experience enhancements could make the application more intuitive and powerful. Implementing multi-shape recognition would allow the system to identify multiple distinct shapes within a single drawing, expanding its applicability to more complex diagrams. A shape suggestion system could provide real-time guidance as users draw, helping them create more recognizable shapes. Enhanced drawing tools including shape creation templates, text annotation, and layer support would transform the application from a recognition demonstrator to a complete drawing tool with intelligent recognition capabilities.

Platform expansion represents another potential development path. Creating web and mobile versions of the application would increase accessibility, allowing users to benefit from shape recognition across different devices. Cloud-based training and recognition would enable sharing improvements across the user base, with each user’s corrections potentially improving the system for everyone. API development would allow third-party integration, enabling other applications to leverage the recognition capabilities for their own purposes.

Educational applications offer particularly promising opportunities. Developing specialized modes for teaching geometry could help students learn shape properties through interactive drawing and recognition. Creating games based on shape recognition would make learning engaging while simultaneously gathering valuable training data. Implementing custom shape sets would allow teachers to create domain-specific recognition tasks targeting particular educational objectives.

Accessibility improvements could make the system more inclusive. Enhanced text-to-speech integration would better serve users with visual impairments, providing more detailed auditory feedback about recognition results and drawing state. Implementing alternative input methods beyond the current mouse/touchpad and Arduino options could accommodate users with different abilities and preferences. Creating profiles for different user needs would allow the interface and recognition parameters to adapt automatically based on individual requirements.

The continuous improvement framework established in the current implementation provides a solid foundation for these enhancements. The modular architecture facilitates adding new components without disrupting existing functionality, while the dual-approach recognition strategy can incorporate new techniques alongside proven methods. As the system evolves, it will continue building on its core strengths while expanding to address new challenges and opportunities in shape recognition and interactive drawing.

Conclusion

The Hand-Drawn Shapes Recognition system represents my creation of a sophisticated blend of computer vision, machine learning, and interactive design, resulting in an application that not only recognizes hand-drawn shapes but continuously improves through user interaction. By implementing a hybrid approach combining geometric analysis with machine learning, my system achieves high recognition accuracy even with limited initial training data, while establishing a framework for ongoing enhancement through user feedback.

My development journey illustrates the iterative process of building intelligent interactive systems, progressing from basic geometric analysis to sophisticated machine learning while continuously refining the user experience. This journey included overcoming significant setbacks, most notably losing 60% of the codebase and an advanced model trained on over 10 million Quick Draw shapes that represented tens of hours of training time. Despite these challenges, I persevered, rebuilding critical components and implementing performance optimizations that transformed promising algorithms into a responsive application suitable for regular use, demonstrating how theoretical approaches can be successfully adapted to practical applications through thoughtful engineering, resilience, and attention to user needs.

The system’s modular architecture and dual-approach recognition strategy provide a flexible foundation for future development, supporting enhancements from advanced machine learning techniques to expanded platform support and specialized applications. This extensibility ensures the project can evolve to address new requirements and incorporate emerging technologies while maintaining its core functionality and user-friendly design, and provides resilience against potential future setbacks similar to my experience with data loss.

Throughout development, the balance between sophistication and accessibility remained a central consideration. While implementing advanced recognition techniques, I maintained focus on creating an intuitive interface that hides complexity from users while providing transparent feedback about recognition results. This approach makes the technology accessible to users regardless of their technical background, fulfilling my project’s goal of bridging the gap between human perceptual abilities and computer vision capabilities.

The Hand-Drawn Shapes Recognition system stands as both a practical application and a technological demonstration of my work, showing how computer vision and machine learning can enhance human-computer interaction in creative contexts. The project also represents my perseverance through significant technical challenges and data loss, emerging stronger with more efficient algorithms and robust error handling. As the system continues to evolve under my development, it will further narrow the gap between how humans and computers perceive and interpret visual information, creating increasingly natural and intuitive interaction experiences while maintaining resilience against the inevitable challenges of complex software development.

Concept

My project is a DIY Disco that combines Arduino and p5.js to create an interactive audio-visual experience. The Arduino handles the physical buttons, DC motor and controls a NeoPixel LED strip, while p5.js is responsible for generating a visualizer that reacts to the music.

How It Works

The project consists of two main components:

Arduino Control and LED Display

• Three physical buttons are connected to the Arduino:
  • • Button 3: Activates the airhorn sound effect
    • Button 2: Increases the speed of the music
    • Button 1: Shuffles to a different song

• The NeoPixel strip lights up with different patterns that shuffle when Shuffle button is pressed

p5.js Visualizer and Music Control

• p5.js generates a circular music visualizer that responds to the song’s frequencies. It receives signals from the Arduino through serial communication to trigger changes in the visualizer based on button presses.

Interaction Design

The interaction design is simple and engaging: pressing physical buttons on the Arduino changes the music’s speed, shuffles the track, or plays a sound effect while the NeoPixel lights and p5.js visualizer respond instantly.

Arduino Code

My project features three physical buttons connected to the Arduino. Button 1 triggers a signal (1) that is sent to p5.js and triggers the airhorn effect,  Button 2 sends a different signal (2) to p5.js, which affects the speed of the song playing. Button 3  sends a signal (3) to shuffle to the next song as well as cycles through five distinct LED animation patterns. Alongside this, the Arduino also manages the motor that spins the vinyl. Through serial communication with p5.js, the motor’s state is toggled based on signals received. A ‘1’ signal turns the motor on, while a ‘0’ stops it and clears the LED display. Below is the code:

#include <Adafruit_NeoPixel.h>

const int button1Pin = 2;
const int button2Pin = 3;
const int button3Pin = 4;
const int bin1Pin = 8;
const int bin2Pin = 7;
const int pwmBPin = 9;
bool motorRunning = true;

#define NEOPIXEL_PIN 6
#define LED_COUNT    60
#define Num_LED      56

Adafruit_NeoPixel strip(LED_COUNT, NEOPIXEL_PIN, NEO_GRBW + NEO_KHZ800);

unsigned long lastButtonPress1 = 0;
unsigned long lastButtonPress2 = 0;
unsigned long lastButtonPress3 = 0;
unsigned long lastActionTime1 = 0;
unsigned long lastActionTime2 = 0;
unsigned long lastActionTime3 = 0;
const unsigned long debounceDelay = 100;
const unsigned long cooldown = 1000;

int currentPattern = 0;
const int totalPatterns = 5;

unsigned long lastPatternUpdate = 0;
const unsigned long patternInterval = 80;

int snakeIndex = 0;
float hueOffset = 0;

void setup() {
  Serial.begin(9600);
  pinMode(button1Pin, INPUT_PULLUP);
  pinMode(button2Pin, INPUT_PULLUP);
  pinMode(button3Pin, INPUT_PULLUP);
  pinMode(bin1Pin, OUTPUT);
  pinMode(bin2Pin, OUTPUT);
  pinMode(pwmBPin, OUTPUT);
  strip.begin();
  strip.show();
  strip.setBrightness(180);
}

void loop() {
  unsigned long currentMillis = millis();

  handleButtons(currentMillis);
  handleSerial();
  controlMotor();

  if (currentMillis - lastPatternUpdate >= patternInterval) {
    lastPatternUpdate = currentMillis;
    runPattern(currentPattern);
  }
}

void handleButtons(unsigned long currentMillis) {
  if (digitalRead(button1Pin) == LOW && currentMillis - lastButtonPress1 >= debounceDelay && currentMillis - lastActionTime1 >= cooldown) {
    Serial.println("1");
    lastButtonPress1 = currentMillis;
    lastActionTime1 = currentMillis;
  }

  if (digitalRead(button2Pin) == LOW && currentMillis - lastButtonPress2 >= debounceDelay && currentMillis - lastActionTime2 >= cooldown) {
    Serial.println("2");
    lastButtonPress2 = currentMillis;
    lastActionTime2 = currentMillis;
  }

  if (digitalRead(button3Pin) == LOW && currentMillis - lastButtonPress3 >= debounceDelay && currentMillis - lastActionTime3 >= cooldown) {
    Serial.println("3");
    currentPattern = (currentPattern + 1) % totalPatterns;
    lastButtonPress3 = currentMillis;
    lastActionTime3 = currentMillis;
  }
}

void handleSerial() {
  if (Serial.available() > 0) {
    char incomingByte = Serial.read();
    if (incomingByte == '1') {
      motorRunning = true;
    } else if (incomingByte == '0') {
      motorRunning = false;
      digitalWrite(bin1Pin, LOW);
      digitalWrite(bin2Pin, LOW);
      analogWrite(pwmBPin, 0);
      strip.clear();
      strip.show();
    }
  }
}

void controlMotor() {
  if (motorRunning) {
    digitalWrite(bin1Pin, HIGH);
    digitalWrite(bin2Pin, LOW);
    analogWrite(pwmBPin, 50);
  } else {
    digitalWrite(bin1Pin, LOW);
    digitalWrite(bin2Pin, LOW);
    analogWrite(pwmBPin, 0);
  }
}

// === Pattern Dispatcher ===
void runPattern(int pattern) {
  switch (pattern) {
    case 0: discoFlash(); break;
    case 1: snakeCrawl(); break;
    case 2: colorWave(); break;
    case 3: sparkleStars(); break;
    case 4: fireGlow(); break;
  }
}

// Pattern 0: Disco Flash
void discoFlash() {
  for (int i = 0; i < Num_LED; i++) {
    strip.setPixelColor(i, randomColor());
  }
  strip.show();
}

// Pattern 1: Snake Crawl
void snakeCrawl() {
  strip.clear();
  int snakeLength = 6;
  for (int i = 0; i < snakeLength; i++) {
    int index = (snakeIndex + i) % Num_LED;
    strip.setPixelColor(index, Wheel((index * 5 + hueOffset)));
  }
  snakeIndex = (snakeIndex + 1) % Num_LED;
  hueOffset += 1;
  strip.show();
}

// Pattern 2: Smooth Rainbow Wave
void colorWave() {
  for (int i = 0; i < Num_LED; i++) {
    int hue = (i * 256 / Num_LED + (int)hueOffset) % 256;
    strip.setPixelColor(i, Wheel(hue));
  }
  hueOffset += 1;
  strip.show();
}

// Pattern 3: Sparkle Stars
void sparkleStars() {
  for (int i = 0; i < Num_LED; i++) {
    strip.setPixelColor(i, (random(10) < 2) ? strip.Color(255, 255, 255) : strip.Color(0, 0, 10));
  }
  strip.show();
}

// Pattern 4: Fire Glow
void fireGlow() {
  for (int i = 0; i < Num_LED; i++) {
    int r = random(180, 255);
    int g = random(0, 100);
    int b = 0;
    strip.setPixelColor(i, strip.Color(r, g, b));
  }
  strip.show();
}

// Helpers
uint32_t randomColor() {
  return strip.Color(random(256), random(256), random(256));
}

uint32_t Wheel(byte WheelPos) {
  WheelPos = 255 - WheelPos;
  if (WheelPos < 85) {
    return strip.Color(255 - WheelPos * 3, 0, WheelPos * 3);
  }
  if (WheelPos < 170) {
    WheelPos -= 85;
    return strip.Color(0, WheelPos * 3, 255 - WheelPos * 3);
  }
  WheelPos -= 170;
  return strip.Color(WheelPos * 3, 255 - WheelPos * 3, 0);
}

P5 code

The code uses createSerial() to open a serial port, allowing it to send and receive data between the p5 sketch and the arduino. The sensor values received from the Arduino (via serialPort.readUntil(“\n”)) trigger different actions within the p5 sketch based on specific sensor inputs. For example, a sensor value of 1 plays an airhorn sound, 2 toggles the playback speed of the song, and 3 shuffles to a new random song. The sensor values are continuously checked in the checkButtonPress() function, which responds accordingly by performing actions like playing sounds or changing song attributes.

The logic behind the visualizer relies on the FFT (Fast Fourier Transform) analysis of the audio. The fft.analyze() function breaks the audio into different frequency bands, so it will give a spectrum that represents the amplitude of different frequencies in the sound. The visualizer then maps these frequency intensities, for my project I decided to do so in a circular arrangement around the center of the screen, where each bar’s height is determined by the amplitude of its corresponding frequency band. The visualizer is updated in real time, so if the music is changed, or the speed is changed it will reflect those changes.

// === GLOBAL VARIABLES === //
let state = "landing";
let song;
let fft;
let selectedSong = "";
let sensorValue = 0;
let serialPort;
let serialSpeed = 9600;

let partySongs = ["nowahala.mp3", "umbrella.mp3", "yeah.mp3", "onlygirl.mp3", "hips.mp3","feeling.mp3","romance.mp3","monalisa.mp3","move.mp3","saywhat.mp3","yamore.mp3","adore.mp3","gorah.mp3"];
let airhornSound;

let startButton, continueButton, restartButton;
let isFastSpeed = false;

let bgImg;
let Instructions;
let vis;

// === PRELOAD === //
function preload() {
  bgImg = loadImage('bg.png');
  Instructions = loadImage('instructions.png');
  vis =loadImage('vis.png')
  airhornSound = loadSound("airhorn.mp3");
}

// === SETUP === //
function setup() {
  createCanvas(1460, 760);
  textAlign(CENTER, CENTER);
  angleMode(DEGREES);
  colorMode(HSB);

  // Start Button
  startButton = createButton("Start");
  styleButton(startButton, width / 2 - 45, height / 2 + 200);
  startButton.mousePressed(() => {
    state = "instructions";
    hideAllButtons();
    continueButton.show();
  });

  // Continue Button
  continueButton = createButton("Continue");
  styleButton(continueButton, width / 2 - 60, height / 2 + 200);
  continueButton.mousePressed(() => {
    selectedSong = random(partySongs);
    state = "visualizer";
    hideAllButtons();
  });
  continueButton.hide();

  // Restart Button
  restartButton = createButton("Restart");
  styleButton(restartButton, 20, 20, true);
  restartButton.mousePressed(() => {
    if (song && song.isPlaying()) song.stop();
    song = undefined;
    state = "landing";
    isFastSpeed = false;
    hideAllButtons();
    startButton.show();
    if (serialPort.opened()) serialPort.write("0");
  });
  restartButton.hide();

  // Serial setup
  serialPort = createSerial();
  let previous = usedSerialPorts();
  if (previous.length > 0) {
    serialPort.open(previous[0], serialSpeed);
  }
}

// === STYLE BUTTONS === //
function styleButton(btn, x, y, small = false) {
  btn.position(x, y);
  btn.style("padding", small ? "8px 16px" : "12px 24px");
  btn.style("font-size", small ? "16px" : "18px");
  btn.style("background-color", "#ffc700");
  btn.style("border", "none");
  btn.style("border-radius", "12px");
  btn.mouseOver(() => btn.style("background-color", "#FFFF"));
  btn.mouseOut(() => btn.style("background-color", "#ffc700"));
  btn.hide();
}

function hideAllButtons() {
  startButton.hide();
  continueButton.hide();
  restartButton.hide();
}

// === DRAW === //
function draw() {
  let data = serialPort.readUntil("\n");
  if (data.length > 0) {
    sensorValue = int(data);
    checkButtonPress(sensorValue);
  }

  if (state === "landing") {
    showLanding();
    startButton.show();
    if (serialPort.opened()) serialPort.write("0");
  } else if (state === "instructions") {
    showInstructions();
    continueButton.show();
  } else if (state === "visualizer") {
    restartButton.show();
    if (song === undefined) {
      loadSong(selectedSong);
    }
    runVisualizer();
    if (serialPort.opened()) serialPort.write("1");
  }
}

// === LANDING SCREEN === //
function showLanding() {
  image(bgImg, 0, 0, width, height);
}

// === INSTRUCTIONS SCREEN === //
function showInstructions() {
  image(Instructions, 0, 0, width, height);
}

// === LOAD SONG === //
function loadSong(songName) {
  song = loadSound(songName, startSong);
  fft = new p5.FFT();
}

function startSong() {
  song.rate(isFastSpeed ? 1.5 : 1.0);
  song.loop();
}

// === VISUALIZER === //
function runVisualizer() {
  let spectrum = fft.analyze();
  let lowerLimit = 0;
  let upperLimit = Math.floor(spectrum.length / 2);
  let numBars = upperLimit - lowerLimit;
  let radius = 70;
  let angleStep = 360 / numBars;
  let maxBarHeight = height / 1.8;


  image(vis, 0, 0, width, height);

  push();
  translate(width / 2, height / 2);

  for (let j = 0; j < 4; j++) {
    push();
    rotate(j * 90);
    for (let i = lowerLimit; i < upperLimit; i++) {
      let angle = (i - lowerLimit) * angleStep;
      let barHeight = map(spectrum[i], 0, 500, 15, maxBarHeight);
      let xEnd = cos(angle) * (radius + barHeight);
      let yEnd = sin(angle) * (radius + barHeight);
      stroke('#ffc700');
      line(0, 0, xEnd, yEnd);
    }
    pop();
  }

  pop();
}

// === CHECK SERIAL INPUT === //
function checkButtonPress(sensorValue) {
  if (state === "visualizer") {
    if (sensorValue === 1) {
      playAirhorn();
    } else if (sensorValue === 2) {
      toggleSpeed();
    } else if (sensorValue === 3) {
      shuffleNextSong();
    }
  }
}

// === SHUFFLE SONG === //
function shuffleNextSong() {
  let nextSong = random(partySongs);
  if (song && song.isPlaying()) song.stop();
  selectedSong = nextSong;
  isFastSpeed = false;
  loadSong(selectedSong);
}

// === TOGGLE SPEED === //
function toggleSpeed() {
  if (song) {
    isFastSpeed = !isFastSpeed;
    song.rate(isFastSpeed ? 1.5 : 1.0);
  }
}

// === PLAY AIRHORN === //
function playAirhorn() {
  if (airhornSound.isLoaded()) {
    airhornSound.play();
  }
}

 

Reflection/Difficulties

The project was not without its difficulties. One of my primary challenges involved oversensitive buttons. The issue arose because the buttons were triggering multiple actions from a single press, causing  these unintended rapid cycling of effects. To address this, I implemented a debounce mechanism and a cooldown timer, which made sure that each button press only activated an action once and prevented continuous cycling. This solution helped smooth out the interaction, making the experience more intuitive for the user.

Navigating the FFT (Fast Fourier Transform) in this project was also a challenge, as it involves converting an audio signal into its frequency components, which then drive the visual effects. The concept of analyzing an audio signal in terms of its frequency spectrum was at first a bit tricky to grasp. The FFT function takes the sound data, decomposes it into various frequency bands, and produces an array of amplitude values that represent the strength of each frequency. The biggest hurdle was understanding how to properly interpret and map these frequency values to create my visuals. For instance, the fft.analyze() function returns an array of amplitudes across a range of frequencies, but to effectively use this data, I needed to determine which frequency bands would be most useful for creating the visualizer’s elements. After some experimentation, I decided to focus on the lower and mid-range frequencies, which seemed to correspond best with the types of beats and musical elements I wanted to visualize.

Another significant issue was the motor’s weakness, which required a jump start for it to function properly. This created confusion for users, as they struggled to understand why the motor wasn’t working correctly.

Overall, I am very happy with how my project turned out, as it encompassed most if not all the things I wanted it to do. If I were to improve it however, I would maybe make the neopixels visualize the music as well, get the data from the FFT and send it to arduino to visaulize the changes in music. Maybe also add more sound effects, and additional interactive physical elements.

Concept

A physically interactive “Piano Tiles” game where players tap hand-buttons or pedal-buttons in time to falling/color coordinated tiles. Red tiles correspond to hand buttons (pins 2–5), blue tiles to foot pedals (pins 6–9). Finish the song without losing all three lives to win!

Interaction Demo

user testing:

copy_71E82669-1896-402A-A155-F76C8BE9E1AD

Me testing it:

copy_0FB66FA1-2527-4DBE-B6C6-86ACE6076571 2

Implementation Overview

Startup

-Menu screen with background image and “Start”/“Info” buttons

-Info screen explains with instructions on how to play

Song and Difficulty Selection

-Custom “song.png” and “difficulty.png” backgrounds

-3 levels (easy, medium, hard), as difficulty increased, speed of tiles moving down increases

Gameplay

-Tiles fall at a speed set by difficulty

-Serial data from Arduino (1–4 = hand zones, 5–8 = foot zones) drives hit detection

-Score and lives update live; MISS flashes when button misscliks/spam

Game Over

-Game over screen, shows score of the user: whether lost or won

-“Try Again” or “Home” appear and auto-returns to home screen after 5s

Interaction Design

Color coding: red/blue zones on screen match button colors

Audio cues: Crazy Frog sample plays in background

Lives and Score: heart icons and score button reinforce progress

Arduino Sketch

const int pins[] = {2,6,  3,7,  4,8,  5,9}; // even indices = hand, odd = foot

void setup() {
  Serial.begin(9600);
  for (int i = 0; i < 8; i++) pinMode(pins[i], INPUT);
}

void loop() {
  for (int i = 0; i < 8; i++) {
    if (digitalRead(pins[i]) == HIGH) {
      int baseZone = (i / 2) + 1;          // 1–4
      int sendVal  = (i % 2 == 0)
                     ? baseZone           // hand: 1–4
                     : baseZone + 4;      // foot: 5–8

      Serial.println(sendVal);

      // wait release
      while (digitalRead(pins[i]) == HIGH) delay(5);
      delay(300);
    }
  }
}

This Arduino sketch  scans eight digital inputs (pins 2–9, paired as hand vs. foot buttons), and whenever it detects a button press it:

1. Determines which of the four “zones” you pressed (buttons in pairs 2/6: zone 1, 3/7: zone 2, etc).

2. Adds 4 to the zone number if it was a “foot” button, so hands send 1–4 and feet send 5–8.

3. Sends that zone ID over the serial port.

4. Waits for you to release the button and debounces for 300 ms before scanning again.

Circuit Schematic

p5.js Sketch

let menuImg, infoImg, gameOverImg;
let axelSound;           
let songStarted = false;

let serial, canvas, scoreBtn;
let gameState = 'menu';

let useSound = false; 
let osc;

function preload(){ //preloading images and sounds
  menuImg = loadImage('menu.png');
  infoImg  = loadImage('info.png');
  gameOverImg = loadImage('gameover.png');
  axelSound = loadSound('axelf.mov');
  songImg = loadImage('song.png');
  diffImg = loadImage('difficulty.png');

}



const tileH = 60, zoneH = 20,  
      zoneY = 320, // tiles get removed at this point
      removalY = 400, colWidth = 400 / 4;


// UI buttons for navigation buttons, menu and gameover
const navButtons = { back:{x:10,y:10,w:80,h:30,label:'Back'}, home:{x:100,y:10,w:80,h:30,label:'Home'} };

const menu = { title:'Piano Tiles', start:{x:159,y:222,w:86,h:20,label:'Start'}, info:{x:160,y:261,w:86,h:20,label:'Info'} };

const gameOverButtons = { tryAgain:{x:150,y:180,w:100,h:40,label:'Try Again'}, home:{x:150,y:240,w:100,h:40,label:'Home'} };

// Song and pattern deciding what foot/hand tile goes to what column in order
const songs = [
   { id: 'axelf', name: 'Crazy Frog', sample: axelSound, pattern: [
  { col: 0, isFoot: true  },  
  { col: 1, isFoot: true },  
  { col: 0, isFoot: true  },  
  { col: 2, isFoot: false  },  
  { col: 3, isFoot: false },
  { col: 3, isFoot: true  },  
  { col: 0, isFoot: false }, 
  { col: 2, isFoot: false  },  
  { col: 0, isFoot: true  },  
  { col: 2, isFoot: false }, 
  { col: 1, isFoot: true  }, 
  { col: 1, isFoot: false },  
  { col: 3, isFoot: true  },  
  { col: 1, isFoot: true  },  
  { col: 0, isFoot: false },
  { col: 3, isFoot: true  },  
  { col: 2, isFoot: false },  
  { col: 1, isFoot: true  }, 
  { col: 0, isFoot: true  },  
  { col: 3, isFoot: true },
  { col: 0, isFoot: true  },  
  { col: 1, isFoot: true },  
  { col: 0, isFoot: true  },  
  { col: 2, isFoot: false  },  
  { col: 3, isFoot: false },
  { col: 3, isFoot: true  },  
  { col: 0, isFoot: false },  
  { col: 2, isFoot: false  },  
  { col: 0, isFoot: true  },  
  { col: 2, isFoot: false }, 
  { col: 1, isFoot: true  },  
  { col: 1, isFoot: false },  
  { col: 3, isFoot: true  }, 
  { col: 1, isFoot: true  },  
  { col: 0, isFoot: false },
  { col: 3, isFoot: true  },  
  { col: 2, isFoot: false },  
  { col: 1, isFoot: true  },  
  { col: 0, isFoot: true  },  
  { col: 3, isFoot: true }, 
 { col: 3, isFoot: true  },  
  { col: 2, isFoot: false },  
  { col: 1, isFoot: true  },  
  { col: 0, isFoot: true  },  
  { col: 3, isFoot: true },
  { col: 3, isFoot: true  },  
  { col: 2, isFoot: false }
] }
];
const songBoxes = []; 
// speed increases as difficulty increases
const difficulties = [ {label:'Easy',speed:4}, {label:'Medium',speed:6}, {label:'Hard',speed:8} ];
const diffBoxes = [];


let currentSong, noteIndex=0;
let score=0, lives=3, currentSpeed=0; // set score to 0 and lives 3
let tile=null, missTime=0;
let gameOverStartTime=0, gameOverTimeout=5000;

function setup(){
    songs[0].sample = axelSound;

  canvas=createCanvas(400,400);
  canvas.mousePressed(handleMouse);
  canvas.elt.oncontextmenu=()=>false;

  // audio
  useSound = typeof p5.Oscillator==='function';
  if(useSound) osc=new p5.Oscillator('sine');

  // serial
  serial=createSerial();
  const ports=usedSerialPorts(); if(ports.length) serial.open(ports[0],{baudRate:9600});

  // UI boxes for the song
  songBoxes.push({ x: 129, y: 216, w: 145, h:  75, idx: 0 });

  // and for difficulties:
  diffBoxes.length = 0; 

  //levels
diffBoxes.push({ x: 158, y: 182, w:  86, h:  32, idx: 0 }); //easy
diffBoxes.push({ x: 158, y: 235, w:  86, h:  32, idx: 1 }); //med
diffBoxes.push({ x: 158, y: 289, w:  86, h:  32, idx: 2 }); //hard

  // white button that represents score
  scoreBtn=createButton('Score: 0');
  scoreBtn.position(width-90,10);
  scoreBtn.style('background-color','#FFFFFF').style('color','rgb(25,1,1)').style('padding','6px 12px');
}

function draw(){
  
    // console.log(`x: ${mouseX}, y: ${mouseY}`); used for coordinates for the ui boxes

  background(30);
  switch(gameState){
    case 'menu':            drawMenu();         break;
    case 'info':            drawInfo();         break;
    case 'songSelect':      drawSongSelect();   break;
    case 'difficultySelect':drawDiffSelect();   break;
    case 'playing':         drawGame();         break;
    case 'gameOver':        drawGameOver();     break;
    
  }
}

function drawMenu()
{ 

  textAlign(CENTER,CENTER);
  fill(255); textSize(32);
  text(menu.title, width/2, 100);
  drawButton(menu.start);
  drawButton(menu.info);
  
    // draw menu background image
  image(menuImg, 0, 0, width, height);
} 

function drawInfo() {
  // full screen info background
  image(infoImg, 0, 0, width, height);

  // then draw back button on top:
  drawButton(navButtons.back);
}

function drawSongSelect()

{ 
    image(songImg, 0, 0, width, height);

 
drawButton(navButtons.back); 
drawButton(navButtons.home); }

function drawDiffSelect()
{ 
    image(diffImg, 0, 0, width, height);
  //  // debug draw the clickable areas
  // diffBoxes.forEach(b => {
  //   noFill();
  //   stroke(255, 0, 0);
  //   rect(b.x, b.y, b.w, b.h);
  // });
drawButton(navButtons.back); drawButton(navButtons.home); }

function drawGame(){
  
  // Lives
  noStroke(); fill('red'); for(let i=0;i<lives;i++) ellipse(20+i*30,20,20,20);
// Dividers and zones in the game
rectMode(CORNER);
stroke(255); strokeWeight(4);
line(100,0,100,height);
line(200,0,200,height);
line(300,0,300,height);

noStroke();
// draws the 4 coloured hit zones:
const colors = ['#3cb44b','#4363d8','#ffe119','#e6194b'];
noStroke();
colors.forEach((c,i) => {
  fill(c);
  rect(i * colWidth, zoneY, colWidth, zoneH);
});

// draws the falling tiles
if (tile) {
  tile.y += tile.speed;

  if (tile.y - tileH/2 > removalY) {
    missTime = millis();
    advanceTile(false);
  } else {
    rectMode(CENTER);
    noStroke();
    fill(tile.isFoot ? '#4363d8' : '#e6194b');
    rect(tile.x, tile.y, tile.w, tileH);
  }
}




  if(serial && serial.available()>0){
    let raw=serial.readUntil('\n').trim(); let z=int(raw)-1;
    if(z>=0&&z<8){ let col=z%4; let isHand=z<4; handleHit(col,isHand); }
  }


  // keeps count of the score
  noStroke(); 
  fill(255); 
  textAlign(LEFT,TOP); 
  textSize(16); 
  text('Score:'+score,10,40);
  if(millis()-missTime<500){ textAlign(CENTER,CENTER); textSize(32); fill('red'); text('MISS',width/2,height/2); }
}

function startPlaying() {
  // only play once
  if (currentSong.sample && !songStarted) {
    currentSong.sample.play();
    songStarted = true;
  }
}


function drawGameOver()
{ 
  if (currentSong.sample && songStarted) {
  currentSong.sample.stop();
}
  image(gameOverImg, 0, 0, width, height);
 
 let e=millis()-gameOverStartTime; 
 if(e>=gameOverTimeout){ resetGame(); gameState='menu'; return;} let r=ceil((gameOverTimeout-e)/1000); 
 textAlign(RIGHT,BOTTOM); 
 textSize(14); 
 fill(255); 
 text('Menu in:'+r,width-10,height-10); }

function drawButton(b)
{ rectMode(CORNER);
 fill(100); 
 rect(b.x,b.y,b.w,b.h);
 fill(255); 
 textAlign(CENTER,CENTER);
 textSize(16); 
 text(b.label,b.x+b.w/2,b.y+b.h/2); }

function handleMouse() {
  // game over screen
  if (gameState === 'gameOver') {
    if (hitBox(gameOverButtons.tryAgain)) {
      resetGame();
      noteIndex = 0;
      spawnNextTile();
      // replay sample if any
      if (currentSong.sample) {
        let s = (typeof currentSong.sample === 'function')
                  ? currentSong.sample()
                  : currentSong.sample;
        if (s && typeof s.play === 'function') s.play();
      }
      gameState = 'playing';
    } else if (hitBox(gameOverButtons.home)) {
      resetGame();
      gameState = 'menu';
    }
    return;
  }

  // info screen
  if (gameState === 'info') {
    if (hitBox(navButtons.back)) {
      gameState = 'menu';
    }
    return;
  }

  // navigation (Home/Back) before the game starts
  if (gameState !== 'playing') {
    if (hitBox(navButtons.home)) {
      resetGame();
      gameState = 'menu';
      return;
    }
    if (gameState === 'songSelect' && hitBox(navButtons.back)) {
      gameState = 'menu';
      return;
    }
    if (gameState === 'difficultySelect' && hitBox(navButtons.back)) {
      gameState = 'songSelect';
      return;
    }
  }

  // main menu
  if (gameState === 'menu') {
    if (hitBox(menu.start)) {
      gameState = 'songSelect';
    } else if (hitBox(menu.info)) {
      gameState = 'info';
    }
    return;
  }

  if (gameState === 'songSelect') {
    songBoxes.forEach(b => {
      if (hitBox(b)) {
        currentSong = songs[b.idx];
        gameState = 'difficultySelect';
      }
    });
    return;
  }

  // select difficulty
if (gameState === 'difficultySelect') {
  diffBoxes.forEach(b => {
    if (hitBox(b)) {
      currentSpeed = difficulties[b.idx].speed;
      resetGame();
      noteIndex = 0;
      spawnNextTile();

      // play sample only once
      if (currentSong.sample && !songStarted) {
        currentSong.sample.play();
        songStarted = true;
      }

      gameState = 'playing';
    }
  });
  return;
}


}


function resetGame() {
  score = 0;
  scoreBtn.html('Score: 0');
  lives = 3;
  missTime = 0;
  gameOverStartTime = 0;
  tile = null;
  songStarted = false;   // reset song playing here
}


function handleHit(col, isHandButton) {
  // wrong button type
  if ((tile.isFoot && isHandButton) ||
      (!tile.isFoot && !isHandButton)) {
    missTime = millis();
    lives--;
    if (lives <= 0) {
      if (currentSong.sample) axelSound.stop();
      if (useSound)         osc.stop();
      gameState = 'gameOver';
      gameOverStartTime = millis();
    }
    return;
  }

  // otherwise the normal hit test
  if (tile && col === tile.col
      && tile.y - tileH/2 < zoneY + zoneH
      && tile.y + tileH/2 > zoneY) {
    advanceTile(true);
  } else {
    missTime = millis();
    lives--;
    if (lives <= 0) {
      if (currentSong.sample) axelSound.stop();
      if (useSound)         osc.stop();
      gameState = 'gameOver';
      gameOverStartTime = millis();
    }
  }
}




function advanceTile(sc) {
  if (sc) {
    // check win condition: if this was the last note in the pattern, flag a win and go straight to Game Over
    if (noteIndex === currentSong.pattern.length - 1) {
      isWinner = true; // mark winner
      gameState = 'gameOver';
      gameOverStartTime = millis();
      return;
    }

    if (useSound && !currentSong.sample) {
      osc.freq(currentSong.melody[noteIndex]);
      osc.start();
      osc.amp(0.5, 0.05);
      setTimeout(() => osc.amp(0, 0.5), 300);
    }

    score++;
    scoreBtn.html('Score:' + score);

  } else {
    lives--;
    if (lives <= 0) {
      // lose flag
      isWinner = false;
      if (currentSong.sample) axelSound.stop();
      if (useSound)         osc.stop();

      gameState = 'gameOver';
      gameOverStartTime = millis();
      return;
    }
  }

  // advance to the next note
  noteIndex = (noteIndex + 1) % currentSong.pattern.length;
  spawnNextTile();
}


function spawnNextTile() {

  const p = currentSong.pattern[noteIndex];
  tile = {
    x:    p.col * colWidth + colWidth/2,
    y:    0,
    w:    colWidth,
    h:    tileH,
    speed: currentSpeed,
    col:  p.col,
    isFoot: p.isFoot,
    immune: false
  };
}



function hitBox(b){ return mouseX>=b.x&&mouseX<=b.x+b.w&&mouseY>=b.y&&mouseY<=b.y+b.h; }

 

Key p5.js Highlights

-preload() loads menu.png, info.png, gameoverwinner.png, gameoverloser.png, and axelf.mov

-drawMenu()/drawInfo() render full screen images before buttons

-handleMouse() manages navigation 

-drawGame() reads serial, falls and draws tiles, enforces hit logic

-advanceTile() checks win/lose and plays audio

Arduino and p5.js Communication

-Arduino sends an integer (1–8) on each button press

-p5.js reads via serial.readUntil(‘\n’), maps zone%4 to columns and zone<5 to hand/foot

-Immediate feedback loop: tile hit/miss on-screen mirrors physical input

Proud of

i’m proud of how I designed my pedals to work so that it holds the button right under the top part of the pedal, and it doesn’t activate till stepped on it. I’m also proud of how I soldered the wires together since it’s my first time. I also like how my code aesthetics turned out.

Future Improvements

I was thinking of maybe trying to have a multiplayer mode would be cooler: split-screen hand vs. foot competition, and maybe implementing a leaderboard function: save high scores.

My user testing for the DIY Disco Set revealed that the project was generally intuitive and easy to navigate, largely due to the  instructions page I added and the exploratory nature of the buttons. Users were able to understand the mapping between controls and the resulting effects without much guidance, which speaks to the project’s accessible design. However, I think there was noticeable confusion around the vinyl mechanism. The instructions I wrote mentioned that they should spin the vinyl to start dj-ing, on a mechanical perspective this was supposed to just jump start the motor and then it would start spinnning on its own, but users continued to manually spin and touch it even after the motor had taken over, unintentionally disrupting its movement. This suggests that while the concept was understood, the transition from manual to motorized spinning was not entirely clear. To address this, I think the instructions could be refined to emphasize that spinning is only required at the beginning, and clearer visual feedback, so such as an indicator light or animation, could help users recognize when the motor is engaged.

For my final project, I decided to shift away from my original idea which was a traditional music box, because I realized it wasn’t interactive enough. Instead, I’ve reworked the concept into a DIY DJ set. So, the new version still keeps the essence of the music box, especially the spinning element, but now users will be spinning a vinyl or maybe a disco ball.

The project will allow users to shuffle through songs, change the playback speed, and add sound effects to simulate dj-ing. I’m also thinking of  incorporating NeoPixels to either synchronize with the music or enhance the overall atmosphere visually.  For this project, the Arduino would handle the  physical input, so when users press buttons, specific actions are triggered (e.g shuffle, speed, sound effect). On the p5.js side, I would build a music visualizer that will deconstruct the audio and represent it visually by showing changes in loudness and frequency over time.

 

Overall: All navigated through the Start, Song, Difficulty successfully and began playing within 30 s.

copy_71E82669-1896-402A-A155-F76C8BE9E1AD

Confusion points:

the hit zones being the same colors as the hand buttons, they think the colors of the tile corresponds to the ‘color’ of the hand button.

It was avoided when people actually read the instructions page.

What worked well:

Physical button feedback, Audio cues, Lives (heart icons) and score counter provided clear feedback.

Areas to improve:

Add visual cues for “red = hand” and “blue = foot” like an icon for each tile

Piano Tiles: Hand and Foot Rhythm Game

1. Concept

I’m building a two-input rhythm game that connects an Arduino button  (4 hand + 4 foot buttons) with a p5.js “Piano Tiles”

Goal: Hit falling tiles in time to the “Crazy Frog” melody.
Win: Complete the entire pattern before running out of lives.
Lose: Lose all 3 lives by hitting the wrong side (hand vs foot) or missing too many tiles.

2. Arduino to p5 Communication

-Inputs and Outputs

Inputs (Arduino): 8 buttons

Pins 2–5: “hand” buttons
Pins 6–9: “foot” buttons

Outputs (Arduino to p5):

On each press, Arduino sends a single integer ‘1…8’ over Serial:

3. Progress to Date

Arduino:

Wired 8 buttons with external pull-downs on pins 2–9

p5 Sketch detects presses and sends 1–8 over Serial

p5.js:

Created menu/info/song/difficulty screens with custom images
Implemented serial reading + tile engine + hit logic + lives/score UI
Integrated “Crazy Frog” sample playback and game over logic

Next Steps:

Fill out the full 44-step Crazy Frog pattern for a complete song

Piano Tiles: Hand and Foot

1. overview

A rhythm-game where you must hit tiles falling in four columns using Arduino-connected hand (red) and foot (blue) buttons. Complete the Crazy Frog song to win; miss 3 times and you lose.

2. Arduino

Buttons: 4 hand inputs (pins 2–5), 4 foot inputs (pins 6–9)
Wiring: Buttons wired to +5 V with external pull-downs to GND

3. p5.js

Tile engine:

-Spawns one tile at a time per a predefined ;pattern[]’ of ‘{col, isFoot}’ steps
-Draws tile in its column, colored by ‘isFoot’ (blue) or hand (red)
-Moves tile downward at speed chosen by difficulty

-Lose if lives hit zero; win if you clear the last tile before the sample ends

Video:

Fixes made:

The p5 program had issues with counting the score and incrementing it. Turns out, there was a Upper case – lower case difference between the Serial.print by arduino and what was being expected by the P5 program. This in return rendered no score on the P5 canvas. This was identified and fixed before the showcase.

The P5 canvas also had text mis-alignment, which was worked towards as well.

In addition to it, the bearing  kept on falling off, this was then fixed by sharpening the wooden stick using a giant sharpner Mr Dustin got me. Using it, the diameter was reduced enough to fit into the hole inside the bearing!

Drilling a pilot hole to put a M3 screen through was hard, hence didn’t take the risk, but implementation of that surely would have made sure it never came off.

Feedback from the audience:

The audience loved the action packed mechanical project. The best part was the frustration and the eagerness to compete. The concept of timer garnered total attention, and moving mechanism alongside trajectory of markers and tape made it unique in its on way. Some people did suggest going for more easier level, which was taken into consideration.

The open-rounded lid as stationary holder was used. Maybe in the future, one with bigger diameter can be used to facilitate the user in their gameplay.

So far, what stood out for me as complement, was the genuineness  the project had in terms of how it was built and was structured. I take pride in getting my hands dirty and full of splinters and hot glue, and hence, I found it to be an overall success.

Of course the release mechanism would have taken to it to next level, and that is something I will be working towards over the Summers!