BEATHOVEN

CONCEPT

The concept is inspired by Incredibox, a website first introduced in 2009 that offered a unique experience for creating music. No musical talent or knowledge of music composition is required. It’s designed so that any combination of audio seamlessly works with each other, allowing individuals to create their own music pieces. I found playing music on this website very satisfying and rewarding.

This inspiration drove the concept behind “Beat-Hoven.” Instead of different musical instruments, the music mixed is beatboxing. For the visuals, initially colorless avatars stand idle; once assigned a beatbox, their outfits change accordingly. Additionally, everything is kept in sync with an 8 sec cycle.

IMPLEMENTATION

First, I started by creating the basic mechanism of the project. I researched how buttons worked for the first time and created a small model of a single button that plays a beat when clicked. However, with this first step, I faced many issues. The first button I used was faulty, which took me a lot of time to realize. Next, I realized that a single button press sent multiple true values at once instead of the one return value I was looking for. This caused a lot of issues with the implementation of the button; however, I figured out how to fix the problem.

Next, I started creating the foundation of the project by connecting the button press to start the audio of the intended player. The first issue I faced was how to send an instruction to stop the audio. Since the button only sends a high signal when pressed, I had to interpret it and check if the button pressed was intended to start or stop the audio. I was able to do that with a few checks, if functions, and flags.

After figuring out the start and stop of the audio, it was time to implement it on a larger scale. I started soldering the rest of the buttons and got them ready. Then I started importing the audio assets to each character one by one to p5. After that, I started linking them to the audio and adjusting the Arduino code to accommodate the buttons.

However, the challenges were not over. The next problem was creating the cycle that decides when an audio is played and when it is stopped to keep the beats in sync. Unfortunately, Google wasn’t helpful in any of the problems I faced since everything worked differently in my code. After 3 hours of figuring it out, I moved on to the next problem: Images! Since I had 8 characters, each character had its own audio, image, and logo. It was the image stage, and I faced many issues, especially since I had a background that was refreshing with each loop. The character that appeared when the button was pressed disappeared since the background was called in the next frame. I had to redesign the whole page to accommodate this.

After figuring that out and adjusting it so that the characters change when they’re not playing, I had to find a way to inform the user that the button-pressed signal has been received. I had this problem since the character does not play until the new cycle starts. So, if the user selected a character between cycles, they would not know if the press was successful or not, and they would press again, which would deactivate the character. I decided to use their logos. One would be colored, and the other monochrome, symbolizing that the character is not selected. After implementing it and linking it with a new function, the project started to take its form.

The next step was to work on the aesthetics. I designed a logo for the page, decided on the iconic name “Beat-Hoven,” and designed the homepage. However, the homepage was missing something – music! I found a mix that was created using beatbox and applied it for the background music. Next, I created the info page that explained the game. Now, the code was almost done, and it was time to work on the container design. I designed the box using a template from the web and laser-cut it on acrylic. Next, I assembled everything into the container and drilled a hole for the Arduino data wire. Finally, I printed an image of the characters and attached it to the box to make it stand out. And with this the project was completed.

 For the Arduino code:

I used 8 buttons which the Arduino received signals from and sent it to p5. It has a delay to only receive one input from the button per press to avoid the spamming of inputs.

For the p5

There are so many checks to ensure the program runs smoothly.

• receives the input of the button
• turns the flag related to the character to active if not active and turns it off if active
• checks to see the next cycle has started
  • if activation flag is active
    • activate the character audio and display the image
  • if flag is deactivated
    • deactivate the character, stop the audio and change the image

• parallel check
  • • if the activation flag is active
      • display the colored logo immediately and do not wait for the new cycle
    • if flag is deactivated
      • display the monochrome logo for the character
    • if the page title is pressed
      • deactivate all characters
      • turn all the flags to deactivated.
      • return to the main page
      • start the background music
    • if the info page is pressed
      • switch the page and do not turn off the background music

Aspects I am particularly proud of

There are so many to mention, but here’s a few

creating a fully functioning game that was created by a team of professionals in a few days and nights

figuring out ways around the obstacles I faced

project turning out to be better than I imagined and not just satisfying the goals

learning how to use a laser cutter and how to solder for the first time

Areas for improvement

I would love to add animation to the phase to be in sync with the audio and maybe another version with different characters.

BEATHOVEN:

Showcase: showcase

My final project is a motion tracking CCTV camera that utilizes poseNet. The basic idea is to use poseNet to locate a person on the screen, and then send that data to a 3D printed CCTV camera mounted on top of a servo motor. By mapping the location data to the angles of the servo motor, it creates the illusion that the CCTV camera is following the person. In addition to the camera tracking, I also wanted to create a motion detecting p5 interface. After watching coding train tutorials on this effect, I discovered a method that uses pixel array data to isolate the moving person from the background, which I found really cool.

A large part of my process involved testing whether the servo-poseNet idea would work or not, and my draft of the final project documents this discovery. For the final project, I had several challenges ahead of me, including creating the p5 interface, figuring out the CCTV camera, and building a base for the camera and motor.

First, with the CCTV camera, I referred to the professor’s slides and came across a website with various 3D models that could be 3D printed. With the professor’s guidance on using the Ultimaker 3, I successfully 3D printed a CCTV camera that was the perfect size for the motor, in my opinion.

Next, I focused on the p5 interface. As mentioned earlier, I aimed to achieve a motion detection look. By applying multiple layers of effects such as grain, blur, and posterize, I was able to create an old-school CCTV footage vibe while also giving it a unique appearance that doesn’t resemble a typical CCTV camera. I wanted to capture the point of view of a camera trying to detect motion.

The final step for me was priming and spray painting the CCTV camera white, and finding the right base for it. Since I wanted to position it behind the laptop, I needed a base of suitable height. I found a cardboard box in my room and repurposed it as the shell for the CCTV camera base. I drilled a large piece of wood into it, which serves as a sturdy base for the motor. I then used wood glue to attach the motor to the wood slab, and glued the motor’s base plate to the CCTV camera.

The following is the code for my Arduino and p5 project:

// video, previous frame and threshold for motion detection
let video; 
let prev;
let threshold = 25;

// Variables for motion functions and positions

let mfun = 0;
let motionY = 0;

let lerpX = 0;
let lerpY = 0;

// Font for overlay text and PoseNet related variables

let myFont;
let poseNet;
let pose;
let skeleton;
let loco = 0;

function preload() {
  myFont = loadFont("VCR_OSD_MONO_1.001.ttf");
}

function setup() {
//   low frame rate for a cool choppy motion detection effect
  frameRate(5);

  createCanvas(windowWidth, windowHeight);
  pixelDensity(1);

  video = createCapture(VIDEO);
  video.size(windowWidth, windowHeight);
  video.hide();

    // Create an image to store the previous frame

  prev = createImage(windowWidth, windowHeight);

    // Initialize PoseNet and set up callback for pose detection

  poseNet = ml5.poseNet(video, modelLoaded);
  poseNet.on("pose", gotPoses);
}

// Callback for when poses are detected by PoseNet

function gotPoses(poses) {
  //console.log(poses);
  if (poses.length > 0) {
    pose = poses[0].pose;
    skeleton = poses[0].skeleton;
  }
}

// Callback for when PoseNet model is loaded

function modelLoaded() {
  console.log("poseNet ready");
}

function draw() {
  
//   Check for serial port
  if (!serialActive) {
    text("Press Space Bar to select Serial Port", 20, 30);
  } else {
    text("Connected", 20, 30);
  }
  
//   Check for pose and get nose pose data

  if (pose) {
    fill(255, 0, 0);
    ellipse(pose.nose.x, pose.nose.y, 20);

//     location of pose nose
    loco = int(pose.nose.x);
//     value mapped for servo motor
    val = int(map(loco, 0, windowWidth, 60, 120));

    print(val);
  }
  
  
  background(0);
// load pixels for motion detection
  
  video.loadPixels();
  prev.loadPixels();

  threshold = 40;

  let count = 0;
  let avgX = 0;
  let avgY = 0;

  // Flip the canvas for video display
  push();
  translate(width, 0);
  scale(-1, 1);
  image(video, 0, 0, video.width, video.height);
  pop();

    // Analyzing the pixels for motion detection

  loadPixels();
  for (let x = 0; x < video.width; x++) {
    for (let y = 0; y < video.height; y++) {
            // Current and previous pixel colors

      let loc = (x + y * video.width) * 4;
      let r1 = video.pixels[loc + 0];
      let g1 = video.pixels[loc + 1];
      let b1 = video.pixels[loc + 2];
      let r2 = prev.pixels[loc + 0];
      let g2 = prev.pixels[loc + 1];
      let b2 = prev.pixels[loc + 2];

      // Calculate color distance

      let d = distSq(r1, g1, b1, r2, g2, b2);

      if (d > threshold * threshold) {
        avgX += x;
        avgY += y;
        count++;
        
        // Fliped motion effect pixels
        let flippedLoc = (video.width - x - 1 + y * video.width) * 4;
        pixels[flippedLoc + 0] = 155;
        pixels[flippedLoc + 1] = 155;
        pixels[flippedLoc + 2] = 255;
      } else {
        let flippedLoc = (video.width - x - 1 + y * video.width) * 4;
        pixels[flippedLoc + 0] = 190;
        pixels[flippedLoc + 1] = 255;
        pixels[flippedLoc + 2] = 155;
      }
    }
  }

    // Updating the pixels on the canvas

  updatePixels();

    // Calculate the average motion position if significant motion is detected

  if (count > 200) {
    motionX = avgX / count;
    motionY = avgY / count;
  }

  // Mirror the motion tracking coordinates
  //     let flippedMotionX = width - motionX;

  //     lerpX = lerp(lerpX, flippedMotionX, 0.1);
  //     lerpY = lerp(lerpY, motionY, 0.1);

  //     fill(255, 0, 255);
  //     stroke(0);
  //     strokeWeight(2);
  //     ellipse(lerpX, lerpY, 36, 36);
  
//   MOREE EFFECTZZZZ

  filter(INVERT);
  prev.copy(
    video,
    0,
    0,
    video.width,
    video.height,
    0,
    0,
    prev.width,
    prev.height
  );
  filter(ERODE);
  filter(POSTERIZE, random(10, 20));
  
  drawGrid(); // Draw the grid on top of your content

  drawSurveillanceOverlay(); //surveillance overlay cam
  drawGrain(); // grain effect for old school cctv vibes

  filter(BLUR, 1.5); // blur effect to achieve that vhs quality
}

function distSq(x1, y1, z1, x2, y2, z2) {
  return sq(x2 - x1) + sq(y2 - y1) + sq(z2 - z1);
}

  // toggle full screen

function mousePressed() {
  if (mouseX > 0 && mouseX < width && mouseY > 0 && mouseY < height) {
    let fs = fullscreen();
    fullscreen(!fs);
  }
}


function drawGrain() {
  loadPixels();
  for (let i = 0; i < pixels.length; i += 4) {
    let grainAmount = random(-10, 10);
    pixels[i] += grainAmount; // red
    pixels[i + 1] += grainAmount; // green
    pixels[i + 2] += grainAmount; // blue
    // pixels[i + 3] is the alpha channel
  }
  updatePixels();
}

function drawSurveillanceOverlay() {
  textFont(myFont); // Set the font
  textSize(32); // Set the text size
  
  // Draw border
  noFill();

  strokeWeight(5);

  stroke(0, 0, 0, 255);

  rect(9, 9, width - 16, height - 16);
  stroke(250, 250, 250, 255);
  strokeWeight(2.1);

  rect(9, 9, width - 16, height - 16);

  // Display timestamp
  fill(250, 50, 50);
  fill(250, 250, 250);

  stroke(0, 120);

  textSize(30);
  textAlign(CENTER, TOP);
  text(
    new Date().toLocaleString(),
    windowWidth / 2,
    windowHeight - windowHeight / 11
  );

//  cam 01
  textSize(17);
  fill(50, 250, 55);
  text("CAM 01", width - width / 19, windowHeight / 29);
}

function drawGrid() {
  let gridSize = 15; // Size of each grid cell
  
//   only the horizontal lines

  stroke(205, 3); // Grid line color (white with some transparency)
  strokeWeight(1); // Thickness of grid lines

  for (let x = 0; x <= width; x += gridSize) {
    for (let y = 14; y <= height + 16; y += gridSize) {
      // line(x, 10, x, height);
      line(11, y, width - 10, y);
    }
  }
}

// serial connection
function keyPressed() {
  if (key == " ") {
    // important to have in order to start the serial connection!!
    setUpSerial();
  }
}

function readSerial(data) {
  ////////////////////////////////////
  //READ FROM ARDUINO HERE
  ////////////////////////////////////

  if (data != null) {
    // make sure there is actually a message
    // split the message
    let fromArduino = split(trim(data), ",");
    // if the right length, then proceed
    if (fromArduino.length == 2) {
      // only store values here
      // do everything with those values in the main draw loop
      print("nice");
      // We take the string we get from Arduino and explicitly
      // convert it to a number by using int()
      // e.g. "103" becomes 103
    }

    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    let sendToArduino = val + "\n";
    writeSerial(sendToArduino);
  }
}

p5 👆

#include <Servo.h>

Servo myservo;  // create servo object to control a servo


void setup() {

  Serial.begin(9600);

  myservo.attach(9);

  // start the handshake
  while (Serial.available() <= 0) {
    digitalWrite(LED_BUILTIN, HIGH); // on/blink while waiting for serial data
    Serial.println("0,0"); // send a starting message
    delay(300);            // wait 1/3 second
    digitalWrite(LED_BUILTIN, LOW);
    delay(50);
           myservo.write(0);                  // sets the servo position according to the scaled value

  }
}

void loop() {
  // wait for data from p5 before doing something
  
  while (Serial.available()) {
    Serial.println("0,0");
    digitalWrite(LED_BUILTIN, HIGH); // led on while receiving data
    int value = Serial.parseInt();
    if (Serial.read() == '\n') {
       myservo.write(value);                  // sets the servo position according to the scaled value
    }
  }
}

arduino 👆

Overall, I am happy with how the project was realized. It has been a very educational experience for me, as it has allowed me to learn about posenet, 3D printing, and visual effects. These skills will be valuable for my future capstone project, which will focus on surveillance.

Documentation/User Testing from the IM Showcase:

Automated Trash Sorting System: A Sustainable Waste Solution

Concept:

The project’s main goal is to develop a trash sorting system that can help the environment in several ways. This system relies on technologies like Arduino, capacitive, and IR sensors to automatically detect and sort different types of objects placed in a trash bin, distinguishing between plastic and general waste. This technology offers several environmental benefits.

The automatic trash sorting system helps make sure that things we can recycle, like plastic, don’t get thrown away in the trash and sent to landfills. When plastic items are sorted out from regular trash, we can recycle them correctly, which saves important materials and reduces the amount of new plastic things we have to make. This is good for the environment because it means less pollution from plastic and less energy used to make new plastic stuff.

#include <Servo.h>;

Servo GateServo;

#define capSensor A3 //Pin A3 (analog)
#define irSensor 4 //Pin 4 (digital)
#define GateServoPin 3 // GateServo pin

bool openTrash = false;
bool autoMode = true;

//Plastic goes to angle 11
//General goes to angle 158
//The closing angle of the gate which is 85 degree
 
int initialServoPosition = 85;
int anglePlastic = 11;
int angleGeneral = 150;

int capValue;

unsigned long GateServoReturnTime = 0;
const unsigned long GateServoReturnDelay = 1000; // 1 seconds in milliseconds

void loop()
{
  while (Serial.available() > 0) {
    // Read the angle value from serial
    String incomingData = Serial.readStringUntil('\n');

    //If the mode is manual enable controlling by the p5js panel
    if (!autoMode) {
      // Check if the incoming data starts with "angle:"
      if (incomingData.startsWith("angle:")) {
        // Extract the angle value from the incoming data
        int angleValue = incomingData.substring(6).toInt();
  
        // Process the angle value if it is within the valid range
        if (angleValue >= 11 && angleValue <= 158) {
//          Serial.println("Received angle: " + String(angleValue));
          // Set the GateServo angle
          GateServo.write(angleValue);
  
          // Reset the timer each time a valid angle is received
          openTrash = true;
          GateServoReturnTime = millis();
        }
      }
    }

 if (incomingData.startsWith("Auto")) {
      autoMode = true;
//      Serial.println("Auto mode: " + String(autoMode));
    }
    else if (incomingData.startsWith("Manual")){
      autoMode = false;
//      Serial.println("Manual mode: " + String(autoMode));
    }
  }

  // Check if it's time to return the GateServo to 85 degrees
  if (millis() - GateServoReturnTime >= GateServoReturnDelay && openTrash) {
    openTrash = false;
    GateServo.write(initialServoPosition);
//    Serial.println("Trash Closed");
  }

Introduction

Transitioning from the conceptualization of ArtfulMotion, a project centered around translating gestures into visual art, I sought to elevate the interactive experience by integrating physical computing. This blog post outlines the genesis of the gesture-controlled robot concept, the nuanced implementation, and the resultant user experiences.

Inspiration and Conceptualization

The inception of this project emanated from a desire to imbue physicality into the realm of gesture-controlled art, a departure from the digital interface. Initially considering an “electronic buddy” or an “art robot,” inspiration struck upon encountering a multidirectional moving robot in a video shared by Professor Riad. The challenge was to replicate this unique motion with standard tires and integrate Bluetooth technology, ultimately opting for a tethered connection.

Gesture Recognition

Leveraging the handpose model from ml5.js, the implementation of gesture recognition unfolded by identifying 21 keypoints on a hand. The model, confined to recognizing one hand at a time, prompted the division of the video feed into segments, each corresponding to a distinct direction of motion. The chosen gestures prioritize intuitive and natural user interactions.

Interaction Design

User interaction revolves around using hand movements captured by a webcam, transforming them into navigational commands for the robot. An onboard button toggles the robot’s state, turning it on or off. Although the current iteration lacks auditory feedback, prospective enhancements will explore the integration of sound cues. The unique motion of the robot necessitates users to rely on intuition, adding an element of engagement.

Technical Implementation

The interaction between the p5.js sketch and the Arduino board relies on tethered serial communication, facilitated by the p5.web-serial.js library. A singular value is dispatched from p5 to Arduino, intricately mapped to specific motion sets.

p5.js sketch:

preload():

function preload() {
  
  instructionPage = loadImage('instructions.png');
  
  for (let i = 1; i <= carNum; i++) {
    carArray.push(new Sprite(spriteWidth, spriteHeight, 160, 80));
    carArray[i - 1].x = 80 + i * 20;
    carArray[i - 1].y = 100 * i;
    carArray[i - 1].spriteSheet = 'spritesheet.png';
    carArray[i - 1].anis.offset.x = 5;
    carArray[i - 1].anis.frameDelay = 8;

    carArray[i - 1].addAnis({
      move: { row: 0, frames: 16 },
    });
    carArray[i - 1].changeAni('move');
    carArray[i - 1].layer = 2;
    carArray[i - 1].visible = false;
  }
}

The preload() function loads the instruction page image and initializes an array of car sprites.

setup():

function setup() {
  createCanvas(windowWidth, windowHeight);
  video = createCapture(VIDEO);
  video.size(width, height);

  handpose = ml5.handpose(video, modelReady);
  handpose.on("predict", results => {
    predictions = results;
  });

  video.hide();
}

The setup() function serves as the initial configuration for the canvas, video capture, and the Handpose model. It establishes the canvas size based on the current window dimensions and initializes the necessary components, such as the video capture object and Handpose model. The modelReady callback function is triggered when the Handpose model is prepared for use, ensuring that the application is ready to detect hand poses accurately.

draw():

function draw() {
  background(255);

  flippedVideo = ml5.flipImage(video);

  // Calculate the aspect ratios for video and canvas
  videoAspectRatio = video.width / video.height;
  canvasAspectRatio = width / height;

  

  // Adjust video dimensions based on aspect ratios
  if (canvasAspectRatio > videoAspectRatio) {
    videoWidth = width;
    videoHeight = width / videoAspectRatio;
  } else {
    videoWidth = height * videoAspectRatio;
    videoHeight = height;
  }

  // Calculate video position
  video_x = (width - videoWidth) / 2;
  video_y = (height - videoHeight) / 2;

  if (currentPage == 1) {
    // display instructions page
    image(instructionPage, 0, 0, width, height);
  }
  else if (currentPage == 2) {
    // serial connection page
    if (!serialActive) {
      runSerialPage();
    } 

    else {
      // hides car animation
      for (let i = 0; i < carNum; i++) {
        carArray[i].visible = false;
      }

      // controlling page
      if (controlState) {
        runControllingPage();
      } 

      // device has been turned off
      else {
        runTorqueyOff();
      }
    }
  }
}

Within the draw() function, various elements contribute to the overall functionality of the sketch. The calculation of video and canvas aspect ratios ensures that the video feed maintains its proportions when displayed on the canvas. This responsiveness allows the application to adapt seamlessly to different window sizes, providing a consistent and visually appealing user interface.

The draw() function is also responsible for managing different pages within the application. It evaluates the currentPage variable, determining whether to display the instruction page or proceed to pages related to serial connection and hand gesture control. This page-switching behavior is facilitated by the mousePressed function, enabling users to navigate through the application intuitively.

readSerial(data):

function readSerial(data) {

  if (data != null) {
    ////////////////////////////////////
    //READ FROM ARDUINO HERE
    ////////////////////////////////////
    if (int(trim(data)) == maxVoltReading) {
      controlState = true;
    }
    else if (int(trim(data)) == minVoltReading){
      controlState = false;
    }

    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    
    let sendToArduino = value + "\n";
    writeSerial(sendToArduino);
    
    // reset value
    value = defaultState;
  }
}

The readSerial(data) function handles communication with an Arduino device. It interprets incoming data, updates the controlState based on voltage readings, and initiates a handshake with the Arduino. This interaction establishes a connection between the physical device and the digital application, enabling real-time responses to user inputs.

drawKeypoints():

function drawKeypoints() {
  for (let i = 0; i < predictions.length; i += 1) {
    const prediction = predictions[i];
    let area = [0, 0, 0, 0, 0];
    for (let j = 0; j < prediction.landmarks.length; j += 1) {
      const keypoint = prediction.landmarks[j];
      fill(0, 255, 0);
      noStroke();
      let x = map(keypoint[0], 0, 640, 0, videoWidth);
      let y = map(keypoint[1], 0, 480, 0, videoHeight);
      ellipse(x, y, 10, 10);
      
      // count number of trues
      // -- helps to detect the area the detected hand is in
      if (withinLeft(x, y)) {
        area[0] += 1;
      }
      if (withinTopCenter(x, y)) {
        area[1] += 1;
      }
      if (withinRight(x, y)) {
        area[2] += 1;
      }
      if (withinMiddleCenter(x, y)) {
        area[3] += 1;
      }
      if (withinBottomCenter(x, y)) {
        area[4] += 1;
      }
      // end of count
    }
    
    // print index
    for (let i = 0; i < area.length; i += 1) {
      if (area[i] == 21) {
        value = i;
      }
    }
  }
}

The drawKeypoints() function utilizes the Handpose model’s predictions to visualize detected keypoints on the canvas. These keypoints correspond to various landmarks on the hand, and their positions are mapped from the video coordinates to the canvas coordinates. By counting the number of keypoints within specific regions, the function determines the area of the hand’s position. This information is crucial for the application to interpret user gestures and trigger relevant actions.

Robot Movement

Arduino schematic diagram:

The robot’s movement encompasses pseudo-forward, pseudo-backward, and rotational movements in either direction around its center. Achieving these nuanced movements involved a methodical trial-and-error process, aligning gestures with intended actions.

Arduino sketch

const int ain1Pin = 3;
const int ain2Pin = 4;
const int pwmAPin = 5;

const int bin1Pin = 8;
const int bin2Pin = 7;
const int pwmBPin = 6;

int buttonValue = 0;
int prevButtonValue = 0;
const int defaultState = -1;

const unsigned long eventInterval = 1000;
unsigned long previousTime = 0;

void setup() {
  // Start serial communication so we can send data
  // over the USB connection to our p5js sketch
  Serial.begin(9600);

  pinMode(LED_BUILTIN, OUTPUT);

  pinMode(ain1Pin, OUTPUT);
  pinMode(ain2Pin, OUTPUT);
  pinMode(pwmAPin, OUTPUT); // not needed really

  pinMode(bin1Pin, OUTPUT);
  pinMode(bin2Pin, OUTPUT);
  pinMode(pwmBPin, OUTPUT); // not needed really

  // TEST BEGIN
  // turn in one direction, full speed
  analogWrite(pwmAPin, 255);
  analogWrite(pwmBPin, 255);
  digitalWrite(ain1Pin, HIGH);
  digitalWrite(ain2Pin, LOW);
  digitalWrite(bin1Pin, HIGH);
  digitalWrite(bin2Pin, LOW);

  // stay here for a second
  delay(1000);

  // slow down
  int speed = 255;
  while (speed--) {
    analogWrite(pwmAPin, speed);
    analogWrite(pwmBPin, speed);
    delay(20);
  }

  // TEST END

  buttonValue = analogRead(A0);
  prevButtonValue = buttonValue;

  // start the handshake
  while (Serial.available() <= 0) {
    digitalWrite(LED_BUILTIN, HIGH); // on/blink while waiting for serial data
    Serial.println(buttonValue); // send a starting message
    delay(50);            // wait 1/3 second
    digitalWrite(LED_BUILTIN, LOW);
    delay(50);
  }

}

void loop() {

   /* Updates frequently */
  unsigned long currentTime = millis();
  /* This is the event */
  if (currentTime - previousTime >= eventInterval) {
    /* Event code */
    buttonValue = analogRead(A0);
    
   /* Update the timing for the next time around */
    previousTime = currentTime;
  }

  while (Serial.available()) {
    // sends state data to p5
    if (buttonValue != prevButtonValue) {
      prevButtonValue = buttonValue;
      Serial.println(buttonValue);
    }
    else {
      Serial.println(defaultState);
    }
    
    // led on while receiving data
    digitalWrite(LED_BUILTIN, HIGH); 

    // gets value from p5
    int value = Serial.parseInt();

    // changes brightness of the led
    if (Serial.read() == '\n' && buttonValue == 1023) {
      
      if (value == 0) {
        // 0
        digitalWrite(ain1Pin, HIGH);
        digitalWrite(ain2Pin, LOW);
        digitalWrite(bin1Pin, LOW);
        digitalWrite(bin2Pin, HIGH);

        analogWrite(pwmAPin, 255);
        analogWrite(pwmBPin, 255);
        // 0

      }

      else if (value == 1) {
        // 1
        digitalWrite(ain1Pin, HIGH);
        digitalWrite(ain2Pin, LOW);
        digitalWrite(bin1Pin, HIGH);
        digitalWrite(bin2Pin, LOW);

        analogWrite(pwmAPin, 255);
        analogWrite(pwmBPin, 255);
        // 1
      }

      else if (value == 2){
        // 2
        digitalWrite(ain1Pin, LOW);
        digitalWrite(ain2Pin, HIGH);
        digitalWrite(bin1Pin, HIGH);
        digitalWrite(bin2Pin, LOW);

        analogWrite(pwmAPin, 255);
        analogWrite(pwmBPin, 255);
        // 2
      }

      else if (value == 3) {
        analogWrite(pwmAPin, 0);
        analogWrite(pwmBPin, 0);
      }

      else if (value == 4) {
        // 4
        digitalWrite(ain1Pin, LOW);
        digitalWrite(ain2Pin, HIGH);
        digitalWrite(bin1Pin, LOW);
        digitalWrite(bin2Pin, HIGH);

        analogWrite(pwmAPin, 255);
        analogWrite(pwmBPin, 255);
        // 4
      }

      else {
        analogWrite(pwmAPin, 0);
        analogWrite(pwmBPin, 0);
      }
    }
  }
  // led off at end of reading
  digitalWrite(LED_BUILTIN, LOW);
}

The setup() function initializes serial communication, configures pins, and performs an initial motor test. Additionally, it sends the initial buttonValue to the p5.js sketch for the handshake.

The loop() function checks if the eventInterval has elapsed and updates the buttonValue accordingly. It handles incoming serial data from the p5.js sketch, sending state data back and adjusting LED brightness. Motor control logic is implemented based on the received values from the p5.js sketch, allowing for different motor configurations.

User Experience

End users find the robot’s unconventional design intriguing, coupled with a sense of awe at its mobility. The brief learning curve is accompanied by occasional glitches arising from imperfections in handpose detection, which may result in initial user frustration.

User Testing

IM Showcase

The IM showcase went well overall. Despite a few technical hiccups during the presentation, the feedback from people who interacted with the project was positive. Some issues raised were ones I had anticipated from user testing, and I plan to address them in future versions of the project.

User Interaction 1:

User Interaction 2:

Aesthetics and Design

Crafted predominantly from cardboard, the robot’s casing prioritized rapid prototyping, considering time constraints. The material’s versatility expedited the prototyping process, and the strategic use of zipties and glue ensured durability, with easily replaceable parts mitigating potential damage.

Future Enhancements

Subsequent iterations of ArtfulMotion 2.0 aspire to introduce gesture controls for variable speed, operational modes such as tracking, and exploration of more robust machine learning models beyond the limitations of handpose. The quest for wireless control takes precedence, offering heightened operational flexibility, potentially accompanied by a structural redesign.

Reflection

The completion of this project within constrained timelines marks a journey characterized by swift prototyping, iterative testing, and redesign. Future endeavors shift focus towards refining wireless communication, structural enhancements, and the exploration of advanced machine learning models.

p5 rough sketch:

P5 Sketch

Concept:

The idea was to create a bot that can be controlled with hand movement using machine learning with the P5.ml library. The P5 sketch has an in-car design with the steering wheel and the Arduino comprises 4dc motors, two ultrasonic sensors, and some LEDs.

Interactive Design:

For user interactivity, I decided to use the P5.js. The computer video webcam helps the machine learning library to detect the movement of the user’s hand and map these movements to the size of the canvas. This information is used to control the steering wheel, which controls the Arduino.

Arduino Code:

For the Arduino aspect, the motors are controlled using switch cases. The Arduino receives cases from the P5 sketch and based on those cases, the Arduino knows how to control the 4 DC motors. The Arduino uses the ultrasonic sensors to detect obstacles and sends this information to the P5 sketch to be viewed by the user.

#include <Servo.h>//include the Servo library/
#include <time.h>//include the time library for seeding random number

Servo myservo;//creating servo object

const int ain1Pin = 3;//setting pins for motor for left side which is connected in parallel
const int ain2Pin = 4;
const int pwmAPin = 5;

const int bin1Pin = 8;//setting pinf for motor for right side 
const int bin2Pin = 7;
const int pwmBPin = 6;

const int trigPin1 = 11;//setting pins for front ultrasonic sensor
const int echoPin1= A0;//I used A0 pin cause I didnt have space...Please pardon me
const int trigPin2 = 10;//setting pins for back ultrasonic sensor
const int echoPin2 = A1;//here too

const int warningbuzzer=12;//setting the buzzer to 12
//servo motor
const int headmovePin=9;//setting the servo motor pin to 9

int wallstop=0;//initializing the obstacle detection

unsigned long previousMillis = 0;//setting millis to 0
const long interval = 5000;//setting interval for the millis


void setup() {
  myservo.attach(headmovePin);//pass the servo pin to the servo library
  
  randomSeed(time(NULL));//seed random number using the current time 

  pinMode(ain1Pin, OUTPUT);//setting the pins as output and input
  pinMode(2, OUTPUT);//light checker
  pinMode(ain2Pin, OUTPUT);//motor pin
  pinMode(pwmAPin, OUTPUT); // not needed really
  pinMode(bin1Pin, OUTPUT);
  pinMode(bin2Pin, OUTPUT);
  pinMode(pwmBPin, OUTPUT); // not needed really
  // Start serial communication so we can send data
  // over the USB connection to our p5js sketch
  pinMode(trigPin1, OUTPUT);
  pinMode(echoPin1, INPUT);
  pinMode(trigPin2, OUTPUT);
  pinMode(echoPin2, INPUT);
  pinMode(warningbuzzer, OUTPUT);

  Serial.begin(9600);//setting the serial band

  while (!Serial.available()) {
    digitalWrite(LED_BUILTIN, HIGH); // on/blink while waiting for serial data
    Serial.println("0,0"); // send a starting message
    delay(300);            // wait 1/3 second
    digitalWrite(LED_BUILTIN, LOW);
    delay(1000);//delay for a second
  }
}

void loop() {
  wallstop=crushstop();
  //When the serial is detected do this
  while (Serial.available()) {//while the serial connection exists
    digitalWrite(LED_BUILTIN, HIGH); // led on while receiving data
    wallstop=crushstop();//update the wallstop from the crush function
    unsigned long currentMillis = millis();//set current millis

    if (currentMillis - previousMillis >= interval) {
      // save the last time you blinked the LED
      previousMillis = currentMillis;
      movehead();//if the interval is reached, move the head
    }
    int movement = Serial.parseInt();//get the movement case from p5
   if (Serial.read() == '\n') {//when we read a new line,
     switch(movement){//execute the following commmands based on the case
      case 0://no movement
        analogWrite(pwmAPin, 0);
        digitalWrite(ain1Pin, HIGH);
        digitalWrite(ain2Pin, LOW);
        analogWrite(pwmBPin, 0);
        digitalWrite(bin1Pin, LOW);
        digitalWrite(bin2Pin, HIGH);
      break;
      case 1://forward
        analogWrite(pwmAPin, 255);
        digitalWrite(ain1Pin, LOW);
        digitalWrite(ain2Pin, HIGH);
        analogWrite(pwmBPin, 255);
        digitalWrite(bin1Pin, HIGH);
        digitalWrite(bin2Pin, LOW);
      break;
      case 2://reverse
        analogWrite(pwmAPin, 255);
        digitalWrite(ain1Pin, HIGH);
        digitalWrite(ain2Pin, LOW);
        analogWrite(pwmBPin, 255);
        digitalWrite(bin1Pin, LOW);
        digitalWrite(bin2Pin, HIGH);
      break;
      case 3://right front
        analogWrite(pwmBPin, 255);
        digitalWrite(bin1Pin, HIGH);
        digitalWrite(bin2Pin, LOW);
      break;
      case 4://left front
        analogWrite(pwmAPin, 255);
        digitalWrite(ain1Pin, LOW);
        digitalWrite(ain2Pin, HIGH);
      break;
      case 5://right back
       analogWrite(pwmBPin, 255);
        digitalWrite(bin1Pin, LOW);
        digitalWrite(bin2Pin, HIGH);
      break;
      case 6://left back
        analogWrite(pwmAPin, 255);
        digitalWrite(ain1Pin, HIGH);
        digitalWrite(ain2Pin, LOW);
      break;
      default://if not case is gotten within our expected range, stop
        analogWrite(pwmAPin, 0);
        digitalWrite(ain1Pin, HIGH);
        digitalWrite(ain2Pin, LOW);
        analogWrite(pwmBPin, 0);
        digitalWrite(bin1Pin, LOW);
        digitalWrite(bin2Pin, HIGH);
      break;
     }
      delay(5);
      Serial.println(wallstop);//send the detection to p5
      
    }
  }
  
}

int crushstop(){//this function returns 1 when an obstacle is ahead and 2 when an obstacle is behind
  int wallstop=0;//initialize wallstop
  digitalWrite(trigPin1, LOW); 
  delayMicroseconds(2); //basically shooting beems and using the time it takes to bounce back to calculate distance
  digitalWrite(trigPin1, HIGH); 
  delayMicroseconds(10); 
  digitalWrite(trigPin1, LOW); 
  // Time it takes for the pulse to travel back from the object long 
  int duration1 = pulseIn(echoPin1, HIGH); 
  // Universal conversion of time into distance in cm 
  int distance1 = duration1 * 0.034 / 2;//divided by two beause its a two way thing

  digitalWrite(trigPin2, LOW); 
  delayMicroseconds(2); 
  digitalWrite(trigPin2, HIGH); 
  delayMicroseconds(10); 
  digitalWrite(trigPin2, LOW); 
  // Time it takes for the pulse to travel back from the object long 
  int duration2 = pulseIn(echoPin2, HIGH); 
  // Universal conversion of time into distance in cm 
  int distance2 = duration2 * 0.034 / 2;
  if(distance1<5){//if collision is detected behind, send 2
    wallstop=2;
    tone(warningbuzzer,2000);//play the tone
    delay(5);
  }
  else if(distance2<5){
    wallstop=1;//if collision is detected infront, send 1
    tone(warningbuzzer,2000);//play the tone
    delay(5);
  }
  else{
    wallstop=0;
    noTone(warningbuzzer);//if nothing is detected dont play a tone
  }
  return wallstop;//return this info
}

void movehead(){
  myservo.write(random(0, 180));//move the head to a random number between 0 and 180
  delay(15);
}

P5.js:

For this part, the P5.js sketch receives the user hand positions info from the P5.ml library and maps it to get its corresponding points on the canvas. After, these values are averaged and the average-X value is used to control the steer’s left or right turn. The average-Y value is used to control the forward and backward movement. Based on this cases are developed and sent to the Arduino for execution of tasks.

The P5 sketch is divided into five parts

Serial connection:

This part is responsible for connecting the Arduino to the P5 sketch and since its not my code I will not post it

Intro Page:

The next part is the intro page. Is welcomes the user into the project and also gives some info to the user about how to use the vehicle. I made the background picture myself using photoshop and then I added some buttons and sounds

class ipage{
  constructor(IB,S1,S2,F1,F2){//receive all needed variables as in images and sound
    this.BG=IB;//the background
    this.S1=S1;//sound
    this.playbool=false;//boolean to start game
    this.helpbool=false;//boolean to open help page
    this.F2=F2//font
  }
  show(){//this shows the designs in the page
    this.BG.resize(windowWidth,windowHeight);//resize the introimage
    image(this.BG,0,0);//displaying background
    textAlign(CENTER);
    this.createhelp();//calling the help function for the help page
    this.playbox();//calling the playbox button function
    if(this.helpbool){//if the helpbool is true display the help page
      this.helppage();
    }
    return this.playbool;//this returns true if the play button is pressed
  }
  
  playbox(){//this function displays the start button
    rectMode(CORNER);
    fill(150,180,40);
      if(mouseX>windowWidth*0.1&&mouseX<(windowWidth*0.1)+350&&mouseY>windowHeight*0.8&&mouseY<(windowHeight*0.8)+150){
      fill(150,0,0);//creating the hover effect
    }
    if(mouseX>windowWidth*0.1&&mouseX<(windowWidth*0.1)+350&&mouseY>windowHeight*0.8&&mouseY<(windowHeight*0.8)+150&&mouseIsPressed&&!this.helpbool){
      S2.pause();// the play is pressed, play pause the background music
      this.S1.play();//playing the start button pressed sound
      this.playbool=true;//sets the playboolean to true 
    }
    rect(windowWidth*0.1,windowHeight*0.8,350,150,60);//drawing the start button
    fill(0);
    textFont(F1);
    textSize(90);
    textAlign(CENTER);
    text("Start",windowWidth*0.1+180,windowHeight*0.8+110);
  }
  createhelp(){
    fill(100);
    if(dist(mouseX,mouseY,windowWidth*0.9,windowHeight*0.1)<=30){
      fill(60);//create hover effect
    }
    if(dist(mouseX,mouseY,windowWidth*0.9,windowHeight*0.1)<=30&&mouseIsPressed){
      this.helpbool=true;//set boolean to open help page
    }
    circle(windowWidth*0.9,windowHeight*0.1,60);//these following code just creates the help button
    fill(255,255,0);
    textSize(30);
    textFont(NORMAL);
    text("?",windowWidth*0.9,windowHeight*0.1+10);
  }
  helppage(){
    fill(100);
    rect(50,50,windowWidth*0.95,windowHeight*0.9,50);
    textFont(this.F2);
    fill(255);
    textSize(80);
    text('WELCOME',windowWidth*0.5,windowHeight*0.25);
    textSize(30);
    text('Move your Hand Up and Down while pressing the mouse to control the gear',windowWidth*0.5,windowHeight*0.4);
    text('Move your Hand left and right to control the steer',windowWidth*0.5,windowHeight*0.5);
    text('Click on the Q key to go to the Homepage',windowWidth*0.5,windowHeight*0.6);
    text('Press the space bar to connect to Arduino',windowWidth*0.5,windowHeight*0.7);
    text('GOOD LUCK!',windowWidth*0.5,windowHeight*0.8);
    fill(100);
    if(dist(mouseX,mouseY,windowWidth*0.5,windowHeight*0.9)<=30){
      fill(60);//create hover effect
    }
    if(dist(mouseX,mouseY,windowWidth*0.5,windowHeight*0.9)<=30&&mouseIsPressed){
      this.helpbool=false;//set boolean to close help page
    }
    circle(windowWidth*0.5,windowHeight*0.9,60);//these following code just creates the ok button
    fill(255,255,0);
    textSize(30);
    textFont(NORMAL);
    text("OK",windowWidth*0.5,windowHeight*0.9+10);
    
  }
}

Help Page:

This page is just to give the user info about the game and wish them luck as they embark on the Journey.

the code is found in the intro class and the page is controlled with the use of Boolean variables.

Dashboard:

Coming into the main page, the Dashboard shows the gearbox and the collision screen that alerts the user when the user is close to an obstacle.

class dash{//this class prints the dashboard, the gear and the detection screen
  constructor(dashY){
    this.Y=dashY;//the Y coordinate to which the dashboard is drawn
    this.movementFB=0;//the front and back movement counter
    this.gearY=0;//this controls the gearmovement
  }
  showdash(){
    push();//creating the dashboard
    strokeWeight(5)//set stroke weight to 5
    fill(193, 154, 107);//fill with brown
    beginShape();//we draw the dashboard
    curveVertex(0,this.Y);
    curveVertex(0,this.Y);
    curveVertex(windowWidth/2,this.Y-50);//creating the curve look
    curveVertex(windowWidth,this.Y);
    curveVertex(windowWidth,windowHeight);
    curveVertex(0,windowHeight);
    curveVertex(0,this.Y);
    curveVertex(0,this.Y);
    endShape();
    pop();
  }
  showgear(gearY){//this function shows the gear
    this.gearY=gearY;//capies value of the gearY from handpose
    rectMode(CENTER);//set rectange mode
    textSize(50);
    fill(0);//fill with black
    rect(windowWidth*0.5,this.Y+110,150,300,50);
    if(mouseIsPressed){//if the mouse is pressed means gear is being controlled
      if(this.gearY<windowHeight*0.35){
        this.movementFB=1;//set gear to drive
      }
      else if(this.gearY>windowHeight*0.5){
        this.movementFB=2;//set gear to reverse
      }
      else{
        this.movementFB=0;//set gear to P
      }
    }
    textAlign(CENTER);//align text to center
    textSize(50);
    fill(0,128,0);//fill with green
    switch(this.movementFB){
      case 0:
        text('P',windowWidth*0.5,this.Y+110);//p for parking
        break;
      case 1:
        text('D',windowWidth*0.5,this.Y+110);//D for drive
        break;
      case 2:
        text('R',windowWidth*0.5,this.Y+110);//R for reverse
        break;
    }
    textAlign(LEFT);
    textSize(12);
    return this.movementFB;//return the gear movement info 
  }
  showScreen(wallstop){//this shows the detection
    rectMode(CENTER);
    textSize(50);
    textAlign(CENTER);
    fill(0);
    rect(windowWidth*0.75,this.Y+110,500,300,50);//create screen
    fill(128,0,0);//fill the text with red
    if(wallstop==1){//if the front ultrasonic sensor is the one sensing
      text('OBJECT',windowWidth*0.75,this.Y+50);
      text('DETECTED',windowWidth*0.75,this.Y+130);
      text('AHEAD',windowWidth*0.75,this.Y+210);
    }
    else if(wallstop==2){//if the back ultrasonic is the one sensing
      text('OBJECT',windowWidth*0.75,this.Y+50);
      text('DETECTED',windowWidth*0.75,this.Y+130);
      text('BEHIND',windowWidth*0.75,this.Y+210);
    }
    else{
      fill(0,128,0);//fill the text with green
      text('No',windowWidth*0.75,this.Y+50);
      text('OBJECT',windowWidth*0.75,this.Y+130);
      text('DETECTED',windowWidth*0.75,this.Y+210);
    }
    textAlign(LEFT);
    textSize(12);
  }
}

In order to show the gear box info it gets info from the poseNet calculations done in the sketch and for the collision screen it uses info from the ultrasonic sensors.

The Steering wheel:

Though also part of the Dashboard, this is a separate class because it  controls the left and right movement using info from the other parts. That is, is uses info from the gear box and collision screen. It is not a picture downloaded from somewhere but a hand coded diagram.

class Steer{//this class creates the steer and controls the car movement front and back
  constructor(ctr,F2,X=windowWidth/3,Y=windowHeight/2){//takes the steerX,the font and the position for the steer to be placed
    this.angle=0;//equate the angle to 0
    this.X=X;//set the X and Y for the placement of the steering wheel
    this.Y=Y;
    this.ctr=ctr;
    this.movementLR=0;//create this Left right variable to store the movement
    this.F2=F2;
  }
  show(ctr){//takes steer x
    this.ctr=ctr;//update the ctr with steerX
  push();//designing the steering using stack so that it can be rotated entirely at once
  translate(this.X,this.Y);//making the orijin these
  rotate(this.angle);//causes the rotation
  noFill(0);
  strokeWeight(80);
  rectMode(CENTER);
  circle(0,0,500);
  fill(0);
  stroke(0);
  strokeWeight(12);
  beginShape();
  curveVertex(-230,-80);
  curveVertex(-230,-80);
  curveVertex(0,-120);
  curveVertex(230,-80);
  curveVertex(230,0);
  curveVertex(80,70);
  curveVertex(40,230);
  curveVertex(-40,230);
  curveVertex(-80,70);
  curveVertex(-230,0);
  curveVertex(-230,-80);
  curveVertex(-230,-80);
  fill(0);
  endShape();
  fill(100);
    noStroke()
  textSize(50)
    textAlign(CENTER);
  textFont(this.F2);
  text('REKAS',0,0);//steering wheel/car brand
  pop();
    if(this.ctr>0&&this.ctr<windowWidth){//while the value is within our range
  this.angle=map(this.ctr,0,windowWidth,-PI/2,PI/2);//update angle based on this
  }
  }
  steerTurn(movementFB,wallstop){//this function controls the turning of the steer
    if(this.angle<-PI/7.5&&movementFB==1&&wallstop!=1){
        this.movementLR=4;//front left
      }
      else if(this.angle<-PI/7.5&&movementFB==2&&wallstop!=2){
        this.movementLR=6;//back left
      }
      else if(this.angle>=-PI/7.5&&this.angle<-PI/8){
        this.movementLR=0;//to prevent bugs 
      }
      else if(this.angle>=-PI/7&&this.angle<PI/8&&movementFB==1&&wallstop!=1){
        this.movementLR=1;//move straight ahead
      }
      else if(this.angle>=-PI/7&&this.angle<PI/7&&movementFB==2&&wallstop!=2){
        this.movementLR=2;//reverse
      }
      else if(this.angle>=PI/7&&this.angle<PI/5){
        this.movementLR=0;//yeah
      }
      else if(this.angle>=PI/5&&movementFB==1&&wallstop!=1){
        this.movementLR=3;//front right
      }
      else if(this.angle>=PI/5&&movementFB==2&&wallstop!=2){
        this.movementLR=5;//back right
      }
      else{
        this.movementLR=0;//dont move
      }
    return this.movementLR;
  }
}

Sketch:

This is the last part that kind of sums everything up. This is where all the other components come together to produce this artwork. This is also where the poseNet functions and full screen Functions are declared.

let mySteer;//variable going to store steer object
let myDash;//gonna store the dashboard object
let handpose;//gonna store posenet object
let video;//gonna store video object
let predictions = [];//gonna store set of predictions from posenet
let steerX=300;//gonna store the average x coordinate of the hand
let gearY=250;//gonna store the average y coordinate of the hand
let movementLR=0;//gonnna store the left and right movement
let movementFB=0;//gonna store the front and back movement

let wallstop=0;//gonna store the obstacle detection
let steercontrol=0;//gonna control steer to make it feel smooth

let IntroBackground;//intropage background
let S1;//sound 1(button)
let S2;//sound 2
let F1;
let F2;
let introp;//gonna store intropage object
let introbool=true;//going to control the intropage display
let gamebool=false;//going to control the mainpage display

function preload(){
  //in this preload function we will load all the uploads we need before we even start the game.
  IntroBackground=loadImage("intro1.jpg");//this is for the background
  S1=loadSound("button.mp3");//these sets are for the sounds
  S2=loadSound("msound.mp3");
  F1=loadFont("font1.ttf");//these sets are for the fonts 
  F2=loadFont("font2.ttf");
  
}

function setup() {
  createCanvas(windowWidth, windowHeight);
  video = createCapture(VIDEO);//capture video using camera
  video.size(width, height);//set the size of the video to that of the screen
   steercontrol=windowWidth/2//set the steercontrol for smoothness
  handpose = ml5.handpose(video);//get posenet from the video feed using the ml library

  // This sets up an event that fills the global variable "predictions"
  // with an array every time new hand poses are detected
  handpose.on("predict", results => {
    predictions = results;
  });

  // Hide the video element, and just show the canvas
  video.hide();
  introp=new ipage(IntroBackground,S1,S2,F1,F2);//create intropage object
  mySteer=new Steer(steerX,F2,windowWidth*0.5,windowHeight*0.85);//create steer object
  myDash=new dash(windowHeight*0.85);//create dashboard object
  S2.loop();//start playing the sound but with loop property
}

function draw() {
  if(introbool){//if the introbool is true show intropage
    gamebool=introp.show();//update gamebool from intro.show function
    if(gamebool){introbool=false;}//if the gamebool is true,set intro to false
  }
  else{
    background(50,150,255);
  myDash.showdash()//show the dash
  drawKeypoints();//call this function for geting info from the video hand detection
  movementFB=myDash.showgear(gearY);//update the front back movement from the showgear function
  movementLR=mySteer.steerTurn(movementFB,wallstop);//update the leftright movement from the steerturn function
  if(steerX>0&&steerX<windowWidth){//if the steerX is within the range we want
    mySteer.show(steerX);//show the steer with this value
    steercontrol=steerX;//update the steercontrol incase we stop getting data
  }
  else{//if the steerX is not in our range,
    if(steercontrol<windowWidth/2-5){steercontrol+=10;}//using our steercontrol,slowly move the steer to the center
    else if(steercontrol>windowWidth/2+5){steercontrol-=10;}
    mySteer.show(steercontrol);
  }
  myDash.showScreen(wallstop);//show the screen with the wallstop getten form the Arduino
  }
}
function keyPressed() {//if spaebar is pressed connect to arduino
  if (key == " ") {
    // important to have in order to start the serial connection!!
    setUpSerial();//connect to arduino
  }
}
function readSerial(data) {
  

  if (data!=null){//if the data is not null
    //////////////////////////////////
    //READ FROM ARDUINO HERE (handshake)
    //////////////////////////////////
    wallstop= int(trim(data));
    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    let sendToArduino = movementLR + "\n";
    writeSerial(sendToArduino);
  }
    
}
function drawKeypoints() {
  let totalX=0;//set variable to store the sum of the x coordinates of all the predictions
  let totalY=0;//same for y
  let avgctr=0;//set a counter to count the predictions
  let len=0;//I dont use len here but i was experimenting somthing
  for (let i = 0; i < predictions.length; i += 1) {
    const prediction = predictions[i];
    len=predictions.lenght*prediction.landmarks.length;
    for (let j = 0; j < prediction.landmarks.length; j += 1) {
      const keypoint = prediction.landmarks[j];
       totalX+=windowWidth-map(keypoint[0],0,video.width,0,windowWidth);//map the points to our window size and sum it
      totalY+=map(keypoint[1],0,video.height,0,windowHeight);
      avgctr++;//increase this too
    }
  }
  steerX=totalX/avgctr;//update steerX with the average of X
  gearY=totalY/avgctr;//same for Y but with average of Y
}
function windowResized() {
  resizeCanvas(windowWidth, windowHeight);//resize the canvas to go to fullpage
}

function keyTyped() {
  // $$$ For some reason on Chrome/Mac you may have to press f twice to toggle. Works correctly on Firefox/Mac
  if (key === 'f') {
    toggleFullscreen();//if f is pressed, show fullpage
  }
  if(key==='q'){//if q is pressed go to h=intro page
    if(introbool==false){
      introbool=true;
      introp.playbool=false;
      S2.play();
    }
  }
  // uncomment to prevent any default behavior
  // return false;
}

// Toggle fullscreen state. Must be called in response
// to a user event (i.e. keyboard, mouse click)
function toggleFullscreen() {
  let fs = fullscreen(); // Get the current state
  fullscreen(!fs); // Flip it!
}

Rekas Bot:

My inspiration for this project comes from my zeal to use the ml library in p5 and I had fun developing this.

Model:

Test Video:

Arduino Circuit Diagram:

Schematic Diagram:

Clarification on the Schematic motor drawing:

By Aya Riad

Future Improvements:

I Tried to implement Bluetooth but I was not able to because it kept on failing even though I had connected and set up everything right so I wish to be able to find a way to make this connection more reliable and also I plan to make the machine learning more reliable in future.

IM Showcase:

So just before the showcase started, I added one line of code which made the bot move randomly as though it had life.

if(introbool){//if the introbool is true show intropage
    gamebool=introp.show();//update gamebool from intro.show function
    movementLR=int(random(0,6));//do random stuff
    if(gamebool){introbool=false;}//if the gamebool is true,set intro to false
  }

I am really proud of this last decision.

and when you  want to drive it, it stops the random movements.

For more videos

Thank you.

Blind User Testing

First things first:

Concept

I initially planned on extending the functionality of my midterm project through the final project by giving it physical controls using the arduino. However, in the end, I decided to make a game inspired by all the amazing games that I saw in the midterms and final proposals (call it FOMO, I guess 😉 ). I settled with a game we all know and love, the Snake game that I’m sure most of us remember playing on our parents’ Nokias and BlackBerries. You eat fruits, you avoid hitting the boundaries and you avoid eating yourself – pretty simple, right? I settled for a simple aesthetic, making it pixel-y for the whole nostalgia package. Instead of using buttons on our parents’ phones, I’ve made it an arcade game where the snake is controlled by a joystick!

Basically, the game begins by pressing the joystick. For every fruit eaten, the player’s score increments by 1. After every 5 fruits eaten, if the player is still alive, the player levels up and the speed increases until you get to 30 scores after which it remains at the highest speed. After eating 20 fruits, your habitat (background) changes. There’s fun sounds and Minecraft music playing in the background, so the player will never get bored! The game is competitive too; it keeps track of the highest score for all turns!

Visuals

Implementation

I basically coded the game in p5.js initially, with an intention to control the snake with push buttons. After struggling with the implementation and orientation on the small breadboard, I thought about using the big LED buttons in a box, but when I saw Khaleeqa’s project using a joystick, I thought to myself: We can do that??? (thank you for the idea, Khaleeqa. I pray you’re able to do the Tolerance readings and response on time!). So now the snake is controlled by a joystick, adding to the nostalgic arcade feel. The design is extremely user friendly; you couldn’t get stuck or confused if you tried.

Arduino Code and Description of Communication

Basically, the joystick uses two potentiometers on the x and y axes, and I sent these two values to my p5.js sketch to control the direction of the snake’s movement. Pressing the joystick gives a specific value on the x potentiometer (1023) which I use to restart the game (thank you again Professor Aya for lending me a joystick, real lifesaver here).

Arduino Schematic

Arduino Code

int xValue = 0 ;
int yValue = 0 ; 

void setup()	
{	
    Serial.begin(9600) ;

}	

void loop()	
{	
    xValue = analogRead(A2);	
    yValue = analogRead(A1);		
    Serial.print(xValue,DEC);
    Serial.print(",");
    Serial.println(yValue,DEC);

    delay(100);	
}

P5 Code and Sketch

The P5 code is extensive in functionality, but I’ve tried to make the code as short as possible to make up for the 800 line monster I created for my midterm. I’ve basically used 4 classes in the implementation for each element of the snake, the snake as a whole, fruits, and the gameplay. The snake class is a list of snake elements, so it inherits the list class. The various classes, methods and logic are explained in the sketch comments, so instead of describing it again, here is the code:

let RESOLUTION = 500; // Setting canvas resolution
let ROWS = 20; // Number of rows in the grid
let COLS = 20; // Number of columns in the grid
let WIDTH = RESOLUTION / ROWS; // Width of each grid cell
let HEIGHT = RESOLUTION / COLS; // Height of each grid cell
let end_flag = false; // Flag to indicate the end of the game
let start_flag = false;

let bg_flag = 0;
let xVal = 0;
let yVal = 0;
let high_score = 0;

let head_up,
  head_left,
  apple,
  banana,
  game_background1,
  gamebackground2,
  game_font,
  bite_sound,
  level_up,
  game_over,
  background_music; // Loading images for background, snake head, apple, and banana; fonts; sounds

class SnakeElement {
  constructor(x, y, element_num) {
    // Snake element constructor
    this.x = x; // X-coordinate of the element
    this.y = y; // Y-coordinate of the element
    this.element_num = element_num; // Identifier for the type of element
  }

  display() {
    if (this.element_num === 1) {
      // Displaying the head facing up
      noFill();
      stroke(250);
      strokeWeight(2);
      ellipse(this.x + WIDTH / 2, this.y + HEIGHT / 2, WIDTH + 2);
      image(head_up, this.x, this.y, WIDTH, HEIGHT);
    } else if (this.element_num === 2) {
      // Displaying the head facing down (flipped vertically)
      noFill();
      stroke(250);
      strokeWeight(2);
      ellipse(this.x + WIDTH / 2, this.y + HEIGHT / 2, WIDTH + 2);
      push(); // Save the current drawing state
      scale(1, -1); // Flip vertically
      image(head_up, this.x, -this.y - HEIGHT, WIDTH, HEIGHT);
      pop(); // Restore the original drawing state
    } else if (this.element_num === 3) {
      // Displaying the head facing left
      noFill();
      stroke(250);
      strokeWeight(2);
      ellipse(this.x + WIDTH / 2, this.y + HEIGHT / 2, WIDTH + 2);
      image(head_left, this.x, this.y, WIDTH, HEIGHT);
    } else if (this.element_num === 4) {
      // Displaying the head facing right (flipped horizontally)
      noFill();
      stroke(250);
      strokeWeight(2);
      ellipse(this.x + WIDTH / 2, this.y + HEIGHT / 2, WIDTH + 2);
      push(); // Save the current drawing state
      scale(-1, 1); // Flip horizontally
      image(head_left, -this.x - WIDTH, this.y, WIDTH, HEIGHT);
      pop(); // Restore the original drawing state
    } else {
      // Displaying a circle for the body elements
      stroke(250);
      strokeWeight(2);
      if (this.element_num === 5) {
        fill(120, 220, 20); // Green circle
      } else if (this.element_num === 6) {
        fill(200, 48, 32); // Red circle
      } else if (this.element_num === 7) {
        fill(251, 240, 76); // Yellow circle
      }
      ellipse(this.x + WIDTH / 2, this.y + HEIGHT / 2, WIDTH);
    }
  }
}

class Snake extends Array {
  constructor() {
    super();
    // Initializing the snake with head and initial body elements
    this.push(new SnakeElement(RESOLUTION / 2, RESOLUTION / 2, 1));
    this.push(new SnakeElement(RESOLUTION / 2 - WIDTH, RESOLUTION / 2, 5));
    this.push(new SnakeElement(RESOLUTION / 2 - WIDTH * 2, RESOLUTION / 2, 5));
    this.full_flag = false; // Flag to check if the snake has filled the grid
  }

  display() {
    // Displaying all snake elements
    for (let element of this) {
      element.display();
    }
  }

  move(current_dir) {
    // Controlling the movement of the snake
    let head_x = this[0].x;
    let head_y = this[0].y;

    // Updating head position based on the current direction
    if (current_dir === "UP") {
      this[0].element_num = 1; // Updating element_num for head facing up
      this[0].y -= WIDTH; // Moving up
    } else if (current_dir === "DOWN") {
      this[0].element_num = 2; // Updating element_num for head facing down
      this[0].y += WIDTH; // Moving down
    } else if (current_dir === "LEFT") {
      this[0].element_num = 3; // Updating element_num for head facing left
      this[0].x -= WIDTH; // Moving left
    } else if (current_dir === "RIGHT") {
      this[0].element_num = 4; // Updating element_num for head facing right
      this[0].x += WIDTH; // Moving right
    }

    // Moving the body elements
    for (let i = 1; i < this.length; i++) {
      let temp_x = this[i].x;
      let temp_y = this[i].y;
      this[i].x = head_x;
      this[i].y = head_y;
      head_x = temp_x;
      head_y = temp_y;
    }
  }

  collide_self() {
    // Checking if the snake collides with itself
    for (let i = 1; i < this.length; i++) {
      if (this[0].x === this[i].x && this[0].y === this[i].y) {
        end_flag = true; // Collision occurred, end the game
        game_over.play();
      }
    }
  }

  collide_walls() {
    // Checking if the snake collides with the canvas borders
    if (
      this[0].x >= RESOLUTION ||
      this[0].x < 0 ||
      this[0].y < 0 ||
      this[0].y >= RESOLUTION
    ) {
      end_flag = true; // Snake has left the canvas, end the game
      game_over.play();
    }
  }

  board_full() {
    // Checking if the snake has filled the entire grid
    if (this.length === ROWS * COLS) {
      end_flag = true; // Board is full, end the game
      this.full_flag = true; // Player wins
    }
  }
}

class Fruit {
  constructor() {
    // Generating a random position for the fruit
    this.x = Math.floor(Math.random() * ROWS) * WIDTH;
    this.y = Math.floor(Math.random() * COLS) * HEIGHT;
    this.fruit_num = Math.floor(Math.random() * 2); // Randomly choosing apple or banana
  }

  display() {
    // Displaying the fruit based on its type
    if (this.fruit_num === 0) {
      image(apple, this.x, this.y, WIDTH, HEIGHT);
    } else {
      image(banana, this.x, this.y, WIDTH, HEIGHT);
    }
  }
}

class Game {
  constructor() {
    // Initializing the game with snake, fruit, and default direction
    this.snake = new Snake();
    this.fruit = new Fruit();
    this.current_dir = "RIGHT";
    this.score = 0; // Player's score
    this.frames = 12;
    this.eat_count = 0;
  }

  display() {
    // Displaying the snake, checking for fruit collision, and displaying the score
    this.snake.display();
    let n = 0;
    while (n < this.snake.length) {
      // Checking if the fruit is at the same position as any snake element
      if (
        this.fruit.x === this.snake[n].x &&
        this.fruit.y === this.snake[n].y
      ) {
        this.fruit = new Fruit(); // Create a new fruit
      }
      n++;
    }
    this.fruit.display();
    textSize(10);
    fill(0);
    text("Score: " + this.score, RESOLUTION - 100, 30);
  }

  move() {
    // Moving the snake, checking for collisions
    this.snake.move(this.current_dir);
    this.snake.collide_self();
    this.snake.collide_walls();
  }

  eat() {
    // Checking if the snake eats the fruit
    if (this.snake[0].x === this.fruit.x && this.snake[0].y === this.fruit.y) {
      // Adding a new element to the snake based on the current direction
      if (this.current_dir === "DOWN") {
        this.snake.push(
          new SnakeElement(
            this.snake[this.snake.length - 1].x,
            this.snake[this.snake.length - 1].y - HEIGHT,
            6 + this.fruit.fruit_num
          )
        );
      }
      if (this.current_dir === "UP") {
        this.snake.push(
          new SnakeElement(
            this.snake[this.snake.length - 1].x,
            this.snake[this.snake.length - 1].y + HEIGHT,
            6 + this.fruit.fruit_num
          )
        );
      }
      if (this.current_dir === "LEFT") {
        this.snake.push(
          new SnakeElement(
            this.snake[this.snake.length - 1].x + WIDTH,
            this.snake[this.snake.length - 1].y,
            6 + this.fruit.fruit_num
          )
        );
      }
      if (this.current_dir === "RIGHT") {
        this.snake.push(
          new SnakeElement(
            this.snake[this.snake.length - 1].x - WIDTH,
            this.snake[this.snake.length - 1].y,
            6 + this.fruit.fruit_num
          )
        );
      }
      this.fruit = new Fruit(); // Create a new fruit
      this.score += 1; // Increase the score
      if (this.score > 20) {
        bg_flag = 1;
      }
      frames -= 1;
      this.eat_count += 1;

      if (this.eat_count % 5 === 0 && this.eat_count <= 30) {
        this.frames -= 1;
        level_up.play();
      }
      bite_sound.play();
    }
  }

  rungame() {
    // Main method to be called in draw()
    if (frameCount % this.frames === 0) {
      if (!end_flag) {
        // If the game hasn't ended
        if (bg_flag === 0) {
          image(game_background1, 0, 0, RESOLUTION, RESOLUTION);
        } else if (bg_flag === 1) {
          image(game_background2, 0, 0, RESOLUTION, RESOLUTION);
        }
        this.display();
        this.move();
        this.eat();
      } else if (!this.snake.full_flag) {
        // If the game has ended and the board is not full
        if (bg_flag === 0) {
          image(game_background1, 0, 0, RESOLUTION, RESOLUTION);
        } else if (bg_flag === 1) {
          image(game_background2, 0, 0, RESOLUTION, RESOLUTION);
        }
        textSize(30);
        text("Game Over :(", RESOLUTION / 2 - 175, RESOLUTION / 2 - 15);
        textSize(18);
        text(
          "Final Score: " + this.score,
          RESOLUTION / 2 - 120,
          RESOLUTION / 2 + 15
        );
        textSize(10);
        text(
          "Click anywhere to restart :D",
          RESOLUTION / 2 - 130,
          RESOLUTION - 40
        );
        if (this.score > high_score) {
          high_score = this.score;
        }
        push();
        textSize(18);
        fill(255);
        stroke(0);
        text(
          "High Score: " + high_score,
          RESOLUTION / 2 - 110,
          RESOLUTION / 2 + 40
        );
        pop();
      } else {
        // If the game has ended and the board is full
        if (bg_flag === 0) {
          image(game_background1, 0, 0, RESOLUTION, RESOLUTION);
        } else if (bg_flag === 1) {
          image(game_background2, 0, 0, RESOLUTION, RESOLUTION);
        }
        textSize(25);
        text("You Win :D", RESOLUTION / 2 - 140, RESOLUTION / 2);
        textSize(7.5);
        text(
          "Click anywhere to restart :D",
          RESOLUTION / 2 - 100,
          RESOLUTION - 50
        );
      }
    }
  }
}

let game;

function preload() {
  // Loading images before setup()
  head_up = loadImage("images/head_up.png");
  head_left = loadImage("images/head_left.png");
  apple = loadImage("images/apple.png");
  banana = loadImage("images/banana.png");
  game_background1 = loadImage("images/snake_game_background1.png");
  game_background2 = loadImage("images/snake_game_background2.png");
  game_font = loadFont("fonts/snake_game_font.ttf");
  bite_sound = loadSound("sounds/bite.m4a");
  level_up = loadSound("sounds/levelup.mp3");
  game_over = loadSound("sounds/gameover.mp3");
  background_music = loadSound("sounds/backgroundmusic.m4a");
}

function setup() {
  // Setup function for creating the canvas and initializing the game
  createCanvas(RESOLUTION, RESOLUTION);
  game = new Game();
  textFont(game_font);
}

function draw() {
  // Draw function for running the game
  if (xVal >= 1000) {
    start_flag = true;
  }
  if (!background_music.isPlaying()) {
    background_music.play();
  }
  if (start_flag === true) {
    game.rungame();
  } else {
    image(game_background1, 0, 0, RESOLUTION, RESOLUTION);
    push();
    textSize(50);
    text("SNAKE", RESOLUTION / 2 - 120, RESOLUTION / 2 + 20);
    textSize(15);
    strokeWeight(1.5);
    text(
      "the most original game ever",
      RESOLUTION / 2 - 200,
      RESOLUTION / 2 + 40
    );
    pop();
  }

  if (!serialActive) {
    stroke(255);
    strokeWeight(2);
    text("Press Space Bar to select Serial Port", 20, 30);
  } else if (start_flag === true) {
    text("Connected", 20, 30);
  } else {
    text("Connected. Press the joystick to \nstart playing!", 20, 30);
  }

  // changing the direction of the snake using values from arduino
  if (xVal < 300) {
    // joystick moved left
    if (game.current_dir !== "RIGHT") {
      game.current_dir = "LEFT";
    }
  } else if (xVal > 700 && xVal < 1000) {
    // joystick moved right
    if (game.current_dir !== "LEFT") {
      game.current_dir = "RIGHT";
    }
  } else if (yVal < 300) {
    // joystick moved down
    if (game.current_dir !== "UP") {
      game.current_dir = "DOWN";
    }
  } else if (yVal > 700) {
    if (game.current_dir !== "DOWN") {
      // joystick moved up
      game.current_dir = "UP";
    }
  } else if (xVal >= 1000) {
    // restart game when joystick pressed
    if (end_flag === true) {
      end_flag = false;
      game = new Game();
      bg_flag = 0;
    }
  }
}

function mousePressed() {
  // Restart the game on mouse click
  if (end_flag === true) {
    start_flag = true;
    end_flag = false;
    game = new Game();
    bg_flag = 0;
  }
}

function keyPressed() {
  // Change the direction of movement on arrow key press
  if (keyCode === LEFT_ARROW) {
    if (game.current_dir !== "RIGHT") {
      game.current_dir = "LEFT";
    }
  } else if (keyCode === RIGHT_ARROW) {
    if (game.current_dir !== "LEFT") {
      game.current_dir = "RIGHT";
    }
  } else if (keyCode === UP_ARROW) {
    if (game.current_dir !== "DOWN") {
      game.current_dir = "UP";
    }
  } else if (keyCode === DOWN_ARROW) {
    if (game.current_dir !== "UP") {
      game.current_dir = "DOWN";
    }
  }

  if (key == " ") {
    // important to have in order to start the serial connection!!
    setUpSerial();
  }
}

// This function will be called by the web-serial library
// with each new *line* of data. The serial library reads
// the data until the newline and then gives it to us through
// this callback function
function readSerial(data) {
  ////////////////////////////////////
  //READ FROM ARDUINO HERE
  ////////////////////////////////////
  if (data != null) {
    // make sure there is actually a message
    // split the message
    let fromArduino = split(trim(data), ",");
    // if the right length, then proceed
    if (fromArduino.length == 2) {
      // only store values here
      // do everything with those values in the main draw loop
      xVal = int(fromArduino[0]);
      yVal = int(fromArduino[1]);
    }
  }
}

And a link to the sketch:

https://editor.p5js.org/haroonshafi/full/M1HJeVoiy

Aspects That I’m (IM) Proud Of

Actually getting the Snake game to work was a big part of this project. There’s so many conditions to cater to, so much stuff to consider. You don’t think that while you’re on the playing end but boy when you’re on the making end – whew. I’m proud that after messing up a bajillion times, particularly with the serial communication part, that I’ve finally got this to work with the help of the professor.  (I’m also proud of getting a high score of 34, so I wanna see someone beat that lol).

Areas for Future Improvement

There is plenty of room for improvement. Had I had more time, I would have incorporated some lights for indicating when a fruit is eaten or when the snake collides so there would be communication from p5 to arduino as well. I also wanted to use a potentiometer to control the speed of the snake through the framerate, because that would be fun (read: chaotic).

https://editor.p5js.org/Minjae0216/sketches/Qyv31CZ2_

Concept:

Through “Pet the Plant”, I wanted to make an interactive project that allows you to experience the satisfaction of growing your plant friend without the real responsibilities or risks. In this game, there are three different plants that you can choose to grow and make your own Christmas tree.

// Arduino code

// Input:
// A0 - photocell sensor
// A1 - force resistor sensor
// A2 - soil moisture sensor

int photocellPin = A0; //photocell sensor
int forceSensorPin = A1; //force resistor sensor
int soil = A2; //soil moisture sensor
int forceSensorValue = 0;

void setup() {
  Serial.begin(9600);
  pinMode(photocellPin, INPUT);
  pinMode(forceSensorPin,INPUT);
  pinMode(soil, INPUT);
}

void loop() {
  // Read the value from three sensors
  int lightValue = analogRead(photocellPin);
  int forceSensorValue = analogRead(forceSensorPin);
  int soilValue = analogRead(soil);

  // Send the value to p5.js
    Serial.print(soilValue);
    Serial.print(",");
    Serial.print(lightValue);
    Serial.print(",");
    Serial.print(forceSensorValue);
    Serial.println(); // Print a newline to indicate the end of the data

// Define global variables for images, page tracking, and sensor data
let backgroundImage;
let choicebackgroundImage;
let resultbackgroundImage;
let merrychristmasImage;
let cactusImage;
let greenplantImage;
let palmtreeImage;
let currentPage = 1;
let serial;
let latestData = "waiting for data";
let plantSize = 80;
let plantX, plantY;
let xOffset = 0;
let yOffset = 0;
let userChoice = ''

//Sensor Values
let soilMoistureValue = 0;
let photocellValue = 0;
let forceSensorValue = 0;

//Ornament Variables
let ornaments = ['https://i.imgur.com/FPuVyGI.png', 'https://i.imgur.com/fJxVZe3.png', 'https://i.imgur.com/rGZS7Yx.png']; 
let currentOrnament = '';
let ornamentsList = [] ;
let hasOrnament = false;
let isAddingOrnament = false;


//Preload images before setup
function preload() {
  //loadimages
  backgroundImage = loadImage('https://i.imgur.com/lGiEbaQ.png');
  choicebackgroundImage = loadImage('https://i.imgur.com/tmRSqeN.png')
  cactusImage = loadImage('https://i.imgur.com/kIxRg6H.png')
  palmtreeImage = loadImage('https://i.imgur.com/A8tjpQM.png')
  greenplantImage = loadImage('https://i.imgur.com/yMYwr8D.png'); 
  resultbackgroundImage = loadImage('https://i.imgur.com/1IHM8SH.png')
  merrychristmasImage = loadImage ('https://i.imgur.com/MoUEmwD.png')
  
  //Load Ornament Images
   for (let i = 0; i < ornaments.length; i++) {
    ornaments[i] = loadImage(ornaments[i]);
  }
}

// Setup function, called once at the beginning
function setup() {
  // Create canvas fit to the background image
  createCanvas(700,500);
    // Connect to the p5.serialport server
  serial = new p5.SerialPort();
 // Set up serial communication
 serial.list();
 serial.open('/dev/tty.usbmodem101');

 serial.on('connected', serverConnected);

 serial.on('list', gotList);

 serial.on('data', gotData);

 serial.on('error', gotError);

 serial.on('open', gotOpen);

 serial.on('close', gotClose);
 
}

// Callback function when connected to the serial server
function serverConnected() {
 print("Connected to Server");
}

// Callback function when a list of serial ports is received
function gotList(thelist) {
 print("List of Serial Ports:");

 for (let i = 0; i < thelist.length; i++) {
  print(i + " " + thelist[i]);
 }
}

// Callback function when the serial port is opened
function gotOpen() {
 print("Serial Port is Open");
}

// Callback function when the serial port is closed
function gotClose(){
 print("Serial Port is Closed");
 latestData = "Serial Port is Closed";
}

// Callback function when an error occurs in serial communication
function gotError(theerror) {
 print(theerror);
}

// Callback function when new data is received from the serial port
function gotData() {
 
  let currentString = serial.readLine();
  console.log('serial received: ' + currentString);
  trim(currentString);
  if (!currentString) return;
  console.log(currentString);
  latestData = currentString;

  // Split the received data into values
  let dataValues = split(latestData, ',');

  // Assign the values to the corresponding variables
  console.log(dataValues);
  if (dataValues.length === 3) {
    console.log('got 3 values');
    soilMoistureValue = int(dataValues[0]);
    console.log('soil: ' + soilMoistureValue);
    photocellValue = int(dataValues[1]);
    console.log('photocell: ' + photocellValue);
    forceSensorValue = int(dataValues[2]);
    console.log('force: ' + forceSensorValue);
  
}

}


//fullscreen
function keyPressed() {
  if (key === 'f') {
    toggleFullscreen();
  }
}

// Draw function, called continuously
function draw() {
  if (currentPage === 1) {
    drawStartPage();
  } else if (currentPage === 2) {
    drawChoicePage();
  } else if (currentPage === 3) {
    drawResultPage();
  }

}

// Draw function for the start page
function drawStartPage() {
  background(backgroundImage);

  if (mouseIsPressed) {
    currentPage = 2; // Move to the next page when clicked
  }
}

// Draw function for the choice page (next page)
function drawChoicePage() {
  background(choicebackgroundImage);

  textSize(24);
  fill(0);
  

  if (keyIsPressed) {
    // Check the pressed key and move to the corresponding page
    if (key === '1') {
      userChoice = 'cactus';
      currentPage = 3; // Go to the third page with cactus
    } else if (key === '2') {
      userChoice = 'palmTree';
      currentPage = 3; // Go to the third page with green plant
    } else if (key === '3') {
      userChoice = 'greenPlant';
      currentPage = 3; // Go to the third page with palm tree
    }
  }
}


// Draw function for the result page (last page)
function drawResultPage() {
  background(resultbackgroundImage);

   // Display selected plant based on user choice
  if (userChoice === 'cactus') {
    plantX = 485 + xOffset;
    plantY = 337 + yOffset;
    image(cactusImage, plantX, plantY, plantSize, plantSize);
  } else if (userChoice === 'palmTree') {
    plantX = 489 + xOffset;
    plantY = 341 + yOffset;
    image(palmtreeImage, plantX, plantY, plantSize, plantSize);
  } else if (userChoice === 'greenPlant') {
    plantX = 500 + xOffset;
    plantY = 339 + yOffset;
    image(greenplantImage, plantX, plantY, plantSize, plantSize);
  }

  // Check sensor values and simulate plant growth for all plant types
  if (userChoice === 'cactus' || userChoice === 'palmTree' || userChoice === 'greenPlant') {
    // Check soil moisture sensor value
    if (soilMoistureValue > 860) {
      // Increase size when soil moisture is high
      plantSize += 1.5;
      xOffset -= 0.8;
      yOffset -= 1.4;
    }

    // Check photocell sensor value
    if (photocellValue > 70) {
      // Increase size when light is high
      plantSize += 1.5;
      xOffset -= 0.8;
      yOffset -= 1.4;
    }

    // Check force sensor value
    if (forceSensorValue > 1000) {
      // Add ornament when force sensor value is high
      isAddingOrnament = true;
    }
  }

  // Display ornaments
  for (let i = 0; i < ornamentsList.length; i++) {
    let ornament = ornamentsList[i];
    image(ornament.image, ornament.x, ornament.y, ornament.size, ornament.size);
  }

  // If the photocell value is less than 10, switch to a Christmas background
  if (photocellValue < 10) {
    background(merrychristmasImage);
    if (userChoice === 'cactus') {
      plantX = 485 + xOffset;
      plantY = 337 + yOffset;
      image(cactusImage, plantX, plantY, plantSize, plantSize);
      
      // Display the selected plant based on user choice along with ornaments
      if (ornamentsList[0]) {
        for (let i = 0; i < ornamentsList.length; i++) {
          let ornament = ornamentsList[i];
          image(ornament.image, ornament.x, ornament.y, ornament.size, ornament.size);
        }
      }
  } else if (userChoice === 'palmTree') {
      plantX = 489 + xOffset;
      plantY = 341 + yOffset;
      image(palmtreeImage, plantX, plantY, plantSize, plantSize);
    
      if (ornamentsList[0]) {
        for (let i = 0; i < ornamentsList.length; i++) {
          let ornament = ornamentsList[i];
          image(ornament.image, ornament.x, ornament.y, ornament.size, ornament.size);
        }
      }
  } else if (userChoice === 'greenPlant') {
      plantX = 500 + xOffset;
      plantY = 339 + yOffset;
      image(greenplantImage, plantX, plantY, plantSize, plantSize);
    
      if (ornamentsList[0]) {
        for (let i = 0; i < ornamentsList.length; i++) {
          let ornament = ornamentsList[i];
          image(ornament.image, ornament.x, ornament.y, ornament.size, ornament.size);
        }
      }
  }
    
  }
  
}
// MousePressed function checks if the user is ready to add an ornament and if the click is within the plant area
function mousePressed() {
   // Add ornament at the clicked position
  if (isAddingOrnament && mouseX > plantX && mouseX < plantX + plantSize && mouseY > plantY && mouseY < plantY + plantSize) {
    // Add ornament at the clicked position
    let ornament = {
      image: random(ornaments),
      x: mouseX,
      y: mouseY,
      size: 40, // Adjust the size of the ornament as needed
    };

    ornamentsList.push(ornament);

    // Reset the flag
    isAddingOrnament = false;
  }
}

I’ve created a p5.js sketch that brings a virtual Christmas tree to life, responding to Arduino sensors and user interactions. The code begins by initializing global variables, including images for backgrounds, plants, and ornaments, as well as parameters for page tracking and sensor data.

In the setup function, the canvas is created, and a connection to the p5.serialport server is established for communication with an Arduino. Callback functions are set up to handle various serial events, ensuring a smooth connection between the digital representation and physical sensor inputs.

The draw function renders different pages based on user choices and sensor inputs. Users progress through pages by clicking the mouse or pressing keys to select the type of plant displayed on the last page.

The drawResultPage function, specific to the final page, displays the chosen plant with dynamic growth simulation based on soil moisture, light intensity, and force sensor values. The plant size adjusts, and ornaments are added under specific conditions, creating an interactive and festive atmosphere. The code also intelligently switches to a Christmas-themed background in low-light conditions, enhancing the holiday spirit.

My code handles user interactions, allowing users to enter fullscreen mode with the ‘f’ key and add ornaments to the virtual tree by clicking on it. The integration of visual elements, sensor data, and user interactions results in an engaging experience, turning virtual festivities into a tangible and interactive celebration of the Christmas season.

<Communication between Arduino and p5.js>

function drawResultPage() {
  background(resultbackgroundImage);

   // Display selected plant based on user choice
  if (userChoice === 'cactus') {
    plantX = 485 + xOffset;
    plantY = 337 + yOffset;
    image(cactusImage, plantX, plantY, plantSize, plantSize);
  } else if (userChoice === 'palmTree') {
    plantX = 489 + xOffset;
    plantY = 341 + yOffset;
    image(palmtreeImage, plantX, plantY, plantSize, plantSize);
  } else if (userChoice === 'greenPlant') {
    plantX = 500 + xOffset;
    plantY = 339 + yOffset;
    image(greenplantImage, plantX, plantY, plantSize, plantSize);
  }

  // Check sensor values and simulate plant growth for all plant types
  if (userChoice === 'cactus' || userChoice === 'palmTree' || userChoice === 'greenPlant') {
    // Check soil moisture sensor value
    if (soilMoistureValue > 860) {
      // Increase size when soil moisture is high
      plantSize += 1.5;
      xOffset -= 0.8;
      yOffset -= 1.4;
    }

    // Check photocell sensor value
    if (photocellValue > 70) {
      // Increase size when light is high
      plantSize += 1.5;
      xOffset -= 0.8;
      yOffset -= 1.4;
    }

    // Check force sensor value
    if (forceSensorValue > 1000) {
      // Add ornament when force sensor value is high
      isAddingOrnament = true;
    }
  }

  // Display ornaments
  for (let i = 0; i < ornamentsList.length; i++) {
    let ornament = ornamentsList[i];
    image(ornament.image, ornament.x, ornament.y, ornament.size, ornament.size);
  }

  // If the photocell value is less than 10, switch to a Christmas background
  if (photocellValue < 10) {
    background(merrychristmasImage);
    if (userChoice === 'cactus') {
      plantX = 485 + xOffset;
      plantY = 337 + yOffset;
      image(cactusImage, plantX, plantY, plantSize, plantSize);
      
      // Display the selected plant based on user choice along with ornaments
      if (ornamentsList[0]) {
        for (let i = 0; i < ornamentsList.length; i++) {
          let ornament = ornamentsList[i];
          image(ornament.image, ornament.x, ornament.y, ornament.size, ornament.size);
        }
      }
  } else if (userChoice === 'palmTree') {
      plantX = 489 + xOffset;
      plantY = 341 + yOffset;
      image(palmtreeImage, plantX, plantY, plantSize, plantSize);
    
      if (ornamentsList[0]) {
        for (let i = 0; i < ornamentsList.length; i++) {
          let ornament = ornamentsList[i];
          image(ornament.image, ornament.x, ornament.y, ornament.size, ornament.size);
        }
      }
  } else if (userChoice === 'greenPlant') {
      plantX = 500 + xOffset;
      plantY = 339 + yOffset;
      image(greenplantImage, plantX, plantY, plantSize, plantSize);
    
      if (ornamentsList[0]) {
        for (let i = 0; i < ornamentsList.length; i++) {
          let ornament = ornamentsList[i];
          image(ornament.image, ornament.x, ornament.y, ornament.size, ornament.size);
        }
      }
  }
    
  }
  
}
// MousePressed function checks if the user is ready to add an ornament and if the click is within the plant area
function mousePressed() {
   // Add ornament at the clicked position
  if (isAddingOrnament && mouseX > plantX && mouseX < plantX + plantSize && mouseY > plantY && mouseY < plantY + plantSize) {
    // Add ornament at the clicked position
    let ornament = {
      image: random(ornaments),
      x: mouseX,
      y: mouseY,
      size: 40, // Adjust the size of the ornament as needed
    };

    ornamentsList.push(ornament);

    // Reset the flag
    isAddingOrnament = false;
  }
}

In the drawResultPage function, the code checks the sensor values (soil moisture, photocell, and force sensor) and simulates plant growth based on these values. If conditions related to sensor values are met, the plant size is increased, and offsets are adjusted to simulate growth. If the force sensor value is high, the variable isAddingOrnament is set to true, indicating that an ornament should be added. The code also includes a section for displaying ornaments on the plant based on user interactions. If isAddingOrnament is true, the mousePressed function adds an ornament at the clicked position within the plant area.

Concept : “Frenzy Jump”

For my project, I decided to create a simple game on p5js.

Idea of the game : For the Arduino part, there are two buttons one functions as a “restart button” to restart the game after losing, and the other functions as an action  button to make the player “triangle” to jump.
The game starts with a background that says “Press red button to start and white button to jump”, upon pressing the red red button the screen will start the game with the player being a “triangle”, as per instruction at the start the user will press the white button for the player to jump over rectangular obstacles, throughout the game there are golden “coins” which are ellipses created to gain extra points on the scoreboard that can be seen on the top right corner of the game. When the user fails with jumping over a certain obstacle, the screen will give you a message saying “GAMEOVER” . To restart the game, the user should press on the red button, and the game will automatically start again.

Inspiration:

I got this idea from a game i think most people have played once in their life, view the image below :

Components:

• 2 LED buttons , red and white
• Resistors are built in the led button i did not use the LED side of the button therefore i did not add extra resistors on the breadboard
• 6 jumper wires

Controller:

Prototype 1 :

Final prototype :

User Experience and Gameplay :

IMG_4787

IMG_4783

IMG_4784

Embedded sketch :

Schematics:

its probably incorrect but i tried

Testing in Arduino Serial Port :

References :

P5js code : https://editor.p5js.org/mka413/sketches/kwS7JN7Wh

The code that made me proud: I cant choose only one part, because if i must say i am very proud of myself, and what i have achieved this semester, to knowing nothing about coding to creating this simple yet successful simple game. What about difficulties? i will talk about that later on in my blog post.

How does Arduino work with p5js? :

Starting with the code in Arduino after I’ve set up the connections in the controller i wrote the lines of code that we were taught in class about Serial Data communication and put in the corresponding pin numbers for the “restart” and “action”(jump) buttons also i used the Function (pinMode) as  an input for my pins , and the function digitalRead for my pins to be read in values of :false or true lastly, i used the Serial.println to send/print serial data to the port as seen in the lines of code below :

int RedButton= 7;
int whiteButton= 8;
bool isActionButtonPressed = false; //start with false as the buttons are not pressed 
bool isRestartButtonPressed = false;


void setup() {
  Serial.begin(9600); // setting the baud rate for serial data communication 

  pinMode(RedButton, INPUT_PULLUP); 
  pinMode(whiteButton, INPUT_PULLUP);


}

void loop() {
//sending signals whether the buttons are pressed and giving true or false statements to confirm 
  if(digitalRead(RedButton)) {
    if(!isActionButtonPressed) {

      eventActionButtonPressed(RedButton);

      isActionButtonPressed = true;
    }
  }
  else {
    if(isActionButtonPressed) {

      eventActionButtonReleased(RedButton);

      isActionButtonPressed = false;
    }  
  }

  if(digitalRead(whiteButton)) {
    if(!isRestartButtonPressed) {

      eventRestartButtonPressed(whiteButton);

      isRestartButtonPressed = true;
    }
  }
  else {
    if(isRestartButtonPressed) {

      eventRestartButtonReleased(whiteButton);

      isRestartButtonPressed = false; //// printing data to the serial port void eventActionButtonPressed(int pin) 
    }  
  }

}

// printing data to the serial port void eventActionButtonPressed(int pin) 
void eventActionButtonPressed(int pin) {
  Serial.println(String(pin) + ":pressed");
}

void eventActionButtonReleased(int pin) {
  Serial.println(String(pin) + ":released");
}

void eventRestartButtonPressed(int pin) {
  Serial.println(String(pin) + ":pressed");
}

void eventRestartButtonReleased(int pin) {
  Serial.println(String(pin) + ":released");
}

After finishing up the Arduino code, now its time to begin serial data communication in p5JS, with firstly adding the ready web serial file that was made available for us, then in the actual p5js sketch code i added these lines of code to make serial data communication work with Arduino IDE :

// Function to read serial data
function readSerial(data) {
  const pin = data.split(':')[0]; 
  const action = data.split(':')[1];
  
  console.log(data);
  
  if(pin == "7" && action == "pressed") {
    if(!player.isJumping) {
      player.isJumping = true;
      player.timeJump = millis();
      player.forceY = -10;
    }
  }
  
   if(pin == "8" && action == "pressed") {
      if(!gameStarted) {
        gameStarted = true;
      }
     
     if(gameover) {
        gameover = false;
        score = 0;
        player.posX = width / 3;
       
        for(const o of obstacles) o.posX += width;

      }
    }

}

// Function to handle key presses
function keyPressed() {
  // If space is pressed, set up serial communication
  if(key == " ") {
    setUpSerial();
  }
  
  // press f to play the game in fullscreen
  if(key == 'f') {
    const fs = fullscreen();
    fullscreen(!fs);
  }
}
// this function is called every time the browser window is resized
function windowResized() {
  resizeCanvas(windowWidth, windowHeight);
}

Difficulties :

I had a lot of difficulties in making my initial acrylic controller to work, which i firstly thought there must be a problem with my code, after numerous attempts in trying to make the button and flip switch to work, i gathered that the connections were not correct or they have been short circuited. Thinking fast, i created the usual breadboard  connections with push button switches that worked instantly when uploading the code to my arduino. Also i had trouble with making the serial data communication to work on p5js where the browser would not pop up so i can make the connection to the serial port, which was nerve wrecking but i eventually after trial and error i got it to work perfectly.

My initial Conntroller :

Changes for the future:

I would really liked to have my first controller to work, it looked really cool, and would have fit in more with my game and its aesthetic. Still i would like to advance more in my coding skills to create something much more complex. Adding more difficult obstacles, and increasing the speed the longer you play, with a high score page also.

New Concept:

In the true spirit of being a bit indecisive, I’ve opted for a puzzle game. It all started with the certainty that I’d be dealing with a joystick in my physical computing adventures. So, brainstorming around this joystick, the idea of a picture puzzle game struck me. The concept is simple: solve the puzzle using the joystick. To add a personal touch, I thought it’d be cool to let players use their own pictures for the puzzle. I mean, puzzles can be dull, right? So, why not make it more fun by solving a puzzle of yourself?

P5.js Code:

I’ve made some progress in coding the game using P5.js. The groundwork includes screens for welcoming players, providing instructions, selecting difficulty levels, activating the camera, and of course, the puzzle screen itself. The code varies in complexity since it’s still a work in progress. Here’s a snippet that deals with turning a captured picture into a puzzle, which excites me the most.

function captureAndSetupPuzzle(video) {
  if (video) {
    source = video.get();
    source.loadPixels(); // Ensure pixels are loaded
 if (source.width > 0 && source.height > 0) {
    // Resize the source image to fit the canvas
    source.resize(width, height);
    video.hide();

    w = Math.floor(width / cols);
    h = Math.floor(height / rows);

    for (let i = 0; i < cols; i++) {
      for (let j = 0; j < rows; j++) {
        let x = i * w;
        let y = j * h;
        let img = source.get(x, y, w, h); // Get a portion of the image for each tile

        if (i === cols - 1 && j === rows - 1) {
          board.push(-1);
          puzzle.tiles.push(new Tile(-1, img));
        } else {
          let index = i + j * cols;
          board.push(index);
          puzzle.tiles.push(new Tile(index, img));
        }
      }
    }

    puzzle.board = board.slice();
    puzzle.simpleShuffle(puzzle.board);

    currentScreen = 'game';
    puzzle.startTimer();
  } else {
    console.error("Error loading the video source");
    }
  }
}

function setup() {
  createCanvas(600,400);
  //createCanvas(displayWidth, displayHeight);
  
  video = createCapture(VIDEO);
  video.size(400, 400);
  video.position(0, 0);
  video.hide();
  
  
  let timerDuration ;
  let level;

  puzzle = new Puzzle(cols, rows, timerDuration, level); // Example level: 3x3 grid, 600 seconds timer
}

Arduino Code:

Additionally, I’ve been working on Arduino code. It seems mostly complete for now, though I might tweak it as my P5 code progresses. The aim of this code is to control the button movements and the movements of the puzzle tiles using the joystick.

const int XbuttonPin = 2;
const int SbuttonPin = 3;
const int TbuttonPin = 4;
const int CbuttonPin = 5;
const int joystickXPin = A0; // Analog pin for joystick X-axis
const int joystickYPin = A1; // Analog pin for joystick Y-axis
const int threshold = 50; // Threshold for joystick sensitivity
//bool isDifficulty = false;

void setup() {
  Serial.begin(9600);
  pinMode(XbuttonPin, INPUT_PULLUP);
  pinMode(SbuttonPin, INPUT_PULLUP);
  pinMode(TbuttonPin, INPUT_PULLUP);
  pinMode(CbuttonPin, INPUT_PULLUP);
}

void loop() {
  if (digitalRead(XbuttonPin) == LOW) {
    Serial.println("MOUSE_CLICK");
    delay(1000); // Debounce delay
  }
  
  if (digitalRead(SbuttonPin) == LOW) {
    Serial.println('2');
    delay(100); // Debounce delay
  }
  
  if (digitalRead(TbuttonPin) == LOW) {
    Serial.println('1');
    delay(10000); // Debounce delay
  }
  
  if (digitalRead(CbuttonPin) == LOW) {
    Serial.println('3');
    delay(100); // Debounce delay
  }

  if (digitalRead(TbuttonPin) == LOW) {
    Serial.println('C');
    delay(100); // Debounce delay
  }
  
  int xVal = analogRead(joystickXPin); // Read X-axis value
  int yVal = analogRead(joystickYPin); // Read Y-axis value

  if (xVal < 512 - threshold) {
    Serial.println("LEFT");
    delay(100); // Debounce delay
  } else if (xVal > 512 + threshold) {
    Serial.println("RIGHT");
    delay(100); // Debounce delay
  }

  if (yVal < 512 - threshold) {
    Serial.println("DOWN");
    delay(100); // Debounce delay
  } else if (yVal > 512 + threshold) {
    Serial.println("UP");
    delay(100); // Debounce delay
  }
}

Challenges popped up, especially when translating mouse movements to joystick actions. Initially, I aimed to use mouse clicks and key presses in my P5.js code, thinking I could easily convert them to buttons and switches. But handling joystick movements, considering up, down, left, and right, turned out more intricate than merely clicking a mouse to move a tile.

Prototype:

IMG_5120

Tasks on my to-do list seem endless because, well, I’m a bit of a perfectionist. At the moment, I’m focusing on crafting a case for my Arduino and breadboard. Simultaneously, I’m tirelessly refining my P5 sketch for a more appealing look. Adding background music and possibly turning the puzzle into more of an actual game are ideas I’m mulling over. But for now, this is where I stand in my project. I am also thinking of implementing the LED screen to the sketch that would display a message if the puzzle were solved because I think this would be a nice way to implement the p5 to Arduino communication.

Concept:

For my final project, I have created a game called “Dodge It!”. This game consists of moving rectangles- red and blue colored- on screen and there’s a ball that moves horizontally that the users control to navigate through these obstacles while dodging the red rectangles and picking up the blue ones to gain points.  There’s a slight twist to it, instead of using just buttons to control the ball, users use their physical hands to swipe left and right to control the ball’s movement in the game. To increase the interaction, I also implemented a system where every score will produce a unique tone and as the user loses scores, the tone keeps getting deeper and deeper as if they’re reaching the end, and as the score gets higher and higher, the tone gets higher showcasing an increased suspense element. The motivation behind this game is from online games that I used to play back in elementary school. These games were similar to “Dodge it!” but were mostly used by keyboard buttons to control the ball. Whereas, my game can make things much more interactive by using your own physical hands to control the ball’s movement, which adds a whole new spectrum of excitement and adrenaline rush as we navigate through the obstacles. My game also has different difficulty levels for users to choose from to make things more balanced between the enjoyers and the competitors.

Prototype:

Pictures:

Video:

Implementation:

Description of Interaction Design:

In order to balance out the lags that I was getting in between the code due to excessive elements being inputted by the Arduino and outputted to p5, I decided to make things much simpler and quite responsive by adding the necessary elements only. The prototype that I have created consists of an ultrasonic sensor incorporated within it and the users just need to put their hands in front of it and start playing straightforward. While I could have added more buttons to the prototype, this would have ruined the simplicity that I was aiming for. Also, more buttons would result in a bit more delay within the responsiveness of the ultrasonic sensor to the hand movement, and my goal was to make it as smooth as possible, so these changes were necessary.

Description of Arduino Code:

The Arduino part of the game is quite simple. I have added an ultrasonic sensor and a speaker. After connecting it to the digital pins, I utilized the send to p5 feature through serial print to send the outputted value of the ultrasonic sensor. The ultrasonic sensor outputs a value, which can be converted to distance through an equation, that distance is what I send to P5 which will later be reflected to the ball’s x coordinate. The Arduino receives the score value sent by p5; which will then be mapped and sent to the speaker to produce a tone. Additionally, I added an if condition to stop emitting a sound if the user is at the menu or endscreen.

//Define pins for speaker and ultrasonic sensor
const int trigPin = 11;
const int echoPin = 12;
const int speakerPin = 8;

void setup() {
  // Start serial communication so we can send data
  // over the USB connection to our p5js sketch
  Serial.begin(9600);
  pinMode(LED_BUILTIN, OUTPUT);

  // Outputs on these pins
  pinMode(trigPin, OUTPUT);
  pinMode(echoPin, INPUT);
  pinMode(speakerPin, OUTPUT);


}

void loop() {

  //GETS FROM P5
  int score = Serial.parseInt();

//if condition so that when score is 0 at menu and end screen, the sound doesnt emit and annoy us 
if(score==0)
{
  noTone(speakerPin);
}
else
{
  // Map the score to a frequency for the speaker
  int frequency = map(score, 0, 200, 100, 4000);
  // Play the tone
  tone(speakerPin, frequency);
}

  // sends a short pulse of ultrasonic sensor and waits a bit then receives it 
  digitalWrite(trigPin, LOW); 
  delayMicroseconds(2); 
  digitalWrite(trigPin, HIGH); 
  delayMicroseconds(10); 
  digitalWrite(trigPin, LOW); 

  // Time it takes for the pulse to travel back from the object long 
  int duration = pulseIn(echoPin, HIGH); 
  // Universal conversion of time into distance in cm 
  int distance = duration * 0.034 / 2;

  //SENDS TO P5
  Serial.println(distance);
  delay(50);

}

Description of P5.js code:

For my p5 side of the code, this took almost all of the time, designing an interactive interface that had aesthetic elements to it while also an interactive one. The p5 will receive values of distance from Arduino and input it towards the ball’s movements. One major factor that I had difficulty in would be smoothening the ultrasonic sensor’s reading. Initially, the ball was not as smooth, so I added a Lerp function to smoothen it out. Then the ball had abrupt changes in its distances, which is due to external issues from the ultrasonic sensor itself, so I fixed it by adding a maxPosChange variable to make sure that if the next reading is way greater than the previous one, the p5 will neglect that reading, ensuring a smoother path for the ball. As I set the foundation for my game in P5, I started working on the aesthetics, adding different difficulties and an end screen at the end. Each button also produces a hover sound and a click sound when clicked. I added texts explaining each difficulty. Finally, I also added a pause screen and exit to the main menu mid-game features to make things much faster and smoother to navigate.

While the code is long, I’ll only add the serial function where it sends and receives data to and from Arduino. You can navigate to the code by clicking on the P5 link above.

function readSerial(data) {
  ////////////////////////////////////
  // READ FROM ARDUINO HERE
  ////////////////////////////////////

  if (data != null) {
    // Parse the data received from Arduino
    let fromArduino = split(trim(data), ",");
    targetKnobValue = parseFloat(fromArduino[0]);
    print(targetKnobValue);
    
        // Check for abrupt changes in position and neglect them
    if (abs(targetKnobValue - knobValue) < maxPositionChange) {
      // Smooth the value we get from the Arduino using linear interpolation (lerp)
      knobValue = lerp(knobValue, targetKnobValue, easing);
    } 
  }

  //////////////////////////////////
  // SEND TO ARDUINO HERE
  //////////////////////////////////
  
  //if at menu or end screen dont send score to arduino
  if(option== MAINMENU || option== ENDSCREEN)
    {
    let notone=0;
    let sendToArduino= notone + "\n";
    writeSerial(sendToArduino);
    }
  else
    {
    let sendToArduino= score + "\n";
    writeSerial(sendToArduino); 
    }
}

• Description of communication between Arduino and p5.js

As previously mentioned, both the P5 and Arduino complement and expand upon one another, where the Arduino receives the values of the changing score, and then maps that score to a a spectrum to produce a tone within the speaker. The Arduino will also send a distance value from the hand to the ultrasonic sensor to the P5 which is utilized in the ball class to control the ball’s movement. Of course, the distance is mapped to the Arduino’s dynamic canvas.

Some aspects of the project that I’m proud of

I’m mainly proud of the aesthetics and the beauty of the game. Adding a moving background for instance was not easy. I had to navigate through various websites to see how they did it. the implementation of various difficulties and how the screen changes everytime was also another aspect that I’m proud of. In addition to that, I’m also proud of how much I have decreased the errors in the ultrasonic sensor by adding proper lines of code to take care of the errors. Such as the abrupt changes in the ultrasonic sensor reading resulted in a big jump in the ball’s x coordinate, fixed by adding a tolerance which if the sensor exceeded from one reading to another, it disregards it. This made the game much more stable and I’m very proud of it.

Areas of Improvements

Some of the areas of improvement will most likely be towards the overall prototype and user interaction, due to issues with decreasing lag and delays I had to neglect certain buttons and elements that I could have added to the overall prototype while still achieving that simplistic look. For future improvements, more interactive elements can be added to the real-world prototype instead of just within the code. I could maybe also add a car that could be responsive to the scores, the higher the score results in the car moving at a higher speed and vice versa. I initially thought of doing that, but making a car was another big hassle, and adding it with the ultrasonic sensor would create multiple delays and issues within the execution of the ball’s movement.

IM SHOWCASE: