Week 12- Final Project Draft 2

New Concept:

In the true spirit of being a bit indecisive, I’ve opted for a puzzle game. It all started with the certainty that I’d be dealing with a joystick in my physical computing adventures. So, brainstorming around this joystick, the idea of a picture puzzle game struck me. The concept is simple: solve the puzzle using the joystick. To add a personal touch, I thought it’d be cool to let players use their own pictures for the puzzle. I mean, puzzles can be dull, right? So, why not make it more fun by solving a puzzle of yourself?

P5.js Code:

I’ve made some progress in coding the game using P5.js. The groundwork includes screens for welcoming players, providing instructions, selecting difficulty levels, activating the camera, and of course, the puzzle screen itself. The code varies in complexity since it’s still a work in progress. Here’s a snippet that deals with turning a captured picture into a puzzle, which excites me the most.

function captureAndSetupPuzzle(video) {
  if (video) {
    source = video.get();
    source.loadPixels(); // Ensure pixels are loaded
 if (source.width > 0 && source.height > 0) {
    // Resize the source image to fit the canvas
    source.resize(width, height);
    video.hide();

    w = Math.floor(width / cols);
    h = Math.floor(height / rows);

    for (let i = 0; i < cols; i++) {
      for (let j = 0; j < rows; j++) {
        let x = i * w;
        let y = j * h;
        let img = source.get(x, y, w, h); // Get a portion of the image for each tile

        if (i === cols - 1 && j === rows - 1) {
          board.push(-1);
          puzzle.tiles.push(new Tile(-1, img));
        } else {
          let index = i + j * cols;
          board.push(index);
          puzzle.tiles.push(new Tile(index, img));
        }
      }
    }

    puzzle.board = board.slice();
    puzzle.simpleShuffle(puzzle.board);

    currentScreen = 'game';
    puzzle.startTimer();
  } else {
    console.error("Error loading the video source");
    }
  }
}

function setup() {
  createCanvas(600,400);
  //createCanvas(displayWidth, displayHeight);
  
  video = createCapture(VIDEO);
  video.size(400, 400);
  video.position(0, 0);
  video.hide();
  
  
  let timerDuration ;
  let level;

  puzzle = new Puzzle(cols, rows, timerDuration, level); // Example level: 3x3 grid, 600 seconds timer
}

Arduino Code:

Additionally, I’ve been working on Arduino code. It seems mostly complete for now, though I might tweak it as my P5 code progresses. The aim of this code is to control the button movements and the movements of the puzzle tiles using the joystick.

const int XbuttonPin = 2;
const int SbuttonPin = 3;
const int TbuttonPin = 4;
const int CbuttonPin = 5;
const int joystickXPin = A0; // Analog pin for joystick X-axis
const int joystickYPin = A1; // Analog pin for joystick Y-axis
const int threshold = 50; // Threshold for joystick sensitivity
//bool isDifficulty = false;

void setup() {
  Serial.begin(9600);
  pinMode(XbuttonPin, INPUT_PULLUP);
  pinMode(SbuttonPin, INPUT_PULLUP);
  pinMode(TbuttonPin, INPUT_PULLUP);
  pinMode(CbuttonPin, INPUT_PULLUP);
}

void loop() {
  if (digitalRead(XbuttonPin) == LOW) {
    Serial.println("MOUSE_CLICK");
    delay(1000); // Debounce delay
  }
  
  if (digitalRead(SbuttonPin) == LOW) {
    Serial.println('2');
    delay(100); // Debounce delay
  }
  
  if (digitalRead(TbuttonPin) == LOW) {
    Serial.println('1');
    delay(10000); // Debounce delay
  }
  
  if (digitalRead(CbuttonPin) == LOW) {
    Serial.println('3');
    delay(100); // Debounce delay
  }

  if (digitalRead(TbuttonPin) == LOW) {
    Serial.println('C');
    delay(100); // Debounce delay
  }
  
  int xVal = analogRead(joystickXPin); // Read X-axis value
  int yVal = analogRead(joystickYPin); // Read Y-axis value

  if (xVal < 512 - threshold) {
    Serial.println("LEFT");
    delay(100); // Debounce delay
  } else if (xVal > 512 + threshold) {
    Serial.println("RIGHT");
    delay(100); // Debounce delay
  }

  if (yVal < 512 - threshold) {
    Serial.println("DOWN");
    delay(100); // Debounce delay
  } else if (yVal > 512 + threshold) {
    Serial.println("UP");
    delay(100); // Debounce delay
  }
}

Challenges popped up, especially when translating mouse movements to joystick actions. Initially, I aimed to use mouse clicks and key presses in my P5.js code, thinking I could easily convert them to buttons and switches. But handling joystick movements, considering up, down, left, and right, turned out more intricate than merely clicking a mouse to move a tile.

Prototype:

IMG_5120

Tasks on my to-do list seem endless because, well, I’m a bit of a perfectionist. At the moment, I’m focusing on crafting a case for my Arduino and breadboard. Simultaneously, I’m tirelessly refining my P5 sketch for a more appealing look. Adding background music and possibly turning the puzzle into more of an actual game are ideas I’m mulling over. But for now, this is where I stand in my project. I am also thinking of implementing the LED screen to the sketch that would display a message if the puzzle were solved because I think this would be a nice way to implement the p5 to Arduino communication.

Final Project Documentation- Dodge It!

Concept:

For my final project, I have created a game called “Dodge It!”. This game consists of moving rectangles- red and blue colored- on screen and there’s a ball that moves horizontally that the users control to navigate through these obstacles while dodging the red rectangles and picking up the blue ones to gain points.  There’s a slight twist to it, instead of using just buttons to control the ball, users use their physical hands to swipe left and right to control the ball’s movement in the game. To increase the interaction, I also implemented a system where every score will produce a unique tone and as the user loses scores, the tone keeps getting deeper and deeper as if they’re reaching the end, and as the score gets higher and higher, the tone gets higher showcasing an increased suspense element. The motivation behind this game is from online games that I used to play back in elementary school. These games were similar to “Dodge it!” but were mostly used by keyboard buttons to control the ball. Whereas, my game can make things much more interactive by using your own physical hands to control the ball’s movement, which adds a whole new spectrum of excitement and adrenaline rush as we navigate through the obstacles. My game also has different difficulty levels for users to choose from to make things more balanced between the enjoyers and the competitors.

Prototype:

Pictures:

 

 

 

 

 

Video:

 

Implementation:

Description of Interaction Design:

In order to balance out the lags that I was getting in between the code due to excessive elements being inputted by the Arduino and outputted to p5, I decided to make things much simpler and quite responsive by adding the necessary elements only. The prototype that I have created consists of an ultrasonic sensor incorporated within it and the users just need to put their hands in front of it and start playing straightforward. While I could have added more buttons to the prototype, this would have ruined the simplicity that I was aiming for. Also, more buttons would result in a bit more delay within the responsiveness of the ultrasonic sensor to the hand movement, and my goal was to make it as smooth as possible, so these changes were necessary.

Description of Arduino Code:

The Arduino part of the game is quite simple. I have added an ultrasonic sensor and a speaker. After connecting it to the digital pins, I utilized the send to p5 feature through serial print to send the outputted value of the ultrasonic sensor. The ultrasonic sensor outputs a value, which can be converted to distance through an equation, that distance is what I send to P5 which will later be reflected to the ball’s x coordinate. The Arduino receives the score value sent by p5; which will then be mapped and sent to the speaker to produce a tone. Additionally, I added an if condition to stop emitting a sound if the user is at the menu or endscreen.

//Define pins for speaker and ultrasonic sensor
const int trigPin = 11;
const int echoPin = 12;
const int speakerPin = 8;

void setup() {
  // Start serial communication so we can send data
  // over the USB connection to our p5js sketch
  Serial.begin(9600);
  pinMode(LED_BUILTIN, OUTPUT);

  // Outputs on these pins
  pinMode(trigPin, OUTPUT);
  pinMode(echoPin, INPUT);
  pinMode(speakerPin, OUTPUT);


}

void loop() {

  //GETS FROM P5
  int score = Serial.parseInt();

//if condition so that when score is 0 at menu and end screen, the sound doesnt emit and annoy us 
if(score==0)
{
  noTone(speakerPin);
}
else
{
  // Map the score to a frequency for the speaker
  int frequency = map(score, 0, 200, 100, 4000);
  // Play the tone
  tone(speakerPin, frequency);
}

  // sends a short pulse of ultrasonic sensor and waits a bit then receives it 
  digitalWrite(trigPin, LOW); 
  delayMicroseconds(2); 
  digitalWrite(trigPin, HIGH); 
  delayMicroseconds(10); 
  digitalWrite(trigPin, LOW); 

  // Time it takes for the pulse to travel back from the object long 
  int duration = pulseIn(echoPin, HIGH); 
  // Universal conversion of time into distance in cm 
  int distance = duration * 0.034 / 2;

  //SENDS TO P5
  Serial.println(distance);
  delay(50);

}

 

Description of P5.js code:

For my p5 side of the code, this took almost all of the time, designing an interactive interface that had aesthetic elements to it while also an interactive one. The p5 will receive values of distance from Arduino and input it towards the ball’s movements. One major factor that I had difficulty in would be smoothening the ultrasonic sensor’s reading. Initially, the ball was not as smooth, so I added a Lerp function to smoothen it out. Then the ball had abrupt changes in its distances, which is due to external issues from the ultrasonic sensor itself, so I fixed it by adding a maxPosChange variable to make sure that if the next reading is way greater than the previous one, the p5 will neglect that reading, ensuring a smoother path for the ball. As I set the foundation for my game in P5, I started working on the aesthetics, adding different difficulties and an end screen at the end. Each button also produces a hover sound and a click sound when clicked. I added texts explaining each difficulty. Finally, I also added a pause screen and exit to the main menu mid-game features to make things much faster and smoother to navigate.

While the code is long, I’ll only add the serial function where it sends and receives data to and from Arduino. You can navigate to the code by clicking on the P5 link above.

function readSerial(data) {
  ////////////////////////////////////
  // READ FROM ARDUINO HERE
  ////////////////////////////////////

  if (data != null) {
    // Parse the data received from Arduino
    let fromArduino = split(trim(data), ",");
    targetKnobValue = parseFloat(fromArduino[0]);
    print(targetKnobValue);
    
        // Check for abrupt changes in position and neglect them
    if (abs(targetKnobValue - knobValue) < maxPositionChange) {
      // Smooth the value we get from the Arduino using linear interpolation (lerp)
      knobValue = lerp(knobValue, targetKnobValue, easing);
    } 
  }

  //////////////////////////////////
  // SEND TO ARDUINO HERE
  //////////////////////////////////
  
  //if at menu or end screen dont send score to arduino
  if(option== MAINMENU || option== ENDSCREEN)
    {
    let notone=0;
    let sendToArduino= notone + "\n";
    writeSerial(sendToArduino);
    }
  else
    {
    let sendToArduino= score + "\n";
    writeSerial(sendToArduino); 
    }
}

 

  • Description of communication between Arduino and p5.js

As previously mentioned, both the P5 and Arduino complement and expand upon one another, where the Arduino receives the values of the changing score, and then maps that score to a a spectrum to produce a tone within the speaker. The Arduino will also send a distance value from the hand to the ultrasonic sensor to the P5 which is utilized in the ball class to control the ball’s movement. Of course, the distance is mapped to the Arduino’s dynamic canvas.

Some aspects of the project that I’m proud of

I’m mainly proud of the aesthetics and the beauty of the game. Adding a moving background for instance was not easy. I had to navigate through various websites to see how they did it. the implementation of various difficulties and how the screen changes everytime was also another aspect that I’m proud of. In addition to that, I’m also proud of how much I have decreased the errors in the ultrasonic sensor by adding proper lines of code to take care of the errors. Such as the abrupt changes in the ultrasonic sensor reading resulted in a big jump in the ball’s x coordinate, fixed by adding a tolerance which if the sensor exceeded from one reading to another, it disregards it. This made the game much more stable and I’m very proud of it.

Areas of Improvements

Some of the areas of improvement will most likely be towards the overall prototype and user interaction, due to issues with decreasing lag and delays I had to neglect certain buttons and elements that I could have added to the overall prototype while still achieving that simplistic look. For future improvements, more interactive elements can be added to the real-world prototype instead of just within the code. I could maybe also add a car that could be responsive to the scores, the higher the score results in the car moving at a higher speed and vice versa. I initially thought of doing that, but making a car was another big hassle, and adding it with the ultrasonic sensor would create multiple delays and issues within the execution of the ball’s movement.

IM SHOWCASE:

Final Project- User Testing

I conducted the user testing without giving them any prior notice and only reading from the interface, and they had some questions like when the game ends? So I added more text after that to make things more clear for the users. Other than that, once they got the hang of it, they were able to play the game easily and they figured out everything on how it works and how to stop. From the third iteration onwards, they become a master and play even better than I do(at least most of them). One area that could be improved is the addition of a text on the prototype saying Hover here or Put your hand here since they didn’t know where the sensor was in the beginning. That part of explaining where to hover the hand was the issue, which I will address by adding text on the prototype as mentioned.

 

Video:

week 12 : Final project update

 

“frenzy jump”

so previously i thought about making an instrument using the serial communication between arduino and p5, that turned out to be very difficult , i could have done it, but unfortunately with the time on hand i could not get it finished on time. therefore, i decided to create a very simple game on p5js that consists of a ball jumping over obstacles, and for the arduino part, i have created a remote switch/button panel that contains the arduino, and 4 different components, but i will use only two. The 4 components are 3 paddle switches, and one button, i will use one paddle switch, and one button to control the ball in the game.

My switch panel controller :

components:

Arduino protoshield

arduino board

3 paddle switches

one button

Initial code :

const player = {
  posX: 0,
  posY:  0,
  size: 50,
  forceY: 0,
  isJumping: false
}

const obstacles = [{
  posX: 600,
  height: 50,
  width: 70
},{
  posX: 1800,
  height: 50,
  width: 70
},{
  posX: 2500,
  height: 50,
  width: 70
}];

const gravity = 0.3;
const mapSpeed = 5;

const floorPosition = 100;

let gameover = false;

function setup() {
  createCanvas(window.innerWidth, window.innerHeight);
  
  player.posX = width / 3;
  player.posY = height - floorPosition - 100;
  
}

function draw() {
  background(0);
  
  if(!gameover) {
    
    // Check for serial connection
    if(!serialActive) {
      fill(255);
      text("Press space to select serial port.", 20, 30);  
    }
    else {
      fill(255);
      text("Connected", 20, 30);  

    }

    // Draw game
    drawFloor();
    drawObstacles();
    drawPlayer();


    // Move game
    movePlayer();
    moveObstacles();
  }
}

function drawFloor() {
  fill(255);
  
  rect(0, height - floorPosition, width, floorPosition);
}

function drawPlayer() {
  fill(255, 150, 0);
  
  ellipse(player.posX, player.posY, player.size, player.size);
}

function drawObstacles() {
  fill(255);
  for(o of obstacles) {
    rect(o.posX, height - floorPosition, o.width, -o.height);    
  }
}

function testCollisions(x, y) {
  // Test collide with floor
  if(player.forceY > 0 && y > height - floorPosition - player.size/2) {
    return true;
  }
  
  for(o of obstacles) {
    if(x + player.size / 2 > o.posX && x - player.size / 2 < o.posX + o.width && y > height - floorPosition - o.height - player.size / 2) {
      return true;  
    }
  }
  
  return false;
}

function movePlayer() {
  
  if(testCollisions(player.posX, player.posY)) {
    player.posX -= mapSpeed;
    player.isJumping = false;
    
    if(player.posX < 0 - player.size) {
      // gameover
      gameover = true;
    }

  }
  else if(player.posX < width / 3) {
    player.posX += 0.5;
  }
  
  const nextPos = player.posY + player.forceY;
  
  if(testCollisions(player.posX, nextPos)) {
    player.forceY = 0;
    player.isJumping = false;
  }
  else {
    player.posY += player.forceY;
    player.forceY += gravity;
  }
  
  
}

function moveObstacles() {
  for(o of obstacles) {
    o.posX -= mapSpeed;
    
    if(o.posX < -o.width) {
      o.posX = random(width, width*1.5);
      o.width = random(50, 200);
      o.height = random(20, 200);
    }
  }  
  
  
}

function readSerial(data) {
  const pin = data.split(':')[0]; 
  const action = data.split(':')[1];
  
  console.log(action);
  
  if(action == "pressed") {
    pressed = true;
  }
  else {
    pressed = false;
  }

}

function keyPressed() {
  if(key == " ") {
    setUpSerial();
  }
  
  if(key == 'z' && !player.isJumping) {
    player.isJumping = true;
    player.forceY = -8;
  }
}

 

 

idea of possibly how the game will look like:

Final Project Draft

Currently I have figured out the connection between arduino and p5 so that based on poseNet I can move the servo as if it is following me. I have done this on the basis of getting the position of nose pose and sending it to arduino mapped to the servo range. Currently it works as intended however I think it might be a bit off when I am at extreme edges of the canvas video. I am thinking of rectifying by getting a distance variable for my next draft where i will use the distance between eyes to maybe get a more accurate angle.

I will follow a cardboard tutorial to make the CCTV:

let video;
let poseNet;
let pose;
let skeleton;
let loco= 0;

function setup() {
  createCanvas(640, 480);
  video = createCapture(VIDEO);
  video.hide();
  poseNet = ml5.poseNet(video, modelLoaded);
  poseNet.on('pose', gotPoses);
}

function gotPoses(poses) {
  //console.log(poses); 
  if (poses.length > 0) {
    pose = poses[0].pose;
    skeleton = poses[0].skeleton;
  }
}


function modelLoaded() {
  console.log('poseNet ready');
}

function draw() {
  
   if (!serialActive) {
    text("Press Space Bar to select Serial Port", 20, 30);
  } else {
    text("Connected", 20, 30);
  }
  
  image(video, 0, 0);

  if (pose) {
    
    fill(255, 0, 0);
    ellipse(pose.nose.x, pose.nose.y, 20);
   
    loco = int(pose.nose.x);
    val = int(map(loco, 0, 640, 0, 180));

    print(val)
    
  
  }
}

function keyPressed() {
  if (key == " ") {
    // important to have in order to start the serial connection!!
    setUpSerial();
  }
}

function readSerial(data) {
  ////////////////////////////////////
  //READ FROM ARDUINO HERE
  ////////////////////////////////////

  if (data != null) {
    // make sure there is actually a message
    // split the message
    let fromArduino = split(trim(data), ",");
    // if the right length, then proceed
    if (fromArduino.length == 2) {
      // only store values here
      // do everything with those values in the main draw loop
      print("nice");
      // We take the string we get from Arduino and explicitly
      // convert it to a number by using int()
      // e.g. "103" becomes 103
    
    }

    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    let sendToArduino = val + "\n";
    writeSerial(sendToArduino);
    
  }
}

P5 👆

#include <Servo.h>

Servo myservo;  // create servo object to control a servo


void setup() {

  Serial.begin(9600);

  myservo.attach(9);

  // start the handshake
  while (Serial.available() <= 0) {
    digitalWrite(LED_BUILTIN, HIGH); // on/blink while waiting for serial data
    Serial.println("0,0"); // send a starting message
    delay(300);            // wait 1/3 second
    digitalWrite(LED_BUILTIN, LOW);
    delay(50);
           myservo.write(90);                  // sets the servo position according to the scaled value

  }
}

void loop() {
  // wait for data from p5 before doing something
  
  while (Serial.available()) {
    Serial.println("0,0");
    digitalWrite(LED_BUILTIN, HIGH); // led on while receiving data
    int value = Serial.parseInt();
    if (Serial.read() == '\n') {
       myservo.write(value);                  // sets the servo position according to the scaled value
    }
  }
}

arduino 👆

trial video: https://youtube.com/shorts/h1etPCv24vA

 

Final Project Draft 1

My aim in this project was to try and make a game that allows the player to help sell pie.

Using the serial connectios the player should be able to use the potentiometer in order to get the wanted order. When one is chosen they press a button to confirm the choice.

The game goes for 2.5 minuts and the player must complete as many orders as possible.

 

So far I have this

P5.js sketch:

let x = 0;

let backgroundImage;
let rawImage, undercookedImage, perfectImage, overcookedImage, burntImage;

function preload() {
  // Load background image only for now
  backgroundImage = loadImage('Background.jpg');
}

function loadFoodImages() {
  // Load images after the serial connection starts
  rawImage = loadImage('raw.jpg');
  undercookedImage = loadImage('undercooked.jpg');
  perfectImage = loadImage('perfect.jpg');
  overcookedImage = loadImage('overcooked.jpg');
  burntImage = loadImage('burnt.jpg');
}

function setup() {
  createCanvas(640, 480);
  textSize(18);
}

function draw() {
  // Display background image
  image(backgroundImage, 0, 0, width, height);

  // Display image based on the value of x
  displayFoodImage();
  
  // Display serial connection status
  fill(0);
  if (!serialActive) {
    text("Press Space Bar to select Serial Port", 20, 30);
  } else {
    text("Connected", 20, 30);
  }
}

function displayFoodImage() {
  // Define the ranges for each image
  const rawRange = [0, 200];
  const undercookedRange = [201, 400];
  const perfectRange = [401, 600];
  const overcookedRange = [601, 800];
  const burntRange = [801, 1023];

  // Map x to the corresponding image based on the ranges
  let currentImage;
  if (x >= rawRange[0] && x <= rawRange[1]) {
    currentImage = rawImage;
  } else if (x >= undercookedRange[0] && x <= undercookedRange[1]) {
    currentImage = undercookedImage;
  } else if (x >= perfectRange[0] && x <= perfectRange[1]) {
    currentImage = perfectImage;
  } else if (x >= overcookedRange[0] && x <= overcookedRange[1]) {
    currentImage = overcookedImage;
  } else if (x >= burntRange[0] && x <= burntRange[1]) {
    currentImage = burntImage;
  }

  // Display the current image
  if (currentImage) {  // Check if the image is defined
    image(currentImage, width / 2 - 50, height / 2 - 50, 100, 100);
  }
}

function keyPressed() {
  if (key == " ") {
    // Important to have in order to start the serial connection!!
    setUpSerial();
  }
}

// This function will be called by the web-serial library
// with each new *line* of data. The serial library reads
// the data until the newline and then gives it to us through
// this callback function
function readSerial(data) {
  ////////////////////////////////////
  // READ FROM ARDUINO HERE
  ////////////////////////////////////
  if (data != null) {
    x = int(trim(data));
    
    // Load images only when the serial connection starts
    if (serialActive) {
      loadFoodImages();
    }
  }
}

 

Arduino sketch:

int sensorPin = A0;  
int sensorValue;

void setup() {
  Serial.begin(9600);
}

void loop() {
  // Read the sensor value
  sensorValue = analogRead(sensorPin);

  // Print the sensor value to the serial port
  Serial.println(sensorValue);

  delay(100);  
}

 

Circuit:

My potineometer works just fine. I still need to

-Display the required image according the the divided readings of the potineometer

-work on the button as a submitted answer

-figure out how to add timer and end results (maybe a leadboard)

-add sounds

-Finalize Instructions

Final Project – Draft 2 – Walking Buddy

Building on my initial idea to create an assistive device, I have decided to create a robot that serves as a walking assistant by warning the user of possible obstructions along the way. The robot would be voice controlled to give commands such as the direction of the robot. Based on the reading of an ultrasonic sensor the walking buddy would send out signals through the speaker to alert the user. Additionally, the user would be informed of the distance between the next obstruction to have a better sense while walking. If time permits, I plan to include the option of controlling the robot through hand movements as well to make the model more inclusive.

Arduino:

  • Uses the ultrasonic sensor to provide input on possible obstructions and sends it to p5.
  • Based on the ultrasonic reading the speaker plays a tune to warn the user.
  • The movement of the robot is controlled using the DC motors.

P5.js program:

  • Begins with informing the user of the different tunes and voice commands.
  • The user gives a voice command to change the direction of the robot.
  • The direction is communicated to the Arduino and the robot starts moving accordingly
  • P5 stores the distance from the next obstruction received from the Arduino and increases the frequency of the tune as the user gets closer to it.

Final Project: Something ml5.js

Progress:

As previously mentioned in my final project concept, I intend to create an “electronic buddy” or an “art robot” (which sounds better than my previous term). Thus far, I have successfully established communication between p5 and Arduino. Specifically, I utilized the handpose model from the ml5 library to detect if the hand is within specific areas on the screen and transmit a corresponding index to the Arduino.

As this is only a preliminary test, I have two motors connected to the Arduino, rotating in one direction if the hand is in the upper third of the screen, halting if the hand is in the middle portion, and rotating in the opposite direction if the hand is in the lower third. Currently, there is only unidirectional communication from p5 to Arduino, with p5 sending data to the latter. I hope to achieve meaningful communication from the Arduino to p5 as I progress. I have included the p5 sketch, Arduino sketch, and a video of the test run for reference.

p5 sketch:

let handpose;
let value = -1;
let video;
let predictions = [];

function setup() {
  createCanvas(640, 480);
  video = createCapture(VIDEO);
  video.size(width, height);

  handpose = ml5.handpose(video, modelReady);

  // This sets up an event that fills the global variable "predictions"
  // with an array every time new hand poses are detected
  handpose.on("predict", results => {
    predictions = results;
  });

  // Hide the video element, and just show the canvas
  video.hide();
}

function modelReady() {
  console.log("Model ready!");
}

function draw() {
  if (!serialActive) {
    text("Press Space Bar to select Serial Port", 20, 30);
  } else {
    image(video, 0, 0, width, height);

    // We can call both functions to draw all keypoints and the skeletons
    drawKeypoints();

    // draws sections on the screen
    for (let i = 0; i <= width / 2; i += (width / 2) / 3) {
      stroke(255, 0, 0);
      line(i , 0, i, height);
    }

    for (let i = 0; i <= height; i += height / 3) {
      stroke(255, 0, 0);
      line(0, i, width / 2, i);
    }

  }
  
}


function keyPressed() {
  if (key == " ") {
    // important to have in order to start the serial connection!!
    setUpSerial();
  }
}

function readSerial(data) {

  if (data != null) {

    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    let sendToArduino = value + "\n";
    writeSerial(sendToArduino);
  }
}

// A function to draw ellipses over the detected keypoints
function drawKeypoints() {
  for (let i = 0; i < predictions.length; i += 1) {
    const prediction = predictions[i];
    let area = [0, 0, 0, 0, 0, 0, 0, 0, 0];
    for (let j = 0; j < prediction.landmarks.length; j += 1) {
      const keypoint = prediction.landmarks[j];
      fill(0, 255, 0);
      noStroke();
      ellipse(keypoint[0], keypoint[1], 10, 10);
      
      // count number of trues
      // -- helps to detect the area the detected hand is in
      if (withinTopLeft(keypoint[0], keypoint[1])) {
        area[0] += 1;
      }
      if (withinTopCenter(keypoint[0], keypoint[1])) {
        area[1] += 1;
      }
      if (withinTopRight(keypoint[0], keypoint[1])) {
        area[2] += 1;
      }
      if (withinMiddleLeft(keypoint[0], keypoint[1])) {
        area[3] += 1;
      }
      if (withinMiddleCenter(keypoint[0], keypoint[1])) {
        area[4] += 1;
      }
      if (withinMiddleRight(keypoint[0], keypoint[1])) {
        area[5] += 1;
      }
      if (withinBottomLeft(keypoint[0], keypoint[1])) {
        area[6] += 1;
      }
      if (withinBottomCenter(keypoint[0], keypoint[1])) {
        area[7] += 1;
      }
      if (withinBottomRight(keypoint[0], keypoint[1])) {
        area[8] += 1;
      }
      // end of count
    }
    
    // print index
    for (let i = 0; i < area.length; i += 1) {
      if (area[i] == 21) {
        value = i;
      }
    }
  }
}

// returns true if a point is in a specific region
function withinTopLeft(x, y) {
  if (x >= 0 && x < (width / 2) / 3 && y >=0 && y < height / 3) {
    return true;
  }
  return false;
}

function withinTopCenter(x, y) {
  if (x > (width / 2) / 3  && x < 2 * (width / 2) / 3 && 
      y >=0 && y < height / 3) {
    return true;
  }
  return false;
}

function withinTopRight(x, y) {
  if (x > 2 * (width / 2) / 3 && x < (width / 2) && 
      y >=0 && y < height / 3) {
    return true;
  }
  return false;
}

function withinMiddleLeft(x, y) {
  if (x >= 0 && x < (width / 2) / 3 && y > height / 3 && y < 2 * height / 3) {
    return true;
  }
  return false;
}

function withinMiddleCenter(x, y) {
  if (x > (width / 2) / 3  && x < 2 * (width / 2) / 3 && 
      y > height / 3 && y < 2 * height / 3) {
    return true;
  }
  return false;
}

function withinMiddleRight(x, y) {
  if (x > 2 * (width / 2) / 3 && x < (width / 2) && 
      y > height / 3 && y < 2 * height / 3) {
    return true;
  }
  return false;
}

function withinBottomLeft(x, y) {
  if (x >= 0 && x < (width / 2) / 3 && y > 2 * height / 3 && y < height) {
    return true;
  }
  return false;
}

function withinBottomCenter(x, y) {
  if (x > (width / 2) / 3  && x < 2 * (width / 2) / 3 &&
      y > 2 * height / 3 && y < height) {
    return true;
  }
  return false;
}

function withinBottomRight(x, y) {
  if (x > 2 * (width / 2) / 3 && x < (width / 2) &&
      y > 2 * height / 3 && y < height) {
    return true;
  }
  return false;
}

Arduino sketch:

const int ain1Pin = 3;
const int ain2Pin = 4;
const int pwmAPin = 5;

const int bin1Pin = 8;
const int bin2Pin = 7;
const int pwmBPin = 6;


void setup() {
  // Start serial communication so we can send data
  // over the USB connection to our p5js sketch
  Serial.begin(9600);

  pinMode(LED_BUILTIN, OUTPUT);

  pinMode(ain1Pin, OUTPUT);
  pinMode(ain2Pin, OUTPUT);
  pinMode(pwmAPin, OUTPUT); // not needed really

  pinMode(bin1Pin, OUTPUT);
  pinMode(bin2Pin, OUTPUT);
  pinMode(pwmBPin, OUTPUT); // not needed really

  // TEST BEGIN
  // turn in one direction, full speed
  analogWrite(pwmAPin, 255);
  analogWrite(pwmBPin, 255);
  digitalWrite(ain1Pin, HIGH);
  digitalWrite(ain2Pin, LOW);
  digitalWrite(bin1Pin, HIGH);
  digitalWrite(bin2Pin, LOW);

  // stay here for a second
  delay(1000);

  // slow down
  int speed = 255;
  while (speed--) {
    analogWrite(pwmAPin, speed);
    analogWrite(pwmBPin, speed);
    delay(20);
  }

  // TEST END

  // start the handshake
  while (Serial.available() <= 0) {
    digitalWrite(LED_BUILTIN, HIGH); // on/blink while waiting for serial data
    Serial.println("0,0"); // send a starting message
    delay(300);            // wait 1/3 second
    digitalWrite(LED_BUILTIN, LOW);
    delay(50);
  }

}

void loop() {

  while (Serial.available()) {
    // sends dummy data to p5
    Serial.println("0,0");

    // led on while receiving data
    digitalWrite(LED_BUILTIN, HIGH); 

    // gets value from p5
    int value = Serial.parseInt();

    // changes brightness of the led
    if (Serial.read() == '\n') {
      
      if (value >= 0 && value <= 2) {
        digitalWrite(ain1Pin, HIGH);
        digitalWrite(ain2Pin, LOW);
        digitalWrite(bin1Pin, HIGH);
        digitalWrite(bin2Pin, LOW);

        analogWrite(pwmAPin, 255);
        analogWrite(pwmBPin, 255);

      }

      else if (value >= 3 && value <= 5) {
        analogWrite(pwmAPin, 0);
        analogWrite(pwmBPin, 0);
      }

      else {
        digitalWrite(ain1Pin, LOW);
        digitalWrite(ain2Pin, HIGH);
        digitalWrite(bin1Pin, LOW);
        digitalWrite(bin2Pin, HIGH);

        analogWrite(pwmAPin, 255);
        analogWrite(pwmBPin, 255);
      }
    }
  }
  // led off at end of reading
  digitalWrite(LED_BUILTIN, LOW);
}

Video:

Final Draft 1

Concept:

I’ve tentatively decided on a concept for my final project, though I’m not completely set on it yet. I’m considering creating a reaction time game centered around colors, featuring various levels of difficulty. The project involves a platform housing colored buttons capable of lighting up, with two buttons assigned to each of the four colors.

In the first level, a color will display on the screen, prompting the corresponding button to light up. Clicking that button reveals another color, illuminating the alternative button. The subsequent level increases the challenge by lighting up only one of the two buttons of the same color. For the third level, the screen will exhibit the name of the color in a different hue, requiring the player to press the button matching the displayed color rather than its name. This challenge can be reversed, prompting the player to press based on the text rather than the color, adding complexity to the game. a game will be around 10 lights color combination.

p5 interface

Regarding the interface, there will be a welcome screen on the p5 platform featuring a play button as well as an info button. After the player has read the instructions they click the play button. Clicking this will lead to a page displaying multiple levels for the user to select and start the game.

Components:

    • Arduino uno
    • 8 buttons
    • rigid body for the buttons
    • lots of jumper wirds

Arduino (Outgoing Data):

the ardunio would send feedback to p5 about the status of the buttons if pressed or not.

p5.js (Outgoing Commands):

during specific levels, the p5 will signal the Arduino to activate the light for the targeted button.

sketch:

 

Pet the Plant

Concept:
“Pet the Plant” is an interactive project that merges the physical and digital realms to provide users with the satisfaction of nurturing a virtual plant without the responsibilities and risks associated with real plants. Inspired by the creator’s struggles in maintaining live plants, the project employs Arduino and various sensors to simulate the physical actions of caring for a plant, while the digital aspect, coded in p5.js, reflects the plant’s growth.

Components:
1. Arduino Sensors:
– Potentiometer: Used to select from five different virtual plants.
– Force Resistor Sensor: Mimics soil patting motions by squeezing between fingers.
– Water Level Sensor: Allows users to virtually water the plant by pouring water into a cup.
– Photocell: Reads ambient “sunlight” affecting the virtual plant’s size and cloud visibility.

2. Digital Interface:
– Coded in C++ for Arduino and p5.js for the virtual experience.
– Utilizes p5.serialcontrol for communication between Arduino and p5.js.
– Screenshots in a GIF format showcase various scenes of the digital experience.

“Plant the Pet” Set-Up:
The physical interface is designed to align with the plant theme:
– Arduino and wires hidden inside a shoebox with a connecting hole to the laptop.
– Four holes on top for the sensors, placed inside small plant pots.
– Brown air-dry clay serves as simulated dirt, covered with fake grass to enhance the plant-like appearance.

Smart Plant Growth Monitor:
This project introduces a hands-on experience for users to monitor virtual plant growth on a computer screen. Key components include:
– Physical Soil Sensor: Arduino-powered soil moisture sensor.
– Digital Plant Simulation: Developed using P5.js for visual appeal.
– Real-time Data Exchange: Arduino collects and sends soil moisture data to the P5.js environment.
– User Interaction: Users physically water the virtual plant by adding water to the soil sensor.
– Feedback System: Visual feedback in the P5.js simulation reflects plant growth based on soil moisture levels.

Arduino & P5.js Outgoing Ongoing Data:
The Arduino collects real-time soil moisture data and transmits it to the P5.js environment. This ongoing data exchange facilitates a dynamic and responsive virtual plant simulation, allowing users to observe the impact of their actions on the plant’s growth in real-time. The interaction is designed to be user-friendly, creating an engaging and educational experience in the realm of virtual gardening.

 

Arduino Code

int pot; //potentiometer
int fsr; // force sensitive resistor
int humid; // humidity & temperature level sensor
int light; // photocell

void setup() {
  Serial.begin(9600);
}

void loop() {
  pot = analogRead(A0);
  fsr = analogRead(A1);
  humid = analogRead(A2);
  light = analogRead(A3);
  Serial.print(pot);
  Serial.print(',');
  Serial.print(fsr);
  Serial.print(',');
  Serial.print(humid);
  Serial.print(',');
  Serial.print(light);

}