Author: Hassan Hamdani

Hassan, Aaron, Majid.

CONCEPT

How would someone virtually learn how complicated it is to drive a car? Would teaching someone how to drive a car virtually save a lot of money and decrease potential accidents associated with driving? These questions inspired our project, which is to create a remote-controlled car that can be controlled using hand gestures (that imitate the driving steering wheel movements), specifically by tracking the user’s hand position and a foot pedal. The foot pedal will be used to control the acceleration, braking, and reversing. We will achieve all these by integrating a P5JS tracking system into the car, which will interpret the user’s hand gestures and translate them into commands that control the car’s movements. The hand gestures and pedal control will be synced together via two serial ports that will communicate with the microcontroller of the car.

Experience

The entire concept is not based only on a driving experience. We introduce a racing experience by creating a race circuit. The idea is for a user to complete a lap in the fastest time possible. Before you begin the experience, you can view the leaderboard. After your time has been recorded, a pop-up appears for you to input your name to be added to the leaderboard. For this, we created a new user interface on a separate laptop. This laptop powers an Arduino circuit connection which features an ultrasonic sensor. The ultrasonic sensor checks when the car has crossed the start line and begins a timer, and detects when the user ends the circuit. After this, it records the time it took a user to complete the track and sends this data to the leaderboard.

This piece of code is how we’re able to load and show the leaderboard.

function loadScores() {
let storedScores = getItem("leaderboard");
if (storedScores) {
highscores = storedScores;
console.log("Highscores loaded:", highscores);
} else {
console.log("No highscores found.");
}
}

function saveScores() {
// make changes to the highscores array here...
storeItem("leaderboard", highscores);
console.log("Highscores saved:", highscores);
}

 

IMPLEMENTATION(The Car & Foot Pedal)

We first built the remote-controlled car using an Arduino Uno board, a servo motor, a Motor Shield 4 Channel L293D, an ultrasonic sensor, 4 DC motors, and other peripheral components. Using the Motor Shield 4 Channel L293D decreased numerous wired connections and allowed us space on the board on which we mounted all other components. After, we created a new Arduino circuit connection to use the foot pedal. 

The foot pedal sends signals to the car by rotating a potentiometer whenever the pedal is engaged. The potentiometer value is converted into forward/backward movement before it reaches p5.js via serial communication.

P5/Arduino Communication

At first, a handshake is established to ensure communication exists before proceeding with the program:

//////////////VarCar/////
// Define Serial port
let serial;
let keyVal;
//////////////////////////
const HANDTRACKW = 432;
const HANDTRACKH = 34;

const VIDEOW = 320;
const VIDEOH = 240;

const XINC = 5;
const CLR = "rgba(200, 63, 84, 0.5)";

let smooth = false;
let recentXs = [];
let numXs = 0;

// Posenet variables
let video;
let poseNet;

// Variables to hold poses
let myPose = {};
let myRHand;

let movement;
////////////////////////

let Acceleration = 0;
let Brake = 0;
let data1 = 0;
let data2 = 0;

let s2_comp=false;

function setup() {
// Create a canvas
//createCanvas(400, 400);

// // Open Serial port
// serial = new p5.SerialPort();
// serial.open("COM3"); // Replace with the correct port for your Arduino board
// serial.on("open", serialReady);

///////////////////////////
// Create p5 canvas

The Hand Gestures

Two resources that helped detect the user’s hand position were PoseNet and Teachable Machine. We used these two resources to create a camera tracking system which was then programmed to interpret specific hand gestures, such as moving the hand right or left to move the car in those directions. This aspect of our code handles the hand tracking and gestures.

if (myPose) {
try {
// Get right hand from pose
myRHand = getHand(myPose, false);
myRHand = mapHand(myRHand);

const rangeLeft2 = [0, 0.2 * HANDTRACKW];
const rangeLeft1 = [0.2 * HANDTRACKW, 0.4 * HANDTRACKW];
const rangeCenter = [0.4 * HANDTRACKW, 0.6 * HANDTRACKW];
const rangeRight1 = [0.6 * HANDTRACKW, 0.8 * HANDTRACKW];
const rangeRight2 = [0.8 * HANDTRACKW, HANDTRACKW];

// Check which range the hand is in and print out the corresponding data
if (myRHand.x >= rangeLeft2[0] && myRHand.x < rangeLeft2[1]) {
print("LEFT2");
movement = -1;
} else if (myRHand.x >= rangeLeft1[0] && myRHand.x < rangeLeft1[1]) {
print("LEFT1");
movement = -0.5;
} else if (myRHand.x >= rangeCenter[0] && myRHand.x < rangeCenter[1]) {
print("CENTER");
movement = 0;
} else if (myRHand.x >= rangeRight1[0] && myRHand.x < rangeRight1[1]) {
print("RIGHT1");
movement = 0.5;
} else if (myRHand.x >= rangeRight2[0] && myRHand.x <= rangeRight2[1]) {
print("RIGHT2");
movement = 1;
}
// Draw hand
push();
const offsetX = (width - HANDTRACKW) / 2;
const offsetY = (height - HANDTRACKH) / 2;
translate(offsetX, offsetY);
noStroke();
fill(CLR);
ellipse(myRHand.x, HANDTRACKH / 2, 50);
pop();
} catch (err) {
print("Right Hand not Detected");
}
print(keyVal)

 

The Final Result & Car Control

The final result was an integrated system consisting of the car, the pedal, and gesture control in P5.JS. When the code is run in p5.js, the camera detects a user’s hand position and translates it into movement commands for the car.

The entire code for controlling the car.

//////////////VarCar/////
// Define Serial port
let serial;
let keyVal;
//////////////////////////
const HANDTRACKW = 432;
const HANDTRACKH = 34;

const VIDEOW = 320;
const VIDEOH = 240;

const XINC = 5;
const CLR = "rgba(200, 63, 84, 0.5)";

let smooth = false;
let recentXs = [];
let numXs = 0;

// Posenet variables
let video;
let poseNet;

// Variables to hold poses
let myPose = {};
let myRHand;

let movement;
////////////////////////

let Acceleration = 0;
let Brake = 0;
let data1 = 0;
let data2 = 0;

let s2_comp=false;

function setup() {
// Create a canvas
//createCanvas(400, 400);

// // Open Serial port
// serial = new p5.SerialPort();
// serial.open("COM3"); // Replace with the correct port for your Arduino board
// serial.on("open", serialReady);

///////////////////////////
// Create p5 canvas
createCanvas(600, 600);
rectMode(CENTER);

// Create webcam capture for posenet
video = createCapture(VIDEO);
video.size(VIDEOW, VIDEOH);
// Hide the webcam element, and just show the canvas
video.hide();

// Posenet option to make posenet mirror user
const options = {
flipHorizontal: true,
};

// Create poseNet to run on webcam and call 'modelReady' when model loaded
poseNet = ml5.poseNet(video, options, modelReady);

// Everytime we get a pose from posenet, call "getPose"
// and pass in the results
poseNet.on("pose", (results) => getPose(results));
}

function draw() {
// one value from Arduino controls the background's red color
//background(0, 255, 255);

/////////////////CAR///////
background(0);

strokeWeight(2);
stroke(100, 100, 0);
line(0.2 * HANDTRACKW, 0, 0.2 * HANDTRACKW, height);
line(0.4 * HANDTRACKW, 0, 0.4 * HANDTRACKW, height);
line(0.6 * HANDTRACKW, 0, 0.6 * HANDTRACKW, height);
line(0.8 * HANDTRACKW, 0, 0.8 * HANDTRACKW, height);
line(HANDTRACKW, 0, HANDTRACKW, height);
line(1.2 * HANDTRACKW, 0, 1.2 * HANDTRACKW, height);

if (myPose) {
try {
// Get right hand from pose
myRHand = getHand(myPose, false);
myRHand = mapHand(myRHand);

const rangeLeft2 = [0, 0.2 * HANDTRACKW];
const rangeLeft1 = [0.2 * HANDTRACKW, 0.4 * HANDTRACKW];
const rangeCenter = [0.4 * HANDTRACKW, 0.6 * HANDTRACKW];
const rangeRight1 = [0.6 * HANDTRACKW, 0.8 * HANDTRACKW];
const rangeRight2 = [0.8 * HANDTRACKW, HANDTRACKW];

// Check which range the hand is in and print out the corresponding data
if (myRHand.x >= rangeLeft2[0] && myRHand.x < rangeLeft2[1]) {
print("LEFT2");
movement = -1;
} else if (myRHand.x >= rangeLeft1[0] && myRHand.x < rangeLeft1[1]) {
print("LEFT1");
movement = -0.5;
} else if (myRHand.x >= rangeCenter[0] && myRHand.x < rangeCenter[1]) {
print("CENTER");
movement = 0;
} else if (myRHand.x >= rangeRight1[0] && myRHand.x < rangeRight1[1]) {
print("RIGHT1");
movement = 0.5;
} else if (myRHand.x >= rangeRight2[0] && myRHand.x <= rangeRight2[1]) {
print("RIGHT2");
movement = 1;
}
// Draw hand
push();
const offsetX = (width - HANDTRACKW) / 2;
const offsetY = (height - HANDTRACKH) / 2;
translate(offsetX, offsetY);
noStroke();
fill(CLR);
ellipse(myRHand.x, HANDTRACKH / 2, 50);
pop();
} catch (err) {
print("Right Hand not Detected");
}
print(keyVal)

//print(movement);
// print("here")
//print(writers);
}
//////////////////////////

if (!serialActive1 && !serialActive2) {
text("Press Space Bar to select Serial Port", 20, 30);
} else if (serialActive1 && serialActive2) {
text("Connected", 20, 30);

// Print the current values
text("Acceleration = " + str(Acceleration), 20, 50);
text("Brake = " + str(Brake), 20, 70);
mover();
}

}

function keyPressed() {
if (key == " ") {
// important to have in order to start the serial connection!!
setUpSerial1();
} else if (key == "x") {
// important to have in order to start the serial connection!!
setUpSerial2();
s2_comp=true
}
}

/////////////CAR/////////
function serialReady() {
// Send initial command to stop the car
serial.write("S",0);
print("serialrdy");
}

function mover() {
print("mover");

// Send commands to the car based wwon keyboard input
if (Acceleration==1) {
writeSerial('S',0);

//print(typeof msg1)

}else if (Brake==1) {
writeSerial('W',0);
}else if ( movement < 0) {
print("left")
writeSerial('A',0);
}else if ( movement > 0) {
print("right")

writeSerial('D',0);
}else if (movement== 0) {
print("stop");
writeSerial('B',0);
}
}

// When posenet model is ready, let us know!
function modelReady() {
console.log("Model Loaded");
}

// Function to get and send pose from posenet
function getPose(poses) {
// We're using single detection so we'll only have one pose
// which will be at [0] in the array
myPose = poses[0];
}

// Function to get hand out of the pose
function getHand(pose, mirror) {
// Return the wrist
return pose.pose.rightWrist;
}

// function mapHand(hand) {
//   let tempHand = {};
//   tempHand.x = map(hand.x, 0, VIDEOW, 0, HANDTRACKW);
//   tempHand.y = map(hand.y, 0, VIDEOH, 0, HANDTRACKH);

//   if (smooth) tempHand.x = averageX(tempHand.x);

//   return tempHand;
// }
function mapHand(hand) {
let tempHand = {};
// Only add hand.x to recentXs if the confidence score is greater than 0.5
if (hand.confidence > 0.2) {
tempHand.x = map(hand.x, 0, VIDEOW, 0, HANDTRACKW);

if (smooth) tempHand.x = averageX(tempHand.x);
}

tempHand.y = map(hand.y, 0, VIDEOH, 0, HANDTRACKH);

return tempHand;
}

function averageX(x) {
// the first time this runs we add the current x to the array n number of times
if (recentXs.length < 1) {
console.log("this should only run once");
for (let i = 0; i < numXs; i++) {
recentXs.push(x);
}
// if the number of frames to average is increased, add more to the array
} else if (recentXs.length < numXs) {
console.log("adding more xs");
const moreXs = numXs - recentXs.length;
for (let i = 0; i < moreXs; i++) {
recentXs.push(x);
}
// otherwise update only the most recent number
} else {
recentXs.shift(); // removes first item from array
recentXs.push(x); // adds new x to end of array
}

let sum = 0;
for (let i = 0; i < recentXs.length; i++) {
sum += recentXs[i];
}

// return the average x value
return sum / recentXs.length;
}

////////////////////////

// This function will be called by the web-serial library
// with each new *line* of data. The serial library reads
// the data until the newline and then gives it to us through
// this callback function
function readSerial(data) {
////////////////////////////////////
//READ FROM ARDUINO HERE
////////////////////////////////////

if (data != null) {
// print(data.value);
let fromArduino = data.value.split(",");
if (fromArduino.length == 2) {
//print(int(fromArduino[0]));
//print(int(fromArduino[1]));
Acceleration = int(fromArduino[0]);
Brake  = int(fromArduino[1])

}

//////////////////////////////////
//SEND TO ARDUINO HERE (handshake)
//////////////////////////////////
if(s2_comp){
let sendToArduino = Acceleration + "," + Brake + "\n";
// mover()
//print("output:");
//print(sendToArduino);
//writeSerial(sendToArduino, 0);
}
}
}

 

The car moves forward/backward when a user engages the foot pedals, and steers left/right when the user moves his/her hand left/right. The components of the car system are able to communicate via serial communication in p5.js. To enable this, we created 2 serial ports(one for the car and the other for the foot pedal). 

 

CHALLENGES

One challenge we may face during the implementation is accurately interpreting the user’s hand gestures. The camera tracking system required a lot of experimentation and programming adjustments to ensure that it interprets the user’s hand movements while also being light and responsive. Originally the camera was tracking the X and Y axis, but it caused p5 to be very slow and laggy because of the number of variables that it needs to keep track of. The solution was to simply remove one of the axes, this improved the responsiveness of the program drastically.

 

The initial plan was to operate the car wirelessly, however, this was not possible due to many factors, such as using the wrong type of Bluetooth board. With limited time, we resorted to working with two serial ports for communication between the car, the pedals, and the hand gesture control. This introduced a new problem- freely moving the car. However, we solved the issue by using an Arduino USB extension cable for the car to be able to move freely. To allow users at the showcase to be able to move the car around without the car tripping over the wire, we came up with an ingenious solution to tie a fishing line across the pillars of the Art Center where we were set-up, and then use our extended USB cable to basically hang of the fishing line, so it acted like a simulated roof for our setup. This way, the car could freely roam around the circuit and never trip.

 

Another major roadblock was the serial ports in p5js. Since the project uses both a pedal and the car, there was the need to use 2 separate Arduino Uno boards to control both systems. This necessitated the use of 2 serial ports in p5js. The original starter code for connecting p5 to Arduino was only for 1 serial port. A lot of time was spent adjusting the existing code to function with 2 serial ports. 

 

Lessons learned, especially with regard to robots, is be grouped into the following points: 

Planning is key: The project can quickly become overwhelming without proper planning. It’s important to define the project goals, select appropriate devices and how to sync the code to those devices and create a detailed project plan. 

Test as often as you can before the showcase date: Testing is crucial in robotics projects, especially when dealing with multiple hardware components and sensors. This one was no exception. It’s important to test each component and module separately before combining them into the final project.

Future steps needed to take our project to the next level.

1. Expand functionality: While the current design allows for movement in various directions, there are other features that could be added to make the car more versatile. We plan on adding cameras and other sensors(LiDar) to detect obstacles to create a mapping of an environment while providing visual feedback to a user.
2. Optimize hardware and software: We also plan on optimizing the hardware and software components used. This would involve changing the motors to more efficient or powerful motors, using more accurate sensors (not using the ultrasonic sensor), or exploring other microcontrollers that can better handle the project’s requirements. Additionally, optimizing the software code can improve the car’s responsiveness and performance. For example, our software code can detect obstacles but cannot detect the end of a path.  Regardless, we believe we can engineer a reverse obstacle-sensing algorithm to create an algorithm that could detect cliffs and pot-holes, and dangerous empty spaces on roads to ultimately reduce road accidents.

Demonstration

Concept

Our project is to create a remote-controlled car that can be controlled using hand gestures, specifically by tracking the user’s hand position. We will achieve this by integrating a P5JS tracking system into the car, which will interpret the user’s hand gestures and translate them into commands that control the car’s movements.

Implementation

To implement this project, we will first build the remote-controlled car using an Arduino Uno board and other necessary components. We will then integrate the P5JS system, which will detect the user’s hand position and translate it into movement commands for the car. We have two options for detecting user’s hand position, either PoseNet or Teachable Machine.  The camera tracking system will be programmed to interpret specific hand gestures, such as moving the hand forward or backward to move the car in those directions.

Potential challenges

One potential challenge we may face during the implementation is accurately interpreting the user’s hand gestures. The camera tracking system may require experimentation and programming adjustments to ensure that it interprets the user’s hand movements accurately. Also testing will be required to see if PoseNet performs better or Teachable Machine. Responsiveness will also be a factor as we would want the control to be fluid for the user. Additionally, ensuring the car responds correctly to the interpreted movement commands may also present a challenge, in terms of the physical construction of the cars and the variability of the motors. 

Prompt : “take the gravity wind example and make it so every time the ball bounces one led lights up and then turns off, and you can control the wind from one analog sensor”

 

BuzzBoard By Hassan and Majid

Concept

For our project, we aimed to create a unique musical instrument using distance tracking sensors and buttons. After brainstorming various ideas, we were inspired to design a piano-like instrument that could be played by two hands, one is distance tracked and the other is pressing the buttons. There are 7 notes and 3 different octaves that can be played. The notes are determined by the distance and the octave is based on which button is pressed.

implementation

We decided to use ultrasonic sensor for detecting the hand distance and buttons for controlling different notes of the piano. The Arduino was also connected to a piezo sensor for producing the piano sounds. Once the circuit assembly was complete, we tested the circuit to ensure the sensors and buttons were working and registered properly. Then we programmed the buttons to play different octaves of the notes.

Deciding which note to be played is at the core of our code. We did so through making a list of if conditions. First we checked what was the last button pressed, to decide the octave we are playing the notes in. After that, based on the distance from the ultrasonic sensor, in increments of 3cm, we chose a note to be played. For example, here is how the notes are played if the last button pressed was 2:

  if(lastButton == 2){
  if (distance < 3) {
    tone(8,NOTE_A4,noteDuration);
  } else if (distance >= 3 && distance < 6) {
    tone(8,NOTE_B4,noteDuration);
  } else if (distance >= 6 && distance < 9) {
    tone(8,NOTE_C4,noteDuration);
  } else if (distance >= 9 && distance < 12) {
    tone(8,NOTE_D4,noteDuration);
  }
  else if (distance >= 12 && distance < 15) {
    tone(8,NOTE_E4,noteDuration);
  }
      else if (distance >= 15 && distance < 18) {
    tone(8,NOTE_F4,noteDuration);
  }
      else if (distance >= 18 && distance < 21) {
    tone(8,NOTE_G4,noteDuration);
  }
}

Challenges

One of the main challenges we encountered during the project was calibrating the distance sensors to play different notes. We had to experiment with different threshold values and distances for the different notes and octaves.

The Demo

The Concept

The circuit is meant to represent a form of smart plant watering system. The light sensor is meant to detect the level of hypothetical sunlight, and the switch is meant to be pressed when the hypothetical soil is dried out. When the switch is held down, there is a third LED that shows that the plants are being watered, and the brightness of this LED slowly fades out, representing that the plants have been sufficiently watered. The initial brightness of this LED is based on the reading from the light sensor, to portray that you will have to water less if the sunlight isn’t as strong. Also, if the switch isn’t held down until this LED completely fades out, it will stay on to show that more watering still needs to be down.

The circuit

The circuit consists of a light sensor hooked to the Yellow LED, and the brightness of this LED is mapped to the reading from the light sensor

if(lightValue<=400){brightness=0;}
else{brightness = map(lightValue, 400, 1023, 0, 255);}

I realized even if I fully covered the light sensor with my hand, the reading would still be around 350-400. Thus, instead of directly mapping the input from the sensor to the output for the LED brightness, I set a threshold at 400 and below, which means that at 400 I want the light to be off, as this is a negligible amount of light.

The circuit then consists of a switch hooked to the Green LED to represent whether or not the switch is held down. This LED is digital and simply turns on and off.

There is also a third Blue LED, and this is an analog LED that is impacted by both the light sensor and the switch. The LED’s brightness is determined by the variable waterLevel, and this waterLevel is slowly decremented while the switch is pressed down.

if(switchValue==1)
{
  if(waterLevel>0){
    waterLevel --; 
  }
  analogWrite(waterLEDPin, waterLevel);

The initial value of waterLevel is set by the reading of the light sensor. Therefore, both sensors contribute to the behavior of this Blue LED.

The Demo