Final Project – Design and Description

Concept:

The main concept of this project is to have a in-position interactive robot that can talk with, think and reply to the user. Furthermore, a conversation can influence the robot’s mood depending on which its outward interactions will change. These moods will be chosen from a series of 4-5 discrete moods. The outward movements that move include a head that turns towards the user depending on where they are talking from, ears that move when you pat them. Eyes that blink as the character talks, changing facial expressions and lights that indicate the robot’s current mood. Finally, I incorporate minimal movement using 2 DC motors.

P5 Design:

The code uses the p5.Speech library for speech recognition. the ElevenLabs API for realistic text to speech and the OpenAI ChatGPT library for text generation and reasoning. We use regular expressions to interpret the prompt engineered ChatGPT responses, and maintain a list of moods and a class that consists of actions based on every mood type. Finally, the remaining code handles the handshake mechanism between the Arduino and p5. The program will receive light sensor/distance sensor data from the Arduino, and various button related state data such as that to start a new conversation.

Arduino Design:

The Arduino design is relatively simple with Servo motors for head movement, eye blinking and facial expression changes. I use LED lights as is, but may choose to use different lights for a more aesthetic result. I use various sensors such as the light sensor on the head to get results when we attempt to pat the robot and distance/light sensors on the body to get information about how far a person’s hand/other objects are. The Arduino passes all the relevant sensor data to the p5 program and gets all relevant data from the program. Some minimal interactions such as basic ear movements or movements motivated by proximity of any sort are handled directly within the Arduino itself.

Week 12 – Final Project Proposal and Design Documentation

Concept

For the final project, I have committed to the digital petting zoo idea, in which the user can interact with different animals on screen with one physical model and other props.

Design Documentation

p5: In this project, p5js will primarily be the user interface of the petting zoo. It will show the mapping of the zoo, the animals, instructions, and the need of different animals.

The general appearance should represent something like this (but surely a simplistic version):

Let's Build A Zoo review: an absorbing tycoon game that relishes chaos | Rock Paper Shotgun

This interface will be composed of several colliders and the user can walk around and trigger different interactions. A wireframe of this idea is live on p5js now: The big blocks are animal areas and the small one is the player. The player now can walk around with the arrow keys on the keyboard and different messages will be displayed on the LCD screen connected to the Arduino (will explained below) when the player collides with different blocks, showing that they are entering different animal areas.

Arduino: The Arduino will be the controller of the player and the place will physical interactions with the animals on the screen could happen. I plan to craft a model that represents the animal that will be in the zoo with a LCD display that will tell the player which animal it is based on the location of the player in the zoo. There might be other parts connected to this model, such as servo motors and other sensors, which make the interactions possible. There will also be a control panel with four buttons, allowing the player to walk in the zoo, just like the arrow buttons on the keyboard.

week11.reading – Design Meets Disability

In this chapter from Graham Pullin’s book Design Meets Disability, the author delves into the history, complications, applications, and interplay between designs and life-enhancing devices. I believe the main ideas that Pullin is trying to address are highly evident through the examples of eyewear. As mentioned, the concept of physical lenses was never initially intended to be a part of fashion nor to embrace the physical complications. Nevertheless, with time, glasses became a significant part of fashion, even though their main purpose was to enhance the vision and life of those who have physical limitations in regard to their eyesight. The chapter discusses numerous such instances where engineers and scientists develop systems without an idea of the physical design in mind, but society can gradually adapt and change these systems, integrating them seamlessly into our lives.

However, it is critical to comprehend that a balance must exist between design and functionality. Certainly, with devices such as hearing aids, one cannot focus fully on the best visual design and neglect the functionality of the product. Similarly, this ideology influences the progressive and innovative technologies that we see today. When creating such systems, individuals must also consider what are the societal, cultural, and other external influences that will either enhance or diminish the need or usefulness of the product. Ultimately, I believe that this reading has given me a new perspective on how design meets disability and how societies are able to remarkably influence the adaptation and usefulness of products that might have been designed with no intention of including a good visual design.

week11.assignment – Final Project Ideas

Concepts

When we first began working with the Arduino, I was excited about all the possibilities that arose for my final project. Additionally, I found working with Arduino quite intuitive and am hoping to implement unique features for the final project. As of now, there are two main ideas that I have in mind.

Idea 1 – Virtual Arcade with a Twist

My first idea for the final project revolves around creating a virtual arcade, using P5, and implementing the control for the minigames using analog and digital sensors with Arduino. The p5 sketch would present a virtual arcade with at least two or three different minigames that you would be able to select using a physical component, for instance, a button with an arrow right to highlight the object on the right or left and vice-versa. Once the player decides on what game they would like to play by pressing the select button (digital switch), the game screen would change to that of the selected minigame. As of now, I have two interesting and unique ideas for the minigames. The first is a game where a character has a jetpack and is stuck in a box. Randomly, different objects will be generated and move horizontally at different speeds. The character will be controlled by a sound level meter, and the louder the input of the player, the higher the character will float on the screen. Each object avoided is 1 point. After the player dies, the score will be saved as a high school and displayed for other players to try to beat. The next minigame idea I have is to create a Wack-a-Mole but with a twist. In this version of the game, there will be an array of digital switches and LEDs above them. However, every round, two of the buttons will light up, with different colors. This would make the game more challenging as the person would have to press the button with the correct color. If the player presses the button which has a different color, then a point is deducted from the score.

Idea 2 – Unique Controller

My first idea might take a lot of time to fully implement, and if that were the case, then my other idea would be to create just one game that would rely on the input of a unique control. This idea was inspired by the lack of innovation that I have noticed when it comes to gaming console controllers. Evidently, the current controllers are popular because they are ergonomic and intuitive. However, I believe that it would be interesting to explore and try to create a unique controller that relies on different inputs for controlling a game. One such unique controller could be a special glove that uses flex sensors to determine the bending of finders in a hand. A game that could go well with this would be a Plants vs. Zombies type of game, in which a character can be controlled to move on the vertical axis and could shoot projectiles to stop the incoming enemies from reaching the other side of the screen, which would result in a loss of health points. Additionally, the game would have waves of increasing difficulty, in which the enemies would move faster, and there would be more enemies in total. The hand controls for this game would involve bending your fingers into different positions, which would indicate movement and actions for the game.

Possible Challenges

For my first idea, I believe the main challenge would be time and physically constructing the different sensors and interfaces for the minigames. Even though I am confident this idea is possible to implement, I am not too confident about the amount of time that I have to make it happen. This makes me lean towards implementing my second idea. For the second idea, I believe the biggest challenges would be ensuring the accuracy of the flex sensors and creating a well-designed glove for the control of the game, as well as ensuring that the gesture recognition works well. As I am writing this, I have also realized that a glove-like controller, like the one I would like to make, could be used as a translator for sign language. This implementation, of course, would also require additional elements such as an accelerometer and extremely precise readings from the flex sensors; however, this could be a unique opportunity for a future project.

Week 12 – Final Project Idea Proposal, Design & Description

Concept Breakdown:

My idea for the final project will be an interactive glove: a innovative physical way of interacting with the computer. In this glove, we will be able to use certain hand gestures to send input to our Arduino and later on, our computer.  In order to comply with the hardware limitations and time restrictions, I will not utilize the glove as a mouse, as the use of a trackpad will require utilizing a board different than Arduino Uno (For example, the Arduino Micro, Pro Micro or Leonardo).

The Hand Gestures:

The hand gestures utilized will require touching the border of the palm (the other side if we check our knuckles) of our hand with a finger, regardless of which one we decide to use. Try it. Some people will be able to use it with one hand, some people will not. his means we will be able to use either the fingers in the same hand or the other hand. How does this work?

The user will use either one hand or both to control the input, which will be similar to a binary system: 0001. In this case, what we do is utilize the border right under every finger except for the thumb as one of the spaces for the input. This means that touching every part of the palm in different combinations will hold a different output (or input, from the Arduino’s point of view).

This leaves us with 2^4 different combinations that we can map to certain keyboard inputs because remember, a touchpad/mouse-style input requires a different board and honestly, many more inputs.

The Arduino Circuit and Design:

The circuit design is simple but effective. The idea will be to connect cables from the Arduino ports to the border of the palm. By connecting 5v current to all our fingertips, what we will be able to do is effectively create an array of switches that will trigger on our touch. We will attach all the necessary cables to the glove and cover the exterior to avoid disconnections and other problems.  Important to note that we will have to be aware of possible delays and misinputs, as it is a form of physical input.

Interactive Gloves : 8 Steps (with Pictures) - Instructables

The P5js sketch:

The P5js Sketch will be a way of demonstrating the inputs of the glove. We will utilize many of the previous projects we have done, such as the piano, the bouncing ball, the musical instrument and other little games in order to show the public that it is possible to control the input differently.

The Connection:

As much as the communication between p5js and Arduino should be obvious at this point, we will compute the inputs in the circuit before sending a single “key” input to p5js. This means that we have no delay in terms of the computation nor we will need to send more information than necessary. A great addition would be also for p5js to send a response to Arduino. For example, the piano could have its output coming from a speaker.

Week 12 – Final Project Description

FINALIZED CONCEPT

I want to make a Mastermind game, which is a two-player board game where one person is the code maker, and they make a code using colored pegs. The other person is the code breaker, and they have to figure out the code in a limited number of moves. The less moves you take to figure out the code, the higher you score.

Mastermind - Code Breaking Game – Kubiya Games

Mastermind

Originally, I was going to have wooden floats that would complete a circuit through conductive water. However, thinking that would make a mess and potentially be hazardous due to being close to electronics, I switched to using LDRs and LEDs to detect different colors.

The structure will be made using wood of the following dimensions:

I want it to be an enclosed box, so that the readings on the LDR are stable. My idea is that I can use LEDs of different colors, and different brightnesses to “detect” color, and then use that as input for the Mastermind game built in P5Js.

Arduino to P5Js Communication

The pegs will be the interface of communication, and placing a colored peg into one of the holes should communicate what color the peg is at the respective position. The sensor values that are measured are LDR readings, but I do not want to transmit raw sensor values. Instead, I will do the checks to figure out what color the peg is and in what position within Arduino. Then, I want to transmit the respective color positions to Arduino in the form of a code. There are 4 positions, and 5 colors. So, there are 6 options for each position, including having no peg. Therefore, I will generate a 4-digit code with digits 0-5, and that is what will be sent to P5Js.

Also, I want to add a button that is used to confirm the code selection. To convey that information, I can include a fifth digit in the code, with a 0 or a 1 to represent whether or not the button has been pressed. Any extra checks, like prevent the button from clicking more than once in a certain timeframe, will be processed within P5Js, so that communication is simplified.

P5Js to Arduino

Since my project is centered around an interesting control interface, there is nothing that I want to send from P5Js to Arduino, as the Arduino side is a game controller. Unless there needs to be some sort of syncing that needs to be done, which I don’t expect, there will be no communication back to Arduino from P5Js.

Ideas For Final Project

 

Idea 1 

For the first idea, I’m thinking of using either real-time data or just existing data and creating a creative visualization that can be controlled by the user through an ultrasonic sensor. My second option for data visualization is to apply the concept of sonification in the field of astronomy where data is converted into sound as an output demonstrating  some waves that are being radiated in space or a black hole studied by NASA. If I go with this idea, I’ll probably try to connect the sound of those waves with the user’s interaction with the ultrasonic sensor and connect it to what’s displayed on p5js.

Idea 2

My second idea does not involve any data its more like a game. The game would either involve a journey in an old house and the user will be moving through different rooms or it would be a whole night journey in multiple settings like a home, forest, streets, etc. The main concept is that they will be using their phones as a flashlight to search for certain objects asked by the game in order to win or complete the narrative. The light would be detected by the light sensor and it would be connected to p5js so the space they are in is bright enough for them to look for the item. This idea might be tough because I’m not sure to what extent it is plausible to implement.

Class Exercises

  1. make something that uses only one sensor  on Arduino and makes the ellipse in p5 move on the horizontal axis, in the middle of the screen, and nothing on arduino is controlled by p5.

Ex1 video demo

2. make something that controls the LED brightness from p5

Ex2 video demo

3. take the gravity wind example (https://editor.p5js.org/aaronsherwood/sketches/I7iQrNCul) and make it so every time the ball bounces one led lights up and then turns off, and you can control the wind from one analog sensor.

Ex3 video demo

Week 11 Exercises

Exercise 1: Move a circle in P5Js using a single Arduino Sensor

Idea:

My idea was to simply use a potentiometer, since I wanted to practice serial connection between Arduino and P5JS. Since a potentiometer gives stable input, I thought it would be the best to use it to test the waters.

P5Js Code:

let ardVal = 0.0; // global variable to store the incoming data
let circleRad = 25;
function setup() {
  createCanvas(400, 400);
   frameRate(60);
}

function draw() {
  background(220);
  
  //map the input from arduino to the position of the circle
  let xPos = map(ardVal, 0, 1023, circleRad, width - circleRad);
  
  noStroke();
  fill(255,0,0); //red ball
  ellipse(xPos, 200, 2 * circleRad, 2 * circleRad);

  if(!serialActive)
  {
    print("Serial is not active");
  }

}

// to establish the serial connection
function keyPressed() {
  if (key == " ") {
    // important to have in order to start the serial connection!!
    setUpSerial(9600);
    print("SERIAL ACTIVATED");
  }
}

// how we read the incoming data
function readSerial(data) {

  if (data != null) {
    // make sure there is actually a message
    ardVal = data //read the incoming data as a float
    }  
}

I wanted to display a simple red circle that moves across the screen based on the value of the potentiometer. Only the value of the potentiometer is sent from Arduino, and nothing is sent back to Arduino from P5Js.

Arduino Code:

#include <Arduino.h>

const int potInPin = A0;

void setup() {
  Serial.begin(9600); // 9600 bits per second
  pinMode(LED_BUILTIN, OUTPUT);
}

void loop() {
  int potValue = analogRead(potInPin);
  Serial.println(potValue);
}

The Arduino side of things is very simple. I read the value from the potentiometer and write it to the serial port.

Circuit:

The circuit simply takes input from the potentiometer using the A0 pin.

Exercise 2: Move a circle in P5Js using a single Arduino Sensor

Note:

For this exercise, I had issues getting serial communication from P5Js to Arduino to work. I tinkered for a few hours, but then ended up finding this P5Js serial library that seems to be simpler to use, and I was able to get communication up and running with this. I will be using this library for future projects, and the rest of the exercises.

It can be imported via adding this to your P5Js sketch, in the index.html file:

<script src="https://unpkg.com/@gohai/p5.webserial@^1/libraries/p5.webserial.js"></script>

Idea:

Once I had communication running, the idea I (actually my friend who was watching me tinker) came up with was to create a sensor that would let you know where mines in minesweeper were. Thus, this LED would serve as a mine sensor. For this, I needed to create a simple minesweeper game, and add the functionality of detecting mines via the LED in the Arduino circuit.

Instead of making it a minesweeper game though, I decided to turn the game into that of a mouse trying to detect mines on the field. You can press “d” to defuse if you are on a mine. However, if you press “d” on the wrong mine, you lose a turn. The game is won if you defuse all the bombs, and lost if you try to defuse incorrectly too many times. The mouse can be controlled via the arrow keys.

Note: Please open the sketch in a new tab

P5Js Code:

It was very easy to create the port for connection. All I had to do was create a serial port object in the setup() function as follows:

function setup() {
  createCanvas(400, 500);
  //create a 2D array
  grid = create2DArray(numRows, numCols);

  //setup the grid cells
  for (let i = 0; i < numRows; i++) {
    for (let j = 0; j < numCols; j++) {
      grid[i][j] = new Cell(i, j, gridWidth / numRows, gridHeight / numCols);
    }
  }

  //place the mines
  placeMines();

  //create the mouse
  bombMouse = new Mouse();

  //create the serial port
  port = createSerial()

}

And then I asked the user to press space to begin the game, and pressing space asks the user to connect to the correct serial device, in this case our Arduino.

function keyPressed() {
  //to open the serial port
  if (key == " ") {
    if(port.opened())
    {
      return; //already opened
    }
    print("ACTIVATED");
    port.open(9600);
  }

  if (key == "d") {
    bombMouse.defuse();
  }

}

Finally, sending input to Arduino from P5Js is very simple as well:

let distance = bombMouse.distanceFromClosestBomb();

  let message = String(distance) + "\n";

  //send the distance to the arduino
  port.write(message);
  print(distance + "vs" + message);

I send the message as a string, and parse the integers from the string in Arduino.

The rest of the code for the game can be viewed on the P5Js web editor linked above.

Arduino Code:

The Arduino code for this was very simple. Parse distance through Serial.parseInt(), and then map it to brightness values. If the distance is 0, AKA mouse is on a mine, then the LED should blink.

#include <Arduino.h>

const int LED_PIN = 5;

int lastToggleTime = 0;
int currentStatus = LOW;

void setup()
{
  pinMode(LED_PIN, OUTPUT);
  Serial.begin(9600);
}

void loop()
{

  if(Serial.available() == 0)
  {
    //if no data is available, wait
    return;
  }

  //read distance from P5Js
  int distance = Serial.parseInt();

  if(distance == 0)
  {
    if(millis() - lastToggleTime < 30)
    {
      //if less than 1 second has passed, wait
      return;
    }
    lastToggleTime = millis();

    if(currentStatus == LOW)
    {
      currentStatus = HIGH;
    }
    else
    {
      currentStatus = LOW;
    }

    //if distance is 0, start blinking
    digitalWrite(LED_PIN, currentStatus);

  }

  else{
      
  //map distance to brightness but only show values between 0 and 3
  float brightness = map(distance, 0, 8, 255, 0); // max distance is 4 + 4 = 8

  //set brightness
  analogWrite(LED_PIN, brightness);
  }
  
}

Circuit:

The circuit is a basic LED running circuit that takes input using a PWM pin (5).

Exercise 3:

This circuit takes input from the potentiometer to control the wind direction. Also, every time the ball hits the ground, a signal from P5Js is sent to Arduino so that the LED is turned on during the ball’s impact with the ground.

P5Js Code:

let velocity;
let gravity;
let position;
let acceleration;
let wind;
let drag = 0.99;
let mass = 50;

//port for arduino
let port;

function setup() {
  createCanvas(640, 360);
  noFill();
  position = createVector(width/2, 0);
  velocity = createVector(0,0);
  acceleration = createVector(0,0);
  gravity = createVector(0, 0.5*mass);
  wind = createVector(0,0);

  port = createSerial();
}

function draw() {
  if(!port.opened())
  {
    background(220, 200); 
    fill(0);
    textSize(25);
    text("PRESS SPACE TO START", 50, height/2);
    return;
  }

  //read the serial port
  if(port.available())
  {
    let inString = port.readUntil('\n');
    let potVal = -1;
    if(inString.length > 0)
    {
      potVal = int(inString);
      wind.x = map(potVal, 0, 1023, -1, 1);

    }
  }

  print("WIND: " + wind.x);



  background(255);
  applyForce(wind);
  applyForce(gravity);
  velocity.add(acceleration);
  velocity.mult(drag);
  position.add(velocity);
  acceleration.mult(0);
  noStroke();
  fill(255,0,0);
  ellipse(position.x,position.y,mass,mass);
  if (position.y > height-mass/2) {
      velocity.y *= -0.9;  // A little dampening when hitting the bottom
      position.y = height-mass/2;

      //send a signal to arduino to turn on the LED
      port.write("1\n");
    }

  else{
    port.write("0\n");
  }

  //clear the buffer
  port.clear();

  
}

function applyForce(force){
  // Newton's 2nd law: F = M * A
  // or A = F / M
  let f = p5.Vector.div(force, mass);
  acceleration.add(f);
}

function keyPressed(){
  if (keyCode==LEFT_ARROW){
    wind.x=-1;
  }
  if (keyCode==RIGHT_ARROW){
    wind.x=1;
  }
  if (key==' '){
    mass=random(15,80);
    position.y=-mass;
    velocity.mult(0);
  }

  if(key == " ")
  {
    port.open(9600); // open a port at 9600 baud
  }
}

 

Arduino Code:

#include <Arduino.h>

const int ledPin = 5;
const int potPin = A0;

int timeSinceLastSerial = 0;


void setup() {
  pinMode(ledPin, OUTPUT);
  pinMode(potPin, INPUT);
  digitalWrite(ledPin, LOW); //initially off

  Serial.begin(9600);

}

void loop() {
  int val;

  if(millis() - timeSinceLastSerial < 2) //if it has not been more than 20ms since last serial message
  {
    return; //do nothing
  }

  timeSinceLastSerial = millis(); //update time since last serial message

  //serial exchange

  if(Serial.available()) //if there is data to read
  {
    val = Serial.parseInt(); //read the input from serial
  }

  if(val == 1)
  {
    digitalWrite(ledPin, HIGH); //turn on LED
  }
  else
  {
    digitalWrite(ledPin, LOW); //turn off LED
  }

  Serial.println(analogRead(potPin)); //send potentiometer value across serial

}

Circuit:

The circuit is a combination of the circuits from Exercises 1 & 2. There is a potentiometer whose value is read from Pin A0, and an LED powered by PWM pin 5 based on input from the P5Js sketch.

Video Demonstration:

Serial Communication: In-class exercises

EXERCISE 1

Task: make something that uses only one sensor on Arduino and makes the ellipse in p5 move on the horizontal axis, in the middle of the screen, and nothing on arduino is controlled by p5

Code for P5 and Arduino commented at the end of it:

Demonstration: 

EXERCISE 2

Task: make something that controls the LED brightness from p5

Code for P5 and Arduino commented at the end of it:

Demonstration:

EXERCISE 3

Task: take the gravity wind example and make it so every time the ball bounces one led lights up and then turns off, and you can control the wind from one analog sensor.

This was the hardest exercise to do. I made it work, however, instead of fully turning off when the ball is fully on the ground, the LED lights up. This would be a problem to solve in the future.

Code for P5 and Arduino commented at the end of it:

Demonstration: