Concept:

Imagine it’s the year 2100. They apparently discovered a robot from back in 2023 at the museum. They say it used to be the best at one point of time. Robots could just ‘talk’ back then, you know? And even that would be some inferior form of consciousness – simply mimicry. But oh wow, how it enchanted the people back then.

This is a project where the user can interact and communicate with a talking robot. In building this project I made extensive use of the ChatGPT API, for text generation and the p5.speech library for speech to text and text to speech. Additionally, I use the ml5.js library for person tracking that is done physically using a servo motor. Aesthetically, I was inspired by “Wall-E” to use the broken-down and creaky cardboard box aesthetic for my main moving robot head.

User Testing:

Implementation:

Interaction Design:

The user interacts with the robot by talking to it/moving around, patting the robot on its head, and turning a potentiometer/pressing a button. The robot tracks the user around making it seem conscious.

The talking aspect is simple, as in the robot listens to the user when the light is green, processes information when the indicator light is blue, and speaks when the indicator light is green – making it clear when the user can talk. The user can also press the “Click a photo” button and then ask a question to give the robot an image input too. Finally, the user can choose one of three possible moods for the robot – a default mode, a mode that gives the user more thoughtful answers, and a mode where the robot has an excited personality.

Arduino:

The Arduino controls the servo motor moving the robot head, the neopixels lights, the light sensor, the potentiometer and the two buttons. In terms of computations it computes what color the neopixels should be.

#include <Servo.h>
#include <Adafruit_NeoPixel.h>

// Pin where the NeoPixel is connected
#define PIN            11

// Number of NeoPixels in the strip
#define NUMPIXELS      12

// Create a NeoPixel object
Adafruit_NeoPixel strip = Adafruit_NeoPixel(NUMPIXELS, PIN, NEO_GRB + NEO_KHZ800);
Servo myservo;  // Create servo object
int pos = 90;   // Initial position
int neostate=0;
int onButton = 4;
int picButton = 7;
int potpin=A1;

void neo_decide(int neo){
  //starting
  if(neo==0)
  {
    setColorAndBrightness(strip.Color(100, 200, 50), 128); // 128 is approximately 50% of 255
    strip.show();
  }
  //listening
  else if(neo==1)
  {
    setColorAndBrightness(strip.Color(0, 255, 0), 128); // 128 is approximately 50% of 255
    strip.show();
  }
  //thinking
  else if(neo==2)
  {
    setColorAndBrightness(strip.Color(0, 128, 128), 128); // 128 is approximately 50% of 255
    strip.show();
  }
  //speaking
  else if(neo==3)
  {
    setColorAndBrightness(strip.Color(255, 0, 0), 128); // 128 is approximately 50% of 255
    strip.show();
  }
  //standby
  else
  {
    setColorAndBrightness(strip.Color(128, 0, 128), 128); // 128 is approximately 50% of 255
    strip.show();
  }
}

void setColorAndBrightness(uint32_t color, int brightness) {
  strip.setBrightness(brightness);
  for(int i = 0; i < strip.numPixels(); i++) {
    strip.setPixelColor(i, color);
    strip.show();
  }
}

void setup() {
  // Start serial communication so we can send data
  // over the USB connection to our p5js sketch
  myservo.attach(9);  // Attaches the servo on pin 9
  Serial.begin(9600);
  strip.begin();
  strip.show();
  pinMode(onButton, INPUT_PULLUP);
  pinMode(picButton, INPUT_PULLUP);
  // start the handshake
  while (Serial.available() <= 0) {
    digitalWrite(LED_BUILTIN, HIGH); // on/blink while waiting for serial data
    Serial.println("0"); // send a starting message
    delay(300);            // wait 1/3 second
    digitalWrite(LED_BUILTIN, LOW);
    delay(50);
  }
}

void loop() {
  // wait for data from p5 before doing something
  while (Serial.available()) {
    digitalWrite(LED_BUILTIN, HIGH); // led on while receiving data

    pos = Serial.parseInt();
    neostate = Serial.parseInt();
    neo_decide(neostate);
    if (Serial.read() == '\n') {
      myservo.write(pos);   // Move servo to position
      int lightstate=analogRead(A0);
      int onbuttonstate=digitalRead(onButton);
      int picbuttonstate=digitalRead(picButton);
      int potstate=analogRead(potpin);
      Serial.print(lightstate);
      Serial.print(',');
      Serial.print(potstate);
      Serial.print(',');
      Serial.print(onbuttonstate);
      Serial.print(',');
      Serial.println(picbuttonstate);
    }
  }
  digitalWrite(LED_BUILTIN, LOW);
}

P5.js

The p5.js code does most of the work for this project. Firstly, it handles the API calls to GPT3.5 Turbo and GPT-4-vision-preview models. When the user is talking to the robot normally, I send the API calls to the cheaper GPT3.5 turbo model, when the user wants to send an image input, I convert all the previously sent inputs into the format necessary for the GPT4-vision-preview model along with the image.

Second, I use the ml5.js library and the ‘cocossd’ object detection model to detect a human on the camera field of vision and draw a bounding box around it. Then we take the center of the bounding box and attempt to map the servo motor’s movement to this.

The text to speech and speech to text functionalities are done using the p5.Speech library. While doing this, we keep a track of what state we are in currently.

Lastly, we also keep track of whether the system is on right now, the light sensor’s values, and whether the click a photo button was pressed. The ‘on’ button, as the name suggests acts as a toggle for the system’s state, the light sensor starts a specific interaction when it’s value is below a threshold, and the photo button informs us about which API call to make.

Finally, we can also switch between the model’s different personalities by using the potentiometer and this is handled in the updateMood() function.

Communication between Arduino – P5

The Arduino communicates the button states, the potentiometer state, and the light sensor state to the p5.js program and receives inputs for the neopixel state, and the servo motor state.

Highlights:

For me the highlights of this project have to be designing the physical elements and handling the complex API call mechanics. Due to the fact that I use two different models that have different API input structures, the transformation between them was time-consuming to implement. Additionally, the p5.speech library that I use extensively is relatively unreliable and took a lot of attempts for me to use correctly.

Additionally, it was exciting to watch the different ways people interacted with the robot at the IM showcase. A large percent of people were interested in using the robot as a fashion guide assistant! I think overall, this was a really interesting demonstration of the potential of generative AI technology and I would love to build further using it!

Future Work:

There are several features I would love to add to this project in future iterations. Some of these include:

• Do a basic talking animation by moving the robot’s head up and down during the talking phase. Additionally, perhaps make the robot mobile so it can move about.
• Make the camera track the person in the front. This can be done by making the person wear a specific object – or some sophisticated ML trick.
• Have additional interactions and add parts to the robot such as more expressive ears etc.
• Use a more robust box and fit the arduino breadboard, camera, speaker etc. inside the robot head.
• Make the neopixels implementation more aesthetic by employing colored patterns and animations.

Concept

As I mentioned in a previous post, I wanted to make a Mastermind game, which is a two-player board game where one person is the code maker, and they make a code using colored pegs. The other person is the code breaker, and they have to figure out the code in a limited number of moves. The less moves you take to figure out the code, the higher you score.

In the version that I made, this game is actually a 1-player game, where the code is randomly generated, and the players have to guess the code. Players use a physical system of colored pegs and holes to parse input to the board.

Final Game

I’m very happy with the final P5Js interface for the game. It’s very minimalistic, which matches the style that I’ve chosen to go with for all of my projects. The instructions are in the upper left corner, as I didn’t want to clutter the interface by adding buttons to other pages. Also, I wanted to keep interaction with the laptop minimal if not zero, so I opted to have a static screen and focus on the physical input interface.

P5Js Sketch

The sketch does require a serial connection with Arduino. However, I enabled skipping the serial connection check for the purposes of viewing the sketch without an Arduino connection. You can press the “space” key to skip to the game screen. I have also edited the aspect ratio, for viewing it successfully within the Intro to IM blog post. It’s best viewed in fullscreen mode.

The Physical System

The control interface consists of a box with 5 colored pegs. In the picture below, you can see how the pegs fit into the console, and how the order can be changed to alter the input.

There is also a button on the left side of the interface, which can be pressed to confirm your choice. I used an LED momentary button because it reminded me of retro gaming consoles.

The color detection mechanism is rather naive, and it became a pain later on during the showcase as well. I have embedded LEDs in each peg, and each LED has a specific level of brightness. There is a single light sensor (LDR) in each of the 4 holes, which measures the brightness once a peg is placed into the hole. Depending on the brightness of the LED, a value corresponding to the respective color is generated. In the case of yellow, this would be the value “3”, which is the index in the array that contains the color values, as well as the index in which the pegs are fixed into the console, if you start counting from 0.

To ensure the fit was tight for consistent readings, I wrapped electrical tape around the pegs until there was a snug fit.

User Testing

I asked my friend to test the game for me. I told her nothing about how to use the interface, but she already knew about Mastermind.

She found the interface intuitive to use, but I think that it wasn’t as clear for people who didn’t know the game beforehand.

In the end, my friend won the game:

Circuit

I wanted to keep the circuit as neat as possible, so I kept the Arduino on one side of the box, had a long breadboard in the middle, and the wires connecting to the closest part of the breadboard to the corresponding hardware. I then used jumper wires to complete the circuit on the breadboard.

I soldered the wires onto the LDRs and LEDs, and then use screw mounts to connect the other ends to the breadboard. I did the same for the LED button as well.

Communication Between P5Js and Arduino

I only implemented one way communication, as the goal was to have Arduino serve only as an input device. Therefore, I just sent the state of the current row of the game as an integer code, as well as a binary value showing whether or not the button to lock in the choice was pressed.

Challenges

The most challenging aspect was calibrating the LDRs and tuning them so that each color had a particular range. However, due to the sensitivity of the LDRs, it was hard to have the LDR value ranges for each color not intersect. There was a tradeoff, as if I chose to spread out the ranges for each color, it would mean that some colors would have the same brightness as the ambient brightness, which would end up showing that color throughout the board. I chose to keep this in for the sake of having more consistency in the input, but I realize that that decision made the game confusing for some people.

Aspects that I’m proud of

I really like the aesthetics of the game overall, and I think that the P5Js aspect of the project was done very well. There are definitely some features that I would have liked to implement, but the things that I chose to keep in were well done.

I’m also proud of the way I handled an issue that occurred. Since I’m using light sensors, the mechanism needed recalibration whenever the ambient brightness changed. To tackle this, I took two approaches.

Firstly, I took some plastic nuts from the large LED buttons and fixed them on top of the knobs to block out any light from the cracks around the knob. To make sure that there was absolutely no light coming through from around the peg, I wrapped it in tape until it fit snugly. This was the physical steps that I took.

Secondly, I created a calibration script within Arduino. When I needed to calibrate the settings, I would run this, and it would tell me what settings I need for the current environment based on the minimum and maximum readings for each color across all the light sensors. The code for that is below:

void calibrateColorRanges()
{
  Serial.println("Calibration started. Please follow the instructions to calibrate each color.");

  const int numColors = 5; // Assuming 5 colors: Red, Green, Blue, Yellow, Turquoise
  const int numLDRs = 4;   // Assuming 4 LDRs

  int calibrationValues[numColors][2]; // Array to store min and max values for each color across all LDRs

  for (int color = 0; color < numColors; color++)
  {
    Serial.print("Calibrating ");
    switch (color)
    {
    case 0:
      Serial.print("RED");
      break;
    case 1:
      Serial.print("GREEN");
      break;
    case 2:
      Serial.print("BLUE");
      break;
    case 3:
      Serial.print("YELLOW");
      break;
    case 4:
      Serial.print("TURQUOISE");
      break;
    }
    Serial.println("...");

    // Initialize min and max values
    calibrationValues[color][0] = 1023; // Initial min value
    calibrationValues[color][1] = 0;    // Initial max value

    for (int ldr = 0; ldr < numLDRs; ldr++)
    {
      Serial.print("Move ");
      switch (color)
      {
      case 0:
        Serial.print("RED");
        break;
      case 1:
        Serial.print("GREEN");
        break;
      case 2:
        Serial.print("BLUE");
        break;
      case 3:
        Serial.print("YELLOW");
        break;
      case 4:
        Serial.print("TURQUOISE");
        break;
      }
      Serial.print(" to LDR ");
      Serial.print(ldr + 1);
      Serial.println(" and press Enter.");

      // Wait for user input (press Enter in the Serial Monitor)
      while (!Serial.available())
      {
        delay(100);
      }

      // Discard any existing input
      while (Serial.available())
      {
        Serial.read();
      }

      // Perform readings and find min and max values
      for (int i = 0; i < 100; i++)
      {
        int ldrValue = analogRead(LDR_1 + ldr);

        // Update min and max values
        calibrationValues[color][0] = min(calibrationValues[color][0], ldrValue);
        calibrationValues[color][1] = max(calibrationValues[color][1], ldrValue);

        delay(10); // Delay between readings
      }
    }

    // Display the min and max values for the current color across all LDRs
    Serial.print("Min value: ");
    Serial.print(calibrationValues[color][0]);
    Serial.print(", Max value: ");
    Serial.println(calibrationValues[color][1]);
  }

  Serial.println("Calibration complete. Use the following values in getColorFromLdrVal function:");
  for (int color = 0; color < numColors; color++)
  {
    Serial.print("Color ");
    switch (color)
    {
    case 0:
      Serial.print("RED");
      break;
    case 1:
      Serial.print("GREEN");
      break;
    case 2:
      Serial.print("BLUE");
      break;
    case 3:
      Serial.print("YELLOW");
      break;
    case 4:
      Serial.print("TURQUOISE");
      break;
    }
    Serial.print(": Min - ");
    Serial.print(calibrationValues[color][0]);
    Serial.print(", Max - ");
    Serial.println(calibrationValues[color][1]);
  }
}

Although some other approach might have been more elegant, I think that I was able to manage even with the drawbacks of the design that I selected.

Future Ideas

• I think it would be fun to add a leaderboard. Since I have score metrics already, it would be more fun if people competed against people who played the game previously.
• It would be nice to have the option for players to play as codemakers as well, since the original game is a two-player game. That would make my project true to the original.
• It would be better if I had the calibration embedded into the P5Js sketch, so that there is no need to reflash the Arduino. Currently, the way that I do it is to reflash the Arduino with the calibration command, and then flash it again after calibrating it, which is a nuisance.

Arduino Code

#include <Arduino.h>

// colors as ints
const int RED = 0;
const int GREEN = 1;
const int BLUE = 2;
const int YELLOW = 3;
const int TURQUOISE = 4;

// 5 LEDS with variable brightness on Arduino UNO
// LEDs are connected to pins 3, 5, 6, 9, 10
const int LED_PINS[] = {5, 6, 9, 10, 11};

const String colors[] = {"R", "G", "B", "Y", "T", "N"}; // red, green, blue, yellow, turquoise, none
// different brightness levels for LEDS
const int brightnessLevels[] = {LOW, HIGH, HIGH, HIGH, HIGH};
// const int brightnessLevels[] = {10, 0, 4, 153, 255};

// 4 LDRS connected to pins A1, A2, A3, A4
const int LDR_1 = A1; // 15
const int LDR_2 = A2; // 16
const int LDR_3 = A3; // 17
const int LDR_4 = A4; // 18

// 4 variables to store the values from the LDRs
int ldrValue1 = 0;
int ldrValue2 = 0;
int ldrValue3 = 0;
int ldrValue4 = 0;

// button LED
const int buttonLED = 3;
const int buttonPin = 2;

void setup()
{
  // initialize serial communication at 9600 bits per second:
  Serial.begin(9600);
  // initialize the LED pins as an output:
  for (int i = 0; i < 5; i++)
  {
    pinMode(LED_PINS[i], OUTPUT);
  }

  // initialize the LDR pins as an input:
  pinMode(LDR_1, INPUT);
  pinMode(LDR_2, INPUT);
  pinMode(LDR_3, INPUT);
  pinMode(LDR_4, INPUT);

  pinMode(buttonLED, OUTPUT);
  pinMode(buttonPin, INPUT_PULLUP);
}

const int minRed = 980;
const int maxRed = 1023;
const int minGreen = 610;
const int maxGreen = 940;
const int minBlue = 220;
const int maxBlue = 460;
const int minYellow = 120;
const int maxYellow = 220;
const int minTurquoise = 0;
const int maxTurquoise = 70;
int ambientLight[] = {0, 0, 0, 0};

int getColorFromLdrVal(int ldrVal, int ambientLight)
{
  if (ldrVal >= minRed && ldrVal <= maxRed) // range of
  {
    return RED;
  }
  if (ldrVal >= minGreen && ldrVal <= maxGreen) // range of 175
  {
    return GREEN;
  }
  if (ldrVal >= minBlue && ldrVal < maxBlue) // range of 310
  {
    return BLUE;
  }
  if (ldrVal >= minYellow && ldrVal <= maxYellow) // range of 70
  {
    return YELLOW;
  }

  if (ldrVal >= minTurquoise && ldrVal <= maxTurquoise) // range of 80
  {
    return TURQUOISE;
  }

  else
  {
    return 5;
  }
}

void calibrateColorRanges()
{
  Serial.println("Calibration started. Please follow the instructions to calibrate each color.");

  const int numColors = 5; // Assuming 5 colors: Red, Green, Blue, Yellow, Turquoise
  const int numLDRs = 4;   // Assuming 4 LDRs

  int calibrationValues[numColors][2]; // Array to store min and max values for each color across all LDRs

  for (int color = 0; color < numColors; color++)
  {
    Serial.print("Calibrating ");
    switch (color)
    {
    case 0:
      Serial.print("RED");
      break;
    case 1:
      Serial.print("GREEN");
      break;
    case 2:
      Serial.print("BLUE");
      break;
    case 3:
      Serial.print("YELLOW");
      break;
    case 4:
      Serial.print("TURQUOISE");
      break;
    }
    Serial.println("...");

    // Initialize min and max values
    calibrationValues[color][0] = 1023; // Initial min value
    calibrationValues[color][1] = 0;    // Initial max value

    for (int ldr = 0; ldr < numLDRs; ldr++)
    {
      Serial.print("Move ");
      switch (color)
      {
      case 0:
        Serial.print("RED");
        break;
      case 1:
        Serial.print("GREEN");
        break;
      case 2:
        Serial.print("BLUE");
        break;
      case 3:
        Serial.print("YELLOW");
        break;
      case 4:
        Serial.print("TURQUOISE");
        break;
      }
      Serial.print(" to LDR ");
      Serial.print(ldr + 1);
      Serial.println(" and press Enter.");

      // Wait for user input (press Enter in the Serial Monitor)
      while (!Serial.available())
      {
        delay(100);
      }

      // Discard any existing input
      while (Serial.available())
      {
        Serial.read();
      }

      // Perform readings and find min and max values
      for (int i = 0; i < 100; i++)
      {
        int ldrValue = analogRead(LDR_1 + ldr);

        // Update min and max values
        calibrationValues[color][0] = min(calibrationValues[color][0], ldrValue);
        calibrationValues[color][1] = max(calibrationValues[color][1], ldrValue);

        delay(10); // Delay between readings
      }
    }

    // Display the min and max values for the current color across all LDRs
    Serial.print("Min value: ");
    Serial.print(calibrationValues[color][0]);
    Serial.print(", Max value: ");
    Serial.println(calibrationValues[color][1]);
  }

  Serial.println("Calibration complete. Use the following values in getColorFromLdrVal function:");
  for (int color = 0; color < numColors; color++)
  {
    Serial.print("Color ");
    switch (color)
    {
    case 0:
      Serial.print("RED");
      break;
    case 1:
      Serial.print("GREEN");
      break;
    case 2:
      Serial.print("BLUE");
      break;
    case 3:
      Serial.print("YELLOW");
      break;
    case 4:
      Serial.print("TURQUOISE");
      break;
    }
    Serial.print(": Min - ");
    Serial.print(calibrationValues[color][0]);
    Serial.print(", Max - ");
    Serial.println(calibrationValues[color][1]);
  }
}

unsigned long timeSinceLastSerial = 0;
int valFromP5 = 0;

float sum1, sum2, sum3, sum4 = 0;
unsigned int ldrVal1, ldrVal2, ldrVal3, ldrVal4 = 0;
unsigned int iterator = 0;

unsigned long buttonLEDTimer = 0;
unsigned long buttonSendTimer = 0;

int buttonBrightness = 0;
int buttonDirection = 1;

int buttonState = LOW; // if button is not pressed

void loop()
{

  // fade in and out button LED
  if (millis() - buttonLEDTimer > 5)
  {
    buttonLEDTimer = millis();
    if (buttonBrightness >= 255 || buttonBrightness <= 0)
    {
      buttonDirection = buttonDirection * -1;
    }

    buttonBrightness = (buttonBrightness + 1 * buttonDirection);
    if (buttonBrightness >= 255)
    {
      buttonBrightness = 255;
    }
    if (buttonBrightness <= 0)
    {
      buttonBrightness = 0;
    }
    analogWrite(buttonLED, buttonBrightness);
  }

  // turn on the LEDs to the corresponding brightness
  digitalWrite(LED_PINS[RED], brightnessLevels[RED]);
  digitalWrite(LED_PINS[GREEN], brightnessLevels[GREEN]);
  digitalWrite(LED_PINS[BLUE], brightnessLevels[BLUE]);
  digitalWrite(LED_PINS[YELLOW], brightnessLevels[YELLOW]);
  digitalWrite(LED_PINS[TURQUOISE], brightnessLevels[TURQUOISE]);

  // uncomment to calibrate color ranges
  // calibrateColorRanges();

  // send to processing
  if (millis() - timeSinceLastSerial < 10) // if it has not been more than 20ms since last serial message
  {
    return; // do nothing
  }

  timeSinceLastSerial = millis(); // update the time since last serial message

  if (iterator == 10)
  {
    ldrVal1 = int(sum1 / 10);
    ldrVal2 = int(sum2 / 10);
    ldrVal3 = int(sum3 / 10);
    ldrVal4 = int(sum4 / 10);

    // print the values if they are not nan
    if (!isnan(ldrVal1) && !isnan(ldrVal2) && !isnan(ldrVal3) && !isnan(ldrVal4))
    {

      // get the color from the LDR value
      int color1 = getColorFromLdrVal(ldrVal1, ambientLight[0]);
      int color2 = getColorFromLdrVal(ldrVal2, ambientLight[1]);
      int color3 = getColorFromLdrVal(ldrVal3, ambientLight[2]);
      int color4 = getColorFromLdrVal(ldrVal4, ambientLight[3]);
      int buttonState = digitalRead(buttonPin);

      Serial.print(color1);
      Serial.print(",");
      Serial.print(color2);
      Serial.print(",");
      Serial.print(color3);
      Serial.print(",");
      Serial.print(color4);
      Serial.print(",");
      Serial.println(buttonState);
    }

    iterator = 0;
    sum1 = 0;
    sum2 = 0;
    sum3 = 0;
    sum4 = 0;
  }

  // read and add values from the LDRs
  sum1 = sum1 + analogRead(LDR_1);
  sum2 = sum2 + analogRead(LDR_2);
  sum3 = sum3 + analogRead(LDR_3);
  sum4 = sum4 + analogRead(LDR_4);

  iterator++;
}

P5Js Code

let board;

let WIDTH = 2800;
let HEIGHT = 1500;
let colorsArray;

const RED = 0;
const GREEN = 1;
const BLUE = 2;
const YELLOW = 3;
const TURQUOISE = 4;
const BLACK = 5;

let canvas;
let port;

let highScores = [];




function generateCode() {
 let code = [];
  let array = [0, 1, 2, 3 , 4, 5];
  //select random subset of 4 colorsArray
  let colorsArray = shuffleArray(array);

  for (let i = 0; i < 4; i++) { // select only 4 colorsArray
    code.push(colorsArray[i]);
  }

  return code;
}

// Helper function to shuffle an array
function shuffleArray(array) {
  //copy the array
  array_copy = array.slice();

  for (let i = array_copy.length - 2; i > 0; i--) {
    const j = Math.floor(Math.random() * (i + 1));
    [array_copy[i], array_copy[j]] = [array_copy[j], array_copy[i]];
  }
  return array_copy;
}


function calculateScore(numGuesses) {
  return 1500 - 100 * numGuesses;
}

let r, g, b, y, t, bl;


let currentGuess;
let numberOfGuesses;
let lastButtonPressed;
let isButtonPressed;
let buttonClock;
let board_score;
let state;


function setup() {
  canvas = createCanvas(WIDTH, HEIGHT);
  board = new Board();

  r = color(200, 50, 80);
  g = color(20, 255, 20);
  b = color(50, 20, 255);
  y = color(255, 255, 20);
  t = color(20, 255, 255);
  bl = color(0, 0, 0);


  // colorsArray are red, green, blue, yellow, turquoise, black
  colorsArray = [r, g, b, y, t, bl];

  //generate the code
  code = generateCode();

  port = createSerial();

  //initialize important variables
  numberOfGuesses = 0;
  lastButtonPressed = 0;
  isButtonPressed = 0;
  buttonClock = 0;

  board_score = [0, 0];

  state = "AUTH_STATE";

}

function resetGame()
{
  board.reset();
  numberOfGuesses = 0;
  score = 0;
  state = "AUTH_STATE";
  code = generateCode();

}


function getNameInput()
{
  let name = prompt("Please enter your name ");
  if (name == null || name == "") {
    name = "Anonymous";
  }
  return name;
}


let delayClock = 0;
function draw() {

  if (state == "GAME_STATE") {
    background(220);

    board_score = board.score;
      print("score: " + board_score);

      if (board_score[0] == 4) {
        print("GAME WON")
        state = "GAME_WON_STATE";

        delayClock = millis();
      }

      if (board.currentRow == 15) {
        print("GAME LOST")
        state = "GAME_LOST_STATE";
        delayClock = millis();

      }

    // if the port is open, read the data
    if (port.available() > 0) {
      let data = port.readUntil("\n");

      split_data = int(data.split(","));

      // first 4 are the current guess
      // last is the button press
      currentGuess = split_data.slice(0, 4);
      isButtonPressed = split_data[4];
      board.update(currentGuess);
    }


    //if the button is pressed, finalize the guess
    if (isButtonPressed == 1 && lastButtonPressed == 0) {
      if (board.finalizeChoices(code))
      {
        numberOfGuesses++;
      };
      lastButtonPressed = 1;
    }

  
    else if (isButtonPressed == 0 && lastButtonPressed == 1) {
      lastButtonPressed = 0;
    }



    board.show();

    //show the score at the top right
    fill(0);
    textSize(35);
    push();
    textStyle(BOLD);
    text("SCORE: " + calculateScore(numberOfGuesses), width - 300, 100);
    pop();

    // print instructions to the side 
    fill(0);
    textSize(40);

    push();
    textStyle(BOLD);
    text("Instructions", 50, 100);
    pop();

    textSize(30);
    
    push(); 
    fill(y);
    circle(50, 162, 20);
    pop();
    text(": right color, wrong position", 80, 170);

    // red circle 
    push();
    fill(r);
    circle(50, 222, 20);    
    pop();

    text(": right color, right position", 80, 230);

    text("Guess the code in as few guesses as possible!", 40, 300);



  }

  else if (state == "GAME_WON_STATE") {
    background(220, 200);
    fill(0);
    textSize(60);
    textStyle(BOLD);
    text("YOU WON!", width / 2 - 200, height / 2);

    //SHOW SCORE
    fill(0);
    textSize(35);
    push();
    textStyle(BOLD);
    text("SCORE: " + calculateScore(numberOfGuesses), width / 2 - 200, height / 2 + 100);
    pop();


    if (millis() - delayClock < 2000) {
      return;
    }


    //read input from the serial port
    if (port.available() > 0) {
      let data = port.readUntil("\n");
      
      data = int(data.split(","));

      let buttonval = data[4];
      if (buttonval == 1) {
        print("resetting");
        board.reset();
        numberOfGuesses = 0;
        state = "GAME_STATE";
        board_score = [0, 0];
      }
    }
  }

  else if (state == "GAME_LOST_STATE") {
    background(220, 200);
    fill(0);
    textSize(60);
    textStyle(BOLD);
    text("YOU LOST!", width / 2 - 200, height / 2);

    text("The code was: ", width / 2 - 600, height / 2 + 200);
    //print the right code
    for (let i = 0; i < 4; i++) {
      push();
      fill(colorsArray[code[i]]);
      circle(width/2 - 100 + i * 100, height/2 + 185, 50);
      pop();
    }

    //push the button to reset
    push();
    textSize(30);
    text("Press the button to reset", width / 2 - 200, height / 2 + 400);
    pop();


    if (millis() - delayClock < 2000) {
      return;
    }

    //read input from the serial port
    if (port.available() > 0) {
      let data = port.readUntil("\n");
      
      data = int(data.split(","));

      let buttonval = data[4];
      if (buttonval == 1) {
        print("resetting");
        resetGame();
      }
    }
  }



  else if (state == "AUTH_STATE") {

    //ask to connect to the device
    background(220, 200);
    fill(0);
    textSize(60);
    textStyle(BOLD);
    text("PRESS SPACE TO START", width / 2 - 300, height / 2);

    if(port.opened()) {
      state = "GAME_STATE";
    }

  }



}


/**
 * Handles key press events and performs corresponding actions.
 */
function keyPressed() {
  /**
   * If the key pressed is "c" and the state is "GAME_STATE",
   * finalize the choices on the board using the provided code.
   *
   * @returns {void}
   */
  if (key == "c") {
    if (state != "GAME_STATE") {
      return;
    }
    board.finalizeChoices(code);
  }
}

  /**
   * If the key pressed is "n", update the current color of the current row on the board.
   * The color is updated by incrementing the current color index by 1 and wrapping around to 0 if it exceeds 5.
   *
   * @returns {void}
   */
  if (key == "n") {
    board.rows[board.currentRow].currentColor = (board.rows[board.currentRow].currentColor + 1) % 6;
  }

  /**
   * If the key pressed is " " (space) and the state is "AUTH_STATE",
   * open a port at 9600 baud using the port object.
   *
   * @returns {void}
   */
  if (key == " ") {
    if (state != "AUTH_STATE") {
      return;
    }
    port.open(9600);
  }

  /**
   * If the key pressed is "r", reset the game.
   *
   * @returns {void}
   */
  if (key == "r") {
    resetGame();
  }
}

My final project idea is to develop an immersive and interactive surfing simulator game. The idea came to me when I went to the arcade and saw a snowboarding game and it sparked my interest to know how it was made and if I could replicate it with something I enjoyed and miss which is surfing. This game will use a physical skateboard found in the IM lab equipped with an accelerometer to capture the player’s movements, and a P5.js-based game that simulates a surfing and will have the concept of collecting and avoiding.

This is the inspiration below:

Reading Graham Pullin’s “Design Meets Disability” was like opening a window to a new world of possibilities in design. Pullin’s approach is transformative, merging concepts like simplicity, universality, exploratory problem-solving, fashion, and discretion into a cohesive vision for disability aids. He challenges the traditional, function-first mindset and proposes a more inclusive, style-conscious approach, arguing that assistive devices should be as much a statement of personal style as they are functional.

Pullin’s ideas are inspiring, urging us to think beyond the conventional. He’s not just talking about making things easier to use but also about embracing the diversity of users. His call for a blend of aesthetics and functionality in design resonates with me. It’s a fresh take that adds dignity and choice to the equation, offering people with disabilities more than just practical solutions but also products they can feel good about using.

However, as a student thinking critically, I see challenges in Pullin’s vision. His idealistic approach makes me wonder about the real-world implications. How do we balance these ambitious design goals with practical constraints like cost, manufacturing complexities, and the varied needs of individuals with disabilities? It’s one thing to dream of stylish, universally accessible designs, but another to implement them in a way that’s affordable and accessible to all.

“Design Meets Disability” has definitely broadened my understanding of what design can and should do. It’s pushed me to think about how we can make the world more inclusive through thoughtful design. But it’s also left me with questions about the balance between idealism and feasibility. How do we make these innovative designs a reality for everyone who needs them, not just those who can afford them? It’s a challenging question, and Pullin’s book is a compelling starting point for this important conversation.

This is a P5 setup in the beginning:

After you connect to the port by pressing the “/” key, and you press ENTER to take a snapshot, two images appear (animal and hat images).

You can go through an array of images by pressing the red buttons and choosing the hat and animal that you like. Further, you can adjust their position with potentiometer knobs. You can write your name in the input text box and SUBMIT it by clicking the button, so it appears as usual text on the ID badge. Finally, if you are satisfied with everything, you can click the CAPTURE button and save the ID badge.

The final ID badge image looks like this:

Set up during the show:

I have developed my concept further and designed a photobooth from cardboard. I printed out instructions and the table for people to fill in (where they would write their name and net ID, so I can send them the image of the ID badge later). I have designed this box from large cardboard pieces that I found in the lab. Painted the front with black acrylic paint and printed a design that reminds me of the film. 

Process:

Another part of my project is a small white box that I built from foamboard, which holds two red buttons, 4 potentiometer knobs (yellow for the X axis and green for the Y axis), and labels for them. I have designed this box in the Maker Case.  Then I used a ruler and box cutter to cut all the parts and assemble them. They fit nicely, but I additionally used some glue to be sure the box can hold the pressure from pressing buttons and rotating knobs. 

Process:

During the process of building these parts, the biggest challenge was cutting the elements. I did not have an opportunity to use a laser cutter, therefore I spent lots of time cutting and taping everything by hand.

P5 code

Most of the time was spent setting up the P5 sketch. What does it do? The P5 program starts with a display of an ID badge. I design these with primitive rectangular shapes and text. I wanted to include instructions on the screen. however, I did not want to overcomplicate the appearance of the sketch, so I decided to do it on an A3 paper. However, as I noticed not many were reading instructions presented behind the laptop, therefore, it would be better to present them on the screen.

On the P5, participants were presented with a live video which is the size of a snapshot they are about to make. As they press ENTER, the snapshot selfie replaces the live video. This was a very challenging part of the code. However, with some help, I was able to do this. Here is the portion of the code that I used as an example:

let capture;

function setup() {
  createCanvas(500, 500);
  capture = createCapture(VIDEO);
  capture.hide();
}

function draw() {
  let aspectRatio = capture.height/capture.width;
  
  image(capture, 0, 0, width, width * aspectRatio);
  filter(INVERT);
}

As soon as the photograph is taken, and if it is taken,  the signal to Arduino is sent, so it can now send communication to P5. P5 then acts as a receiver and displays different images of hats and animals. Moreover, participants can type their name in the input box and submit it to be displayed on the screen. Finally, by clicking capture they can save their ID cards. This is all done with built-in boxes and buttons in P5.

It was hard to figure out the logic that I wanted my program to have. But finally, I managed to do a set of nested if statements which I am very proud of. This is a part of the code that allows you to go in an array of images and select their position.

 if (snapshot) {
    // Display a captured selfie in place of the live video
    // resizing to the same parameters as video

    image(snapshot, width / 2, height / 2 - 100, 220, 180);

    // A nested if statement:
    // if -1 then no hat image is selected, otherwise the statement is true.
    // If a hat image is selected, then display the image at the index selectedHat from the array of hatImages.
    // The image is placed at coordinates (hatX, hatY).

    if (selectedHat !== -1) {
      image(hatImages[selectedHat], hatX, hatY);
    }

    // A nested if statement:
    // if -1 then no animal image is selected, otherwise the statement is true.
    // If an animal image is selected, then display the image at the index selectedAnimal from the array of animalImages.
    // The image is placed at coordinates (animalX, animalY).

    if (selectedAnimal !== -1) {
      image(animalImages[selectedAnimal], animalX, animalY);
    }
  }
}

Arduino Code

Setting up the Arduino code was pretty easy. I used only the information we learned in class.   The code consists of the declaration of 2 buttons, 4 potentiometers, and their states. The state of each button and the potentiometer is read and sent to p5.  The part that I am proud of is mapping the potentiometer values to the desired range of the canvas. This was a very important part of the Arduino code because, with this, users were able to adjust the position of hat and animal images.

• When pressing the HAT button, which is a digital input on pin 2, participants will see an image of a hat. They can choose their hat by pressing the button again.
• Then participants will need to adjust the position of a hat using 2 potentiometers that also operate on 2 potentiometers, which are analog inputs and located on pins A0 and A1.
• They can also do the same actions to choose their favorite animal by pressing the ANIMAL button. They can also select an appropriate location using two potentiometers, which are analog inputs and located on pins A2 and A3.

const int buttonHatPin = 2;        // Pin for the hat button
const int buttonAnimalPin = 3;     // Pin for the animal button
const int potentiometerHatXPin = A0;  // Pin for the X-axis potentiometer for hat
const int potentiometerHatYPin = A1;  // Pin for the Y-axis potentiometer for hat
const int potentiometerAnimalXPin = A2;  // Pin for the X-axis potentiometer for animal
const int potentiometerAnimalYPin = A3;  // Pin for the Y-axis potentiometer for animal

int buttonHatState = 0;       // Current state of the hat button
int buttonAnimalState = 0;    // Current state of the animal button
int potentiometerHatXValue = 0;  // Current value of the X-axis potentiometer hat
int potentiometerHatYValue = 0;  // Current value of the Y-axis potentiometer hat
int potentiometerAnimalXValue = 0;  // Current value of the X-axis potentiometer animal

int potentiometerAnimalYValue = 0;  // Current value of the Y-axis potentiometer animal

void setup() {
  Serial.begin(9600);               // Initialize the serial communication
  pinMode(buttonHatPin, INPUT);     // Setting up the button pin as input
  pinMode(buttonAnimalPin, INPUT);  // Setting up the button pin as input


  // START THE HANDSHAKE TO SEND 6 VALUES FROM ARDUINO TO P5
  while (Serial.available() <= 0) {
    digitalWrite(LED_BUILTIN, HIGH);  // on/blink while waiting for serial data
    Serial.println("0,0,0,0,0,0");    // send a starting message
    delay(300);                       // wait 1/3 second
    digitalWrite(LED_BUILTIN, LOW);
    delay(50);
  }
}

void loop() {
  buttonHatState = digitalRead(buttonHatPin);           // Read the state of the hat button
  buttonAnimalState = digitalRead(buttonAnimalPin);     // Read the state of the animal button
  potentiometerHatXValue = analogRead(potentiometerHatXPin);  // Read the value of the X-axis potentiometer for hat image
  potentiometerHatYValue = analogRead(potentiometerHatYPin);  // Read the value of the Y-axis potentiometer for hat image
  potentiometerAnimalXValue = analogRead(potentiometerAnimalXPin);  // Read the value of the X-axis potentiometer for animal image
  potentiometerAnimalYValue = analogRead(potentiometerAnimalYPin);  // Read the value of the Y-axis potentiometer for animal image


  // Map the potentiometer values to the desired range
  int hatX = map(potentiometerHatXValue, 0, 1023, 0, 900);  // Map X-axis potentiometer value to the canvas width
  int hatY = map(potentiometerHatYValue, 0, 1023, 0, 900);  // Map Y-axis potentiometer value to the canvas height

  int animalX = map(potentiometerAnimalXValue, 0, 1023, 0, 900);  // Map X-axis potentiometer value to the canvas width
  int animalY = map(potentiometerAnimalYValue, 0, 1023, 0, 900);  // Map Y-axis potentiometer value to the canvas height

  // Send the button states and potentiometer values to p5.js
  Serial.print(buttonHatState);
  Serial.print(",");
  Serial.print(buttonAnimalState);
  Serial.print(",");
  Serial.print(hatX);
  Serial.print(",");
  Serial.print(hatY);
  Serial.print(",");
  Serial.print(animalX);
  Serial.print(",");
  Serial.println(animalY);

  delay(100);  // Delay for stability
}

FUTURE IMPROVEMENTS AND REFLECTIONS

ACKNOWLEDGEMENT AND SOURCES

• Fox: https://pin.it/20JjlYq
• Zebra: https://pin.it/aEQCg0y
• Horse: https://pin.it/24mqc94
• Snail: https://pin.it/4wxrYgG
• Chicken: https://pin.it/17vdTTr
• Snake: https://pin.it/7q03SLi
• Elephant: https://pin.it/3MTOFdn
• Sheep: https://pin.it/42PWV3A
• Hedgehog: https://pin.it/2NW68K8
• Alpaca: https://pin.it/29VzjJ8
• Penguin: https://pin.it/6nq0Gnv
• Cat: https://lovepik.com/image-400441974/poor-little-black-cat-cell-phone-wallpaper.html
• Lion: https://www.everypixel.com/image-15458821448575658589

• https://banner2.cleanpng.com/20180206/suq/kisspng-witch-hat-witchcraft-clip-art-witches-hat-5a79691aeeea39.5466725215179062029786.jpg
• https://www.everypixel.com/image-15458821448575658589
• https://pin.it/3H1w2Sb
• https://pin.it/3aMdTj4
• https://pin.it/1KDsYLw
• https://pin.it/3Nl0AH9
• https://pin.it/2acHL0g
• https://www.pngkey.com/detail/u2w7i1y3r5i1w7o0_party-hat-photo-booth-prop/
• https://www.stickpng.com/img/holidays/christmas/christmas-santa-claus-hat-little-bell

Concept

During my first semester, I made a game called “Galaxian” for one of my final projects. Galaxian was a classic, straightforward space shooting game where players could control the movement of a spaceship on screen across the x axis. Essentially, they could only use left and right arrow keys to move the ship in a different direction. The aliens on screen were static and the user had to shoot lasers at them to kill them.

The simplicity of Galaxian laid the groundwork for  the birth of “Galaxian 2.0.” The initial project became a muse and inspired me to envision a more immersive and evolved gaming experience. In this sequel, I wanted to elevate the gaming dynamics with a physical controller.

User testing

link to video demonstration: https://youtu.be/hGPZssA0Q-Q

Since the menu contained all the necessary information, there wasn’t much explanation required from my end. However, a notable observation was that users often struggled to grasp the concept of a “moving” spaceship in the second level.

Following the initial user testing phase, two crucial insights emerged:

• It was essential to enhance the distinction between alien and spaceship lasers for better user comprehension.
• The second level needed adjustments to make it slightly easier.

P5js Code