Month: May 2024

Initial Interaction Without Instructions:
– Confusion: Users were initially confused about the purpose of the buttons and how to interact with the games. The lack of clear instructions or visual cues made it difficult for them to understand the mapping between the controls and the on-screen actions.
– Exploration: Despite the confusion, users were curious and began experimenting with the buttons to see what would happen. Through trial and error, some users could figure out the basic controls for movement within the games.

Video:

Challenges:
– Button Mapping: The mapping between the button colors and the directions (especially in the Snake game) seemed arbitrary and non-intuitive to users.Game Selection: The process of navigating the menu and selecting games was not immediately apparent.
– Game Mechanics: Users familiar with Tetris and Snake could grasp the gameplay quickly, but those unfamiliar struggled to understand the objectives and rules.

Positive Aspects:
-Visual Appeal: The flip-dot display was visually engaging and drew users’ attention.
– Physical Controls: Users enjoyed the tactile experience of using physical buttons rather than a touchscreen or keyboard.
– Nostalgia Factor: For users familiar with the classic Atari games, the project evoked a sense of nostalgia and brought back positive memories.

Areas for Improvement:
– Clear Instructions: Provide concise and easy-to-understand instructions on the display or through a separate guide.
– Intuitive Controls: Consider using more intuitive button mapping or providing visual cues on the buttons to indicate their functions. For example, arrows could represent direction in the Snake game.

Areas Requiring Explanation:
– Button Functions: The purpose and mapping of each button needed an explanation, especially for the Snake game, where the color-direction association was not intuitive.
– Menu Navigation: Entering the game menu and selecting games required clarification.
– Game Rules (for unfamiliar users): A brief overview of the basic rules and objectives of Tetris and Snake would be beneficial for users who haven’t played these games before.

IM Showcase Gallery:

Concept

For the purposes of the Final Project of Introduction to Interactive Media, I was  presented the challenging task of connecting Software(P5.js) and Hardware(Arduino). To achieve this, I decided to make a room that  promotes sustainability and also reacts to human emotions.

Motivation

The Project was inspired by the words of my roommate who is active member of our campus’s sustainability student interest group.  She constantly urges me to be more mindful of energy usage, especially my tendency to leave lights on unnecessarily. Moreover, she pointed out that I could be more emotionally present. That got me thinking: why not design a room that not only conserves energy but also tunes into human emotions? To achieve this, I incorporated a feature where both the music and lighting adapt to the mood of the person inside, creating an environment that is truly sentient and responsive.

P5.js Part

In the p5.js segment of my project, I began by integrating the camera setup. I utilized ml5.js along with face-api.js for emotion detection. Face-api.js is particularly well-suited for this task due to its ability to analyze specific facial points to ascertain emotions. The library offers a range of emotions including neutral, happy, angry, sad, disgusted, surprised, and fearful. However, for my project’s scope, I’m focusing on neutral, happy, sad, disgusted, and surprised.

I designed the system to handle the analysis for just one individual at a time. Although the camera captures all faces within its view, it processes emotions only for the first face detected. To guide the user, I placed text on the top left of the canvas that displays the detected emotion and its corresponding probability percentage. This not only informs the user about the types of expressions to test but also enhances the interactive aspect of the project.

To make the experience more engaging, I created specific graphics for each emotion using Canva. These graphics dynamically appear and float around the canvas as the detected emotion changes. Additionally, I’ve incorporated adaptive music into the p5.js environment; the music alters based on the detected emotion, thus varying the room’s ambiance to match the user’s current emotional expression. I also added a fullscreen feature that activates when the user presses ‘f’, allowing both the canvas and video to fill the entire screen.

Graphics Used

Happy: Sad: Surprised: Disgusted: Neutral:

Arduino Part

For the Arduino component of my project, I’ve integrated three RGB LEDs and three pressure sensors. Each pressure sensor is linked to a corresponding LED, such that pressing a sensor activates its associated LED.

In p5.js, I am analyzing five expressions and converting the detection results into binary values, represented as either 1 or 0. These values are then transmitted to the Arduino. Based on the received data, if an expression is represented by a ‘1’, the corresponding RGB LED changes its color to match the detected expression.

User Testing:

Hardware and its Pictures:

This prototype of my room features a study table, a bed, and a sofa, each equipped with pressure sensors. Above, there are three LED chandeliers hanging from the ceiling.

How it Works:

Here’s how it works:  Firstly, we initiate serial communication between the Arduino and the p5.js script. The music and camera activate simultaneously with the start of the p5.js sketch. Based on the expressions detected by the camera, the graphics on the display and the music will dynamically change to match your current mood.

When a user presses a pressure sensor on either the bed, chair, or sofa, the RGB LED positioned above that particular sensor will light up. Following this, as your facial expressions change, the color of the LED will also change accordingly: happiness triggers a green light, neutrality a blue light, sadness a red light, disgust a yellow light, and surprise a pink light. This creates a responsive environment that visually and audibly reflects your emotions.

Code:

Here’s the logic for activating the LED based on the sensor value and changing the colour based on the expression detected from the p5.js script.

// Light up the corresponding LED only
if (sensor1Value < 30 ) {
  digitalWrite(redLED1, Sad || disgusted || surprised);
  digitalWrite(greenLED1, Happy || disgusted);
  digitalWrite(blueLED1, Neutral || surprised);
} else if (sensor2Value < 40) {
  digitalWrite(redLED2, Sad || disgusted || surprised);
  digitalWrite(greenLED2, Happy || disgusted);
  digitalWrite(blueLED2, Neutral || surprised);
} else if (sensor3Value < 40) {
  digitalWrite(redLED3, Sad || disgusted || surprised);
  digitalWrite(greenLED3, Happy || disgusted);
  digitalWrite(blueLED3, Neutral || surprised);
}

P5.js Display

Surprised: Disgusted: Happy: Sad: Neutral:

Demonstration:

Part of the Project that I take the most pride in:

The part I’m most proud of is how I mapped the expression values to 0 and 1, based on the percentage of the emotion detected, and then stored them in an array. This simplification made it easier to send binary values to the Arduino. However, figuring out this code took some time, as I initially tried storing emotions and their associated values in a dictionary, which didn’t work.

// maps detected expressions to a set of predefined categories and assigns a binary value based on a threshold

function mapExpressions(expressions) {
    const expressionOrder = ['neutral', 'happy', 'sad', 'disgusted', 'surprised'];
    let expressionValues = [];

    expressionOrder.forEach(expression => {
//       if value deteced is more than 50% make it 1 otherwise 0
        let value = expressions[expression] > 0.5 ? 1 : 0; 
        expressionValues.push(value);
    });

    return expressionValues;
}

Difficulties and Future Improvements:

The most challenging aspect of this project was establishing the serial communication between p5.js and Arduino, which took a solid two days of trial and error. Despite trying numerous approaches, nothing seemed to work until I created a duplicate file, which then functioned flawlessly without errors. Another significant challenge was the coding aspect. Although the code itself was not particularly complex, integrating music and graphics with the face-api was time-consuming, necessitating updates to the HTML file.

Additionally, I encountered difficulties with the pressure sensors. Initially, I used piezo sensors, but they produced inconsistent readings. I switched to force sensors which provided more reliable results, although they required recalibration every five minutes, adding another layer of complexity to the project. I wrote two additional Arduino scripts to calibrate the sensors, which allowed me to run the serial monitor and check the pressure sensor values.

For future improvements, I would consider investing in better pressure sensors. Additionally, instead of relying on my computer’s speakers, I’d like to integrate an external speaker directly connected to the Arduino. This setup would enhance the overall functionality and user experience.

Code:

P5.js:

// Initializing varialbes and arrays

let faceapi;
let detections = [];
let expressionValues=[];
let video;
let canvas;
let happyEmojis=[];
let vx;
let vy;
let songs = [];
let currentSongIndex = -1; 

// loading grpahics and music

function preload() {
  
    happyEmojis[0]= loadImage('1.png'); 
    happyEmojis[1] = loadImage('2.png'); 
    happyEmojis[2] = loadImage('3.png'); 
    happyEmojis[3] = loadImage('4.png'); 
    happyEmojis[4]= loadImage('5.png'); 
    happyEmojis[5] = loadImage('6.png'); 
    happyEmojis[6] = loadImage('7.png'); 
    happyEmojis[7] = loadImage('8.png'); 
    happyEmojis[8]= loadImage('9.png'); 
    happyEmojis[9] = loadImage('10.png'); 
    happyEmojis[10] = loadImage('11.png'); 
    happyEmojis[11] = loadImage('12.png'); 
    happyEmojis[12]= loadImage('13.png'); 
    happyEmojis[13] = loadImage('14.png'); 
    happyEmojis[14] = loadImage('15.png'); 
    happyEmojis[15] = loadImage('16.png'); 
    happyEmojis[16]= loadImage('17.png'); 
    happyEmojis[17] = loadImage('18.png'); 
    happyEmojis[18] = loadImage('19.png'); 
    happyEmojis[19] = loadImage('20.png'); 
    happyEmojis[20] = loadImage('21.png'); 
    happyEmojis[21] = loadImage('22.png'); 
    happyEmojis[22] = loadImage('23.png'); 
  
    songs[0] = loadSound('song1.mp3');
    songs[1] = loadSound('song2.mp3');
    songs[2] = loadSound('song3.mp3');
    songs[3] = loadSound('song4.mp3');
    songs[4] = loadSound('song5.mp3');
}

// Setting up the canvas and video settings
function setup() {
    canvas = createCanvas(windowWidth, windowHeight);
    canvas.id('canvas');
    video = createCapture(VIDEO);
    video.size(windowWidth, windowHeight);
    video.id('video');
  
//   initializes the face detection
  const faceOptions = {
    withLandmarks: true,
    withExpressions: true,
    withDescriptors: true,
    minConfidence: 0.5
  };

  //initialize the model: 
  faceapi = ml5.faceApi(video, faceOptions, faceReady);
  
  image1 = new Emoji(happyEmojis[0],random(0,width-250),0,1,1);
  image2 = new Emoji(happyEmojis[1],random(0,width-250),0,0.5,1);
  image3 = new Emoji(happyEmojis[2],random(0,width-250),0,0.5,1);
  image4 = new Emoji(happyEmojis[3],random(0,width-250),0,1,1.5);
  image5 = new Emoji(happyEmojis[4],random(0,width-250),0,1,0.5);
  image6 = new Emoji(happyEmojis[5],random(0,width-250),0,1,1);
  image7 = new Emoji(happyEmojis[6],random(0,width-250),0,1,1.5);
  image8 = new Emoji(happyEmojis[7],random(0,width-250),0,1,0.5);
  image9 = new Emoji(happyEmojis[8],random(0,width-250),0,2,1);
  image10 = new Emoji(happyEmojis[9],random(0,width-250),0,1,1.5);
  image11 = new Emoji(happyEmojis[10],random(0,width-250),0,1,0.5);
  image12 = new Emoji(happyEmojis[11],random(0,width-250),0,1,1.5);
  image13 = new Emoji(happyEmojis[12],random(0,width-250),0,2,1);
  image14= new Emoji(happyEmojis[13],random(0,width-250),0,1,2);
  image15= new Emoji(happyEmojis[14],random(0,width-250),0,1,1.5);
  image16= new Emoji(happyEmojis[15],random(0,width-250),0,1,1.5);
  image17 = new Emoji(happyEmojis[16],random(0,width-250),0,1,1);
  image18 = new Emoji(happyEmojis[17],random(0,width-250),0,1,1);
  image19 = new Emoji(happyEmojis[18],random(0,width-250),0,1,1.5);
  image20 = new Emoji(happyEmojis[19],random(0,width-250),0,1,0.5);
  image21 = new Emoji(happyEmojis[20],random(0,width-250),0,1,1.5);
  image22 = new Emoji(happyEmojis[21],random(0,width-250),0,1,0.5);
  image23 = new Emoji(happyEmojis[22],random(0,width-250),0,1,0.5);
}

// adjust canvas and video size when window is resized
function windowResized() {
    
    resizeCanvas(windowWidth, windowHeight);
    video.size(windowWidth, windowHeight);
}


function draw(){

  clear();
//   drawaing expressios and drawing graphics on the screen based on the detected emotion
  drawExpressions(detections, 20, 20, 14);
  if (expressionValues.length > 1 && expressionValues[1] === 1) { // 
    image1.display();
    image1.update();
    image2.display();
    image2.update();
    image3.display();
    image3.update();
    image4.display();
    image4.update();
    image5.display();
    image5.update();
    
  
  }
  if (expressionValues.length > 1 && expressionValues[4] === 1) { // 
    image11.display();
    image11.update();
    image12.display();
    image12.update();
    image13.display();
    image13.update();
    image14.display();
    image14.update();
    
  
  }
  if (expressionValues.length > 1 && expressionValues[3] === 1) { // 
    image15.display();
    image15.update();
    image16.display();
    image16.update();
    image17.display();
    image17.update();
    image18.display();
    image18.update();
    image23.display();
    image23.update();
    // playSong(2);
  }
  if (expressionValues.length > 1 && expressionValues[2] === 1) { // 
    image7.display();
    image7.update();
    image8.display();
    image8.update();
    image9.display();
    image9.update();
    image10.display();
    image10.update();
    
  }
  if (expressionValues.length > 1 && expressionValues[0] === 1) { // 
    image6.display();
    image6.update();
    image19.display();
    image19.update();
    image20.display();
    image20.update();
    image21.display();
    image21.update();
    image22.display();
    image22.update();
    
  }

//   playingn songs based on the emotion detected
  if (expressionValues.length > 1 && expressionValues[1] === 1) {
    
    playSong(3);
  } else if (expressionValues.length > 1 && expressionValues[4] === 1) {
    // play song 1
    playSong(0);
  } else if (expressionValues.length > 1 && expressionValues[3] === 1) {
    // play song 2
    playSong(1);
  } else if (expressionValues.length > 1 && expressionValues[2] === 1) {
    // play song 3
    playSong(2);
  } else if (expressionValues.length > 1 && expressionValues[0] === 1) {
    // play song 4
    playSong(4);
  }
  
}

function playSong(index) {
  //stop any currently playing song
  for (let i = 0; i < songs.length; i++) {
    if (i !== index && songs[i].isPlaying()) {
      songs[i].stop();
    }
  }

  // play the selected song
  if (!songs[index].isPlaying()) {
    songs[index].play();
  }
}

// class to handle the grpahics
class Emoji {
    constructor(img,x,y,vx, vy) {
        this.img = img;
        this.x = x;
      this.y = y;
        this.vx = vx;
        this.vy = vy;
    }
    
    update() {
        this.x += this.vx;
        this.y += this.vy;
        // check for canvas boundaries
        if (this.x < -130 || this.x > width -200) this.vx *= -1;
        if (this.y < -110 || this.y > height -150) this.vy *= -1;
    }
// display the graphics
    display() {
        image(this.img, this.x, this.y, 500, 500);
    }
}

function keyTyped() {
  // $$$ For some reason on Chrome/Mac you may have to press f twice to toggle. Works correctly on Firefox/Mac
  if (key === 'f') {
    toggleFullscreen();
  }
}

// Toggle fullscreen state. Must be called in response
// to a user event (i.e. keyboard, mouse click)
function toggleFullscreen() {
  let fs = fullscreen(); // Get the current state
  fullscreen(!fs); // Flip it!
}

// Start detecting faces
function faceReady() {
  faceapi.detect(gotFaces);
}

// Got faces
function gotFaces(error, result) {
  if (error) {
    console.log(error);
    return;
  }
//now all the data in this detections
  detections = result;　
  
//make back ground transparent
  clear();
  

  storeExpressions(detections); 
  faceapi.detect(gotFaces);
}

// maps detected expressions to a set of predefined categories and assigns a binary value based on a threshold

function mapExpressions(expressions) {
    const expressionOrder = ['neutral', 'happy', 'sad', 'disgusted', 'surprised'];
    let expressionValues = [];

    expressionOrder.forEach(expression => {
//       if value deteced is more than 50% make it 1 otherwise 0
        let value = expressions[expression] > 0.5 ? 1 : 0; 
        expressionValues.push(value);
    });

    return expressionValues;
}

// store expressions 
function storeExpressions(detections) {
    if (detections.length > 0) {
//   for the first person in the list, map expressions
        let expressions = detections[0].expressions;
        expressionValues = mapExpressions(expressions);
        // console.log(expressionValues);

        
    }
}

// it draws the percentage of detected emotion on the left top corner of the canvas
function drawExpressions(detections, x, y, textYSpace){
  if(detections.length > 0){
    
    let {neutral, happy, angry, sad, disgusted, surprised, fearful} = detections[0].expressions;
    textFont('Helvetica Neue');
    textSize(14);
    noStroke();
    fill(44, 169, 225);
// uses nf(value, left, right) to format numbers
    text("Neutral:       " + nf(neutral*100, 2, 2)+"%", x, y);
    text("Happiness: " + nf(happy*100, 2, 2)+"%", x, y+textYSpace);
    text("Sad:            "+ nf(sad*100, 2, 2)+"%", x, y+textYSpace*2);
    text("Disgusted: " + nf(disgusted*100, 2, 2)+"%", x, y+textYSpace*3);
    text("Surprised:  " + nf(surprised*100, 2, 2)+"%", x, y+textYSpace*4);
  
  }else{
    text("Neutral: ", x, y);
    text("Happiness: ", x, y + textYSpace);
    text("Sad: ", x, y + textYSpace*2);
    text("Disgusted: ", x, y + textYSpace*3);
    text("Surprised: ", x, y + textYSpace*4);
  }
}

function keyPressed() {
  if (key == " ") {
    // important to have in order to start the serial connection!!
    setUpSerial();
  }
}

// This function will be called by the web-serial library
// with each new *line* of data. The serial library reads
// the data until the newline and then gives it to us through
// this callback function
function readSerial(data) {
  ////////////////////////////////////
  //READ FROM ARDUINO HERE
  ////////////////////////////////////
  console.log(expressionValues);
  if (data != null) {
    

    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    let sendToArduino = expressionValues[0] + "," + expressionValues[1] + "," + expressionValues[2] + ","  + expressionValues[3] + ","  + expressionValues[4] + "\n";
    writeSerial(sendToArduino);
  }
}

Arduino:

// Define LED pin constants
int redLED1 = 12;   // First RGB LED, red pin
int greenLED1 = 11; // First RGB LED, green pin
int blueLED1 = 10;  // First RGB LED, blue pin

int redLED2 = 9;    // Second RGB LED, red pin
int greenLED2 = 8;  // Second RGB LED, green pin
int blueLED2 = 7;   // Second RGB LED, blue pin

int redLED3 = 6;    // Third RGB LED, red pin
int greenLED3 = 5;  // Third RGB LED, green pin
int blueLED3 = 4;   // Third RGB LED, blue pin

// Define sensor pin constants
int sensor1 = A2;  // First sensor
int sensor2 = A3;  // Second sensor
int sensor3 = A4;  // Third sensor

void setup() {
  Serial.begin(9600);
  pinMode(LED_BUILTIN, OUTPUT);

  // Set LED pins to output mode
  pinMode(redLED1, OUTPUT);
  pinMode(greenLED1, OUTPUT);
  pinMode(blueLED1, OUTPUT);

  pinMode(redLED2, OUTPUT);
  pinMode(greenLED2, OUTPUT);
  pinMode(blueLED2, OUTPUT);

  pinMode(redLED3, OUTPUT);
  pinMode(greenLED3, OUTPUT);
  pinMode(blueLED3, OUTPUT);

  // Start the handshake
  while (Serial.available() <= 0) {
    digitalWrite(LED_BUILTIN, HIGH); // on/blink while waiting for serial data
    Serial.println("0,0"); // send a starting message
    delay(300);            // wait 1/3 second
    digitalWrite(LED_BUILTIN, LOW);
    delay(50);
  }
}

void loop() {
  while (Serial.available()) {
    digitalWrite(LED_BUILTIN, HIGH); // LED on while receiving data
    int sensor1Value = analogRead(sensor1);  // Read first sensor
    int sensor2Value = analogRead(sensor2);  // Read second sensor
    int sensor3Value = analogRead(sensor3);  // Read third sensor

    int Neutral = Serial.parseInt();
    int Happy = Serial.parseInt();
    int Sad = Serial.parseInt();
    int disgusted = Serial.parseInt();
    int surprised = Serial.parseInt();

    if (Serial.read() == '\n') {
      // Reset all LEDs
      digitalWrite(redLED1, LOW);
      digitalWrite(greenLED1, LOW);
      digitalWrite(blueLED1, LOW);
      digitalWrite(redLED2, LOW);
      digitalWrite(greenLED2, LOW);
      digitalWrite(blueLED2, LOW);
      digitalWrite(redLED3, LOW);
      digitalWrite(greenLED3, LOW);
      digitalWrite(blueLED3, LOW);

      // Light up the corresponding LED only
      if (sensor1Value < 30 ) {
        digitalWrite(redLED1, Sad || disgusted || surprised);
        digitalWrite(greenLED1, Happy || disgusted);
        digitalWrite(blueLED1, Neutral || surprised);
      } else if (sensor2Value < 40) {
        digitalWrite(redLED2, Sad || disgusted || surprised);
        digitalWrite(greenLED2, Happy || disgusted);
        digitalWrite(blueLED2, Neutral || surprised);
      } else if (sensor3Value < 40) {
        digitalWrite(redLED3, Sad || disgusted || surprised);
        digitalWrite(greenLED3, Happy || disgusted);
        digitalWrite(blueLED3, Neutral || surprised);
      }

      Serial.print(sensor1Value < 30);
      Serial.print(',');
      Serial.print(sensor2Value < 40);
      Serial.print(',');
      Serial.println(sensor3Value < 40);
    }
  }
  // Optional: Use built-in LED to indicate the system is running
  digitalWrite(LED_BUILTIN, HIGH);
  delay(50);
  digitalWrite(LED_BUILTIN, LOW);
  delay(300);
}

 

The reading dives into the fascinating history of how design intersects with disability, and of how focusing on design for disability doesn’t just solve immediate problems but can also inspire broader changes in the design world.

What’s really interesting about this is the idea that solving specific problems for disabled users can actually lead to major innovations that benefit everyone. It’s a powerful reminder that good design is about more than just looks or functionality; it’s about thoughtful innovation that considers all users. This perspective encourages us to think differently about design and its potential impact, pushing for high standards and creativity in all areas, including those designed for disability.

Concept

Since my midterm project was related to our campus, I wanted to work on something similar for the final project. I came up with the idea to create an interactive NYUAD campus explorer. I thought it would be a great project to showcase the beauty and resourcefulness of our campus.

My project consists of a physical 3D model of our campus and a p5.js sketch that provides information about individual buildings.

https://editor.p5js.org/hk3863/full/5MT2OTbHL

Code

The first thing the user sees is the “MAINPAGE” function, where serial communication begins.

 

 background(mainpage)
  fill('white')
  textSize(width / 20);
  textAlign(CENTER, CENTER);
  textFont(boldFont);
  text("INTERACTIVE \nNYUAD \nCAMPUS EXPLORER", width / 4, height / 2);
  
  textSize(width / 30)
  textFont(regularFont);
  mainpage_message = "PRESS SPACEBAR TO START \nSERIAL COMMUNICATION"
  text(mainpage_message, width / 4, height / 1.2);
  
  if (serialcommunication == "TRUE") {
    mainpage_message = "CLICK TO START";
  }
}

Afterward, the user sees the ID explanation page, where the narrator welcomes them to NYUAD and explains how the virtual tour works. I used the subscription logic from this source (https://editor.p5js.org/hk3863/sketches/OSSNouhkg ), but I’ve adjusted the texts and timings to fit my project.

An essential and creative aspect of this project is the NYU ID card, which is required to access buildings on the NYUAD campus. I’ve used an RFID sensor to replicate the ID card.

When the user clicks on the page, they’re taken to the instruction page, where they can press buttons for specific buildings to obtain information. Here’s the interesting part: after pressing a button, they must scan their ID card to access the information, just as they would when using their ID card to enter buildings.

 

My model includes five buttons, each linked to a page with information about a building: Campus Center, Dining Halls, Research Buildings, Residences, and the Arts Center. Each page includes photographs and buttons to navigate to the next or previous photo.

function information() {
 
  background(87, 6, 140);
  imageMode(CENTER);
  textAlign(CENTER, CENTER);
  
  textSize(width / 15)
  textFont(boldFont);
  text(title, width / 2, height / 12);
  image(photo[i], imageCenterX, imageCenterY, picturesWidth, picturesHeight);

  fill('white');
  noStroke();

  // Right Triangle Button
  triangle(rightXBase + width / 50, imageCenterY, rightXBase, imageCenterY - height / 40, rightXBase, imageCenterY + height / 40);

  // Left Triangle Button
  triangle(leftXBase - width / 50, imageCenterY, leftXBase, imageCenterY - height / 40, leftXBase, imageCenterY + height / 40);
  

  imageMode(CENTER);
  image(photo[i], width * 0.3, height * 0.5, picturesWidth, picturesHeight);
  
  textAlign(LEFT, CENTER);
  textSize(width / 50)
  textFont(regularFont);
  text(information_text, width * 0.6, height * 0.5)
  
  textAlign(CENTER, CENTER);
  textSize(width / 30);
  textFont(boldFont);
  text('PRESS THE BUTTONS OF OTHER BUILDINGS', width / 2, height * 0.9)  

}

Arduino

My Arduino code is sending two types of data. The first is the digital input from the buttons. I’ve created arrays so that when a specific button is pressed, the button number is sent to p5.js, which then triggers the relevant function about that particular building. The second data type comes from the RFID sensor. I watched several tutorials on YouTube to understand how RFID works. In my code, you’ll see that when the tag name of my card is detected by the RFID sensor, the sensor sends a “1” to p5.js, granting access to the information.

#include <SPI.h>
#include <MFRC522.h>

// RFID Setup
#define SS_PIN 10
#define RST_PIN 9
MFRC522 mfrc522(SS_PIN, RST_PIN);  // Create MFRC522 instance

// Button Setup
const int buttonPins[] = {2, 3, 4, 5, 6};  // Digital pins for buttons
const int numButtons = sizeof(buttonPins) / sizeof(buttonPins[0]);
bool lastButtonState[numButtons] = {0};

// RFID Access Card UID
const String AUTHORIZED_CARD_UID = "70 91 42 55";
bool isAuthorized = false;
int cardAccessStatus = 0;
int buttonPressed = -1;

void setup() {
  // Initialize Serial Communication
  Serial.begin(9600);

  // Initialize RFID Reader
  SPI.begin();
  mfrc522.PCD_Init();


  // Initialize Button Pins (Input Mode, assuming external pull-down resistors)
  for (int i = 0; i < numButtons; i++) {
    pinMode(buttonPins[i], INPUT);
  }
}

void loop() {
  // Check RFID Access
  checkRFID();

  // Process Button Presses
  checkButtons();

  // Send Combined Data
  Serial.print(buttonPressed);
  Serial.print(",");
  Serial.println(cardAccessStatus);

  // Reset button state only if no button is currently pressed
  if (buttonPressed != -1) {
    buttonPressed = -1;
  }

  delay(100);  // Debounce delay
}

void checkRFID() {
  // Look for new cards
  if (!mfrc522.PICC_IsNewCardPresent()) {
    return;
  }
  // Select one of the cards
  if (!mfrc522.PICC_ReadCardSerial()) {
    return;
  }
  // Show UID on Serial Monitor
  String content = "";
  for (byte i = 0; i < mfrc522.uid.size; i++) {
    content.concat(String(mfrc522.uid.uidByte[i] < 0x10 ? " 0" : " "));
    content.concat(String(mfrc522.uid.uidByte[i], HEX));
  }

  content.toUpperCase();
  Serial.println("UID tag: " + content);

  // Check if the card is authorized
  if (content.substring(1) == AUTHORIZED_CARD_UID) {
    isAuthorized = true;
    cardAccessStatus = 1;  // Set to 1 for authorized access
    isAuthorized = false;
  } else {
    isAuthorized = false;
    cardAccessStatus = 0;  // Set to 0 for denied access
  }
}

void checkButtons() {
  for (int i = 0; i < numButtons; i++) {
    // Button logic without pull-ups: HIGH = Button pressed
    bool currentButtonState = digitalRead(buttonPins[i]) == HIGH;
    if (currentButtonState != lastButtonState[i]) {
      lastButtonState[i] = currentButtonState;
      if (currentButtonState) {
        buttonPressed = i;  // Store the pressed button index
      }
    }
  }
}

Circuit

What I Am Proud Of

I am particularly proud of creating the physical 3D model. It was my first time using Illustrator and laser cutting, and it was exciting to see the results. Through this project, I learned how to use Illustrator to turn my ideas into reality. Additionally, I’m happy I tried using a new sensor (RFID), which wasn’t covered in class. I believe it aligns well with the project’s concept.

Further Improvements

Because I didn’t have enough time, I couldn’t paint the campus model. However, with more time, I believe I could have made this model much more beautiful. Additionally, I couldn’t include information about some buildings, such as D1, so I decided to combine D2 and D1.

Overall, I am satisfied with the project’s outcome and pleased that I tried several new things and learned so much from this experience.

IM Show

A lot of people came to see my project! I really enjoyed the experience of presenting my work to others. Through this, I was able to identify some bugs I hadn’t noticed before. People loved the 3D model of the campus I created. They suggested it could be used for Marhaba or campus tours.

Concept Introduction : The Crypto Locker Puzzle is an interactive project designed to challenge users with a series of numbers-cracking tasks under time constraints. The objective is to solve hints and enter correct codes using a custom-built interface to unlock a cardboard box and win the prize inside. This game combines physical computing with digital elements to create an engaging and educational experience.

Link to P5.js Sketch: Click Here

Images of the project:

Initial Design , Credits to Dall-E.

Image of the unlocked candy box

Image of the wirings and breadboard (backside)

Image of the main user controller box.

User testing video:

1) Initial Testing: https://youtu.be/sKT85G0hJLI

2) Final Testing: https://youtu.be/hXSzTsB5x3o

Key Features:

1) Dynamic Countdown Timer: A prominently displayed timer adds a sense of urgency and excitement to the gameplay. It counts down from a set time, pushing players to think and act quickly, which enhances the challenge and engagement of the puzzle.

2) Cryptic Hints Display: As players progress, cryptic hints are displayed on the screen to aid in decoding the puzzle. This feature helps to balance the difficulty level and ensures that the puzzle remains solvable while still challenging.

3) Feedback on Input Accuracy: When players enter a code, the interface immediately provides feedback. If the code is incorrect, players are prompted to try again, and if correct, a success message is displayed, and the box unlocks. This immediate feedback loop keeps players engaged and informed.

Components I used:

1) Arduino Uno: Serves as the main controller for input and output operations.
2) Servo Motor: Operates the locking mechanism of the box.
3) Potentiometer: Allows users to select digits for the code
4) Button: Used for entering selected digits and navigating through the game.
5) P5.js: It acts like a digital interface that provides instructions, feedback, and manages the timer and attempt counter.

IM Show Pictures: People interacting with my project

Auto Locking Video: https://youtube.com/shorts/AAga9JHQt6c?feature=share

 

What’s the role of P5.js and Aurdino in this project?

P5.js Sketch: The P5.js sketch handles the game logic and user interface. It displays hints and the countdown timer.

Images:

Arduino Sketch: The Arduino controls the hardware aspects of the project. It uses a servo motor to lock and unlock the box. A potentiometer is used for dialing in digits (0-9), and a button confirms the entry of each digit. The Arduino sends these inputs to the P5.js program, which processes them to determine if the entered code matches the required one.

#include <Servo.h> // Include the Servo library

const int buttonPin = 2; // Push button connected to digital pin 2
int buttonState = 0;     // Variable to store the button state
const int potPin = A0;   // Potentiometer connected to analog pin A0
Servo servoMotor;        // Create a servo motor object

void setup() {
  pinMode(buttonPin, INPUT); // Set button pin as input
  Serial.begin(9600);        // Initialize serial communication
  servoMotor.attach(9);      // Attach the servo motor to digital pin 9
  servoMotor.write(0);       // Set initial position of the servo motor to 0 degrees
}

void loop() {
  buttonState = digitalRead(buttonPin); // Read the state of the button
  
  // If the button is pressed (buttonState is LOW)
  if (buttonState == LOW) {
    Serial.println("Button pressed!"); // Send message to serial monitor
    delay(1000); // Delay to debounce the button
  }
  
  // Read the value from the potentiometer and send it to the serial port
  int potValue = analogRead(potPin); // Read the value from the potentiometer
  int mappedValue = map(potValue, 0, 1023, 0, 10); // Map the value to the range 0-9
  Serial.println(mappedValue); // Send the mapped value to the serial port
  
  // Check if a signal is received from p5.js to control the servo motor
  while (Serial.available() > 0) {
    int signal = Serial.read(); // Read the signal from serial port
    if (signal == '0') { // Signal to reset servo motor to 0 degrees
      servoMotor.write(5); // Turn the servo motor to 0 degrees
    } else if (signal == '1') { // Signal to turn servo motor to 90 degrees
      servoMotor.write(90); // Turn the servo motor to 90 degrees
    }
  }
  
  delay(100); // Delay for stability
}

How the communication between P5.js and Aurdino Worked??

Initially I prepared the Arduino sketch to transmit data over the serial port. I wrote code to read sensor inputs, such as button presses or potentiometer values, and sent this data as messages over the serial port using functions like Serial.println() or Serial.write(). In my P5.js sketch, I initialized a serial port object within the setup() function using the new p5.SerialPort() constructor. This object represented the communication channel between my P5.js sketch and the Arduino connected to my computer.

Later I configured the serial port to listen for incoming data by using the serial.on('data', serialEvent) function. This setup ensured that whenever data was received on the serial port, the serialEvent function was automatically called, allowing my P5.js sketch to react to data sent from the Arduino.  Within the serialEvent function of my P5.js sketch, I read incoming data from the serial port using the serial.readLine() function. This function retrieved a line of text sent from the Arduino over the serial port. I then processed this data based on its content, performing actions such as updating the display or triggering events in my P5.js sketch.

Additionally, I enabled bidirectional communication by allowing my P5.js sketch to send data back to the Arduino. This allowed me to control actuators connected to the Arduino, such as servo motors or LEDs, from my P5.js sketch. I used the serial.write() function to send data from my P5.js sketch to the Arduino and vice-versa, facilitating interactive control over hardware components.

Schematics of the complete circuit:

How the initial game works:

Challenges and Code I’m Particularly Proud of:

Among various aspects of my project, there’s one particular area that stands out – the locking mechanism.

Initially, the locking mechanism was designed to activate when the user entered all the codes correctly within a specific time frame. Subsequently, P5.js was supposed to send a signal to Arduino instructing it to rotate the servo motor to 90 degrees. However, I encountered challenges in implementing this functionality. Despite extensive debugging efforts, I struggled to achieve the desired outcome. It took numerous tutorial videos and multiple attempts, but eventually, I successfully resolved the issue.

// Check if the final number matches the correct code
  if (finalNumber === currentNumCode) {
    currentImage = congratsImg; // Display 'congrats.png'
    // Stop all other sounds and play win
    stopAllSounds();
    winSound.play();
    // Send signal to turn servo motor to 90 degrees
    serial.write('1');
  } else {
    currentImage = wrongImg; // Display 'wrong.png'
    // Stop all other sounds and play error
    stopAllSounds();
    errorSound.play();
  }

  // Reset values for the next round
  dashValues = [0, 0, 0, 0];
  currentDashIndex = 0;
  allowNumberSelection = false;
  clearInterval(timerInterval); // Stop the timer
}

Following the adjustment, I found that relying solely on correct numerical input wasn’t effective. Instead, I implemented a solution where upon displaying the “congrats.png” image in P5.js, the servo motor rotates 90 degrees, indicated by the code serial.write('1');. Conversely, if the “congrats.png” image is not displayed, the servo motor remains at 0 degrees, signified by serial.write('0');.

function serialEvent() {
  let message = serial.readLine(); // Read the incoming serial data

  if (message.includes("Button pressed!")) {
    // Toggle between images based on currentImage
    if (currentImage === welcomeImg) {
      serial.write('0');
      currentImage = instImg;
      // Stop all other sounds and play bgm
      stopAllSounds();
      bgmSound.play();
      // Reset the dashes to 0 when returning to welcomeImg
      dashValues = [0, 0, 0, 0];
      currentDashIndex = 0;
      allowNumberSelection = false;
      // Send signal to reset servo motor to 0 degrees
      
    } else if (currentImage === instImg) {
      allowNumberSelection = true; // Allow number selection

After successfully implementing this functionality, I extended its use beyond just locking mechanisms. I ensured that the system would automatically lock after the user restarted the game, enhancing the overall user experience.

Future Improvements and Comments  from Users:

1. As initially planned in the game development, I would like to introduce a concept of levels. This will give players a strong sense of achievement as they complete each level. This is what one of my friend suggested after playing the game after multiple times.

2. To enhance this project even more, I aim to create a compelling story that will engage players deeply in the crypto game and enhance their overall gaming experience. In addition I would also like to add the SFX, for the number selection and opening the box.

3. Finally, I plan to add more locks (servo motors) to the candy box to make it more challenging to open. This will further motivate players to experience the thrill of unlocking the box.