The essay, or more accurately a “rant” as described by Bret Victor, mainly focuses on user interaction both presently and in the distant future. Victor argues that the sense of touch, essential to human work for millions of years, is important to understanding his argument. He emphasizes that technology doesn’t simply happen but is rather a result of careful development. In introducing the concept of “Pictures Under Glass,” which represents “an interaction paradigm of permanent numbness,” the author highlights how this denies our hands their natural capabilities. Consequently, he argues against accepting an Interface Of The Future that is less expressive than a when making a sandwich. To avoid that, Victor highlights how we must understand that the future hinges on our choices as people determine which ideas to pursue, which research to support, and how they’ll shape their professional paths. Last but not least, he wraps up with an interesting question: in a world where we control our entire bodies, should our interactions boil down to just a finger?
I understand where the author is coming from; revolutionary technology comes out of long research, but as we can see nowadays, technology is all around us, and what the author seems to be leaning toward is making technology a part of us. Don’t get me wrong, I support technology being improved & people being innovative, but I must fear that there will come a day when the line will be crossed and it will not simply be “pictures under glass,” but rather “pictures under skin.” For better or for worse, this is one of my concerns, and I believe it is a concern shared by many, which is why we have remained in an era where interactions have “permanent numbness.”The “glass” referred to in “pictures under glass” should always be there; it is when that barrier is broken that ideas will have no limit, which is a good thing, but it is also when many other things will have no limit, and that’s what’s dangerous. That said, I believe the future shouldn’t be limited to just a single finger, but also not utilized by all of them.
I have two potential project ideas/prompts: Plan A involves designing a game that revolves around unlocking a safe. In this game, players will be tasked with guessing the correct digits of a passcode to unlock the safe. The passcode will be generated randomly at the start of the game and transmitted to an Arduino, which will control a servo motor to lock the safe box. Players can input their guesses using either buttons or potentiometers (either limit the digits to 1-5 and use five buttons OR use the potentiometer to choose the digits from 0-9 and press a button to enter OR use four potentiometers for each digit without a button to enter), depending on the chosen design. The safe itself could be modeled after a traditional bank safe or a more modern safe box.
Alternatively (if plan a didn’t work out), Plan B would involve expanding upon my midterm project, which included the Elevator Rush game. This game was inspired by the frustration of waiting for elevators during busy times, such as the rush between classes. Players control an elevator, picking up students to ensure they reach class on time. I’m considering bringing this game to life by creating a physical model of the elevator/C2 space, which would allow users to interact with it physically.
For this final project, I envisioned entering a universe where art and technology collide to create something genuinely unique. I wanted to make it easy for people who have never drawn before to experience it for the first time.
Imagine an interactive space where art and technology merge, transforming bodily movements into a rainbow of colours on a digital canvas. My initiative encourages individuals to exhibit their ideas through motion. They use a paintbrush attached to an Arduino to navigate a symphony of colourful ellipses. This is more than simply an artwork; it’s an experience that captures the essence of movement and transforms it into a personalised digital masterpiece.
Arduino Part:
The foundation of the interaction is the Arduino Uno, on which I worked with a ZX Distance and Gesture Sensor. The sensor is adept at monitoring the paintbrush’s proximity as well as the artist’s minor hand gestures. To be honest, installation was rather simple, but the sensor itself was not as powerful as planned.
Input: Proximity data and gesture commands from the ZX Sensor. Output: Serial communication to relay the sensor data to the computer running the p5.js sketch. Data to p5.js: Real-time Z-axis data for proximity (distance of the hand or brush from the sensor) and X-axis data for lateral movement, along with gesture detections (swipes, taps). From p5.js: Instructions may be sent back to calibrate gesture sensitivity or toggle the sensor’s active state.
P5.js Part:
On the digital front, p5.js will serve as the canvas and palette, with dynamic and malleable capabilities. It will translate the incoming data from the Arduino into a series of colours and movements on the screen.
Receiving Data: Interpreting proximity and gesture data from the Arduino.
Processing Movements: Real-time mapping of hand movements to colour strokes and splashes with varied intensity and spread on a digital canvas. Visual Feedback: Dynamic visual changes that reflect the flow and dance of the user’s motions. To Arduino: Signals for modifying the ZX Sensor parameters in response to real-time performance and user feedback.
Graphics Used:
Gesture: Swipe Left / Right, Tap
Visuals: Dynamic shapes, colors, and brush strokes based on movement data.
Development and User Testing
The ZX Distance and Gesture Sensor has now been integrated with Arduino, and the immediate goal is to ensure that data flows smoothly into the p5.js programme. By the time user testing begins next week, the system should respond to hand motions by presenting relevant visual modifications on the screen.
User Testing Objectives:
Assess how natural and fulfilling it is to paint in midair.
Ensure responsiveness and accuracy of gesture detection.
Gather feedback from participants regarding the ease of use and satisfaction with the interactive art experience.
User Testing Techniques:
Record interactions on video to analyze gesture accuracy and timing.
How it Works:
Arduino Setup: Connect Arduino to the ZX Sensor and establish serial communication with p5.js.
Gesture Detection: The Arduino reads gestures and proximity data and sends this information to the p5.js sketch.
Canvas Response: p5.js interprets the data and creates a dynamic visual display that reflects the gestures and brush movements.
Feedback Loop: p5.js sends calibration data back to Arduino to adjust the sensor settings if necessary.
Code Arduino Code Example:
#include <Wire.h>
#include <ZX_Sensor.h>
// Constants
const int ZX_ADDR = 0x10; // ZX Sensor I2C address
// Global Variables
ZX_Sensor zx_sensor = ZX_Sensor(ZX_ADDR);
uint8_t x_pos;
uint8_t z_pos;
uint8_t handPresent = false;
void setup() {
Serial.begin(9600);
zx_sensor.init();
while (Serial.available() <= 0) {
Serial.println("0,0,0"); // send a starting message
delay(50);
}
}
void loop() {
// If there is position data available, read and print it
if ( zx_sensor.positionAvailable() ) {
uint8_t x = zx_sensor.readX();
if ( x != ZX_ERROR ) {
x_pos=x;
}
uint8_t z = zx_sensor.readZ();
if ( z != ZX_ERROR ) {
z_pos=z;
}
handPresent=true;
} else {
handPresent=false;
}
while (Serial.available()) {
int inbyte = Serial.parseInt();
if (Serial.read() == '\n') {
Serial.print(x_pos);
Serial.print(',');
Serial.print(z_pos);
Serial.print(',');
Serial.println(handPresent);
}
}
}
P5 Code:
// FINAL PROJECT BY STEFANIA PETRE
// FOR INTRO TO IM
let img;
let brushSize = 19;
let colorHue = 0;
let previousX = 0,
previousY = 0;
let xPos = 0;
let zPos = 0;
let smoothedX = 0;
let handPresent = 0;
let showDrawing = false;
let startButton;
let mappedX = 0;
let mappedZ = 0;
function preload() {
img = loadImage("start.webp");
}
function setup() {
createCanvas(640, 480);
colorMode(HSB, 360, 100, 100, 100);
textSize(18);
// Set up the start button
startButton = createButton("Get Creative!");
startButton.position(290, 175);
startButton.mousePressed(startDrawing);
let fullscreenButton = createButton("Fullscreen");
fullscreenButton.position(10, 10);
fullscreenButton.mousePressed(toggleFullScreen);
// Set the initial hue
colorHue = random(360);
}
function draw() {
if (!showDrawing) {
background(img);
} else {
if (!serialActive) {
background(0);
fill(255);
//text("Press the 'Get Creative!' button to start drawing", 20, 30);
} else {
if (handPresent == 1) {
// Adjust mapping ranges according to your actual data
mappedX = map(xPos, 180, 40, 0, width);
mappedZ = map(zPos, 240, 25, 0, height);
mappedX = constrain(mappedX, 0, width);
mappedZ = constrain(mappedZ, 0, height);
let weight = 10; // Adjust as needed
let strokeColor = color(colorHue % 360, 100, 100);
stroke(strokeColor);
strokeWeight(weight);
ellipse(mappedX, mappedZ, weight * 2, weight * 2);
previousX = mappedX;
previousY = mappedZ;
}
colorHue += 2;
noStroke();
fill(0, 0, 0.000000000000005, 1);
rect(0, 0, width, height);
}
}
}
function startDrawing() {
showDrawing = true;
startButton.hide();
setUpSerial();
}
function toggleFullScreen() {
let fs = fullscreen();
fullscreen(!fs);
resizeCanvas(windowWidth, windowHeight);
startButton.position(windowWidth / 2 - 40, windowHeight / 2 - 30);
}
function readSerial(data) {
if (data != null) {
let fromArduino = split(trim(data), ",");
if (fromArduino.length == 3) {
xPos = int(fromArduino[0]);
zPos = int(fromArduino[1]);
handPresent = int(fromArduino[2]);
}
let sendToArduino = 0 + "\n";
writeSerial(sendToArduino);
}
}
function keyPressed() {
if (key == " ") {
setUpSerial();
}
}
function windowResized() {
resizeCanvas(windowWidth, windowHeight);
}
Final thoughts:
Even though the project was not exactly how I have pictured it at the beginning, it still worked out well. People at the showcase liked it and everything worked for 3 hours and I am happy that I have chosen this path.
The concept of this project revolves around creating an interactive UFO game titled “UFO Escape.” The main goal is to score points by avoiding collisions with asteroids and navigating through space. What sets this game apart is its unique control scheme: players use a glove equipped with a gyroscope sensor as the game controller. This setup allows the game to translate the player’s hand tilt movements into navigational commands within the game environment to turn the UFO in different directions.
Final Concept of the project:
Video of interaction:
Description of interaction design
Implementing the “UFO Escape” game involves an interaction design that merges physical computing with digital interfaces to create an immersive gaming experience. Here’s a detailed breakdown of the interaction design and implementation:
Hardware Setup:
Gyroscope Sensor (3-Axis Gyroscope L3GD20H):
This sensor is integrated into a wearable glove made from velcro, foam tape, and cloth. The glove is then connected to an Arduino UNO using 4 jumper wires soldered to a 90-cm length. It is then covered by heat-shrinkable rubber tubing for aesthetics and a better wire management setup.
I kept changing the glove’s prototypes due to either wiring issues or the fact that none were non-adjustable. Below are the different versions of the glove.
First Prototype:
This version did not allow the cables to function properly due to the positioning of the gyroscope.
Second Prototype:
This glove secured the gyroscope, but due to it not being adjustable, it kept causing issues with the wires whenever a different sized hand would wear it.
Final Prototype:
This glove concept fixed all the issues mentioned previously, from sizing difficulties to wire management, making it the ideal version for various hand sizes. It was made from scratch in the IM lab using different materials to secure the glove and a metal piece to lock in the velcro I got from home from a metal glove.
Cabling:
The cable was soldered on a stranded wire to extend the jumper wires for a more usable length for the glove.
How the code works:
The gyroscope sensor extracts the X and Y variables read by the Arduino to determine the hand’s position and movement. The Arduino Uno sends those as instructions for where the UFO should be positioned, which the P5 file reads and translates.
// Function to move the player based on arrow key inputs
move() {
let moveSpeed = 10; // Speed multiplier
this.gotoX += gyroData.x * moveSpeed;
this.gotoY += gyroData.y * moveSpeed;
this.gotoX = constrain(this.gotoX, 30, width - 30);
this.gotoY = constrain(this.gotoY, 30, height - 30);
this.x = lerp(this.x, this.gotoX, 0.1);
this.y = lerp(this.y, this.gotoY, 0.1);
if (frameCount % 100 === 0) {
obstacles.push(new Obstacle()); // Add new obstacles
}
this.gotoX = constrain(this.gotoX, 30, width - 30); // Keeps player within horizontal bounds
this.gotoY = constrain(this.gotoY, 30, height - 30); // Keep player within vertical bounds
// //This is only so the player cannot exist the canvas
}
This processes the gyroscope data to interpret the player’s hand movements as specific commands (left, right, up, down, accelerate).
Serial Communication:
The Arduino transmits the processed data to a computer via serial communication, and in my case, it provided the P5 with the full library of serial communication, which made the process work for me.
/**
* p5.webserial
* (c) Gottfried Haider 2021-2023
* LGPL
* https://github.com/gohai/p5.webserial
* Based on documentation: https://web.dev/serial/
*/
'use strict';
Game Mechanics:
The game is developed in a way that makes it hard for the player to achieve a high score by making the meteorites come down faster and faster as the player progresses to higher scores.
// Function to update obstacle position
update() {
this.y += map(score, 0, 10, gameSpeed, gameSpeed + 5); // Moves obstacles down the screen
//this will make game harder as time grows
}
}
Also, the player should be included in the provided map of the game so no cheating occurs and the game is played fairly based on skill levels.
this.gotoX = constrain(this.gotoX, 30, width - 30); // Keeps player within horizontal bounds this.gotoY = constrain(this.gotoY, 30, height - 30); // Keep player within vertical bounds // //This is only so the player cannot exist the canvas }
Player Interaction:
Players wear the gyroscope-equipped glove and move their hands to control the UFO within the game. Movements are intuitive: tilting the hand to the sides steers the UFO laterally, while tilting forward or backward would control vertical movement, and depending on how fast you tilt your hands, you could possibly send the UFO in any direction of the map faster than needed making you very aware of what move you would wanna make next as the meteoroids are coming down.
Visuals/Audio:
The game provides real-time visual feedback by updating the position and movements of the UFO based on the player’s actions as well as background noise for when the UFO is flying around.
P5 Sketch:
let port; // Serial port instance
let gyroData = { x: 0, y: 0 }; // Placeholder for gyro data
let serialInterval; // Interval for polling serial data
// Variables for game assets
let bgImage; // Variable to hold the background image for menu and game over screens
let bgMusic; // Variable to hold the background music for gameplay
let player; // Player object
let obstacles = []; // Array to store obstacles
let gameSpeed = 6; // Speed at which obstacles move
let score = 0; // Player's score
let gameState = "MENU"; // Initial game state; that is "MENU", "PLAYING", or "GAME OVER"
let rockImage; // Variable to hold the rock image
let gameOverImage;
// Preload function to load game assets before the game starts
function preload() {
// bgImage = loadImage("space.png"); // Loads the background image
menuImage = loadImage("menu.jpeg");
// gameplayBgImage = loadImage("gameplay.jpg"); // Loads the gameplay background image
bgMusic = loadSound("gameplaysound.mp3"); // Loads the background music
rockImage = loadImage("rock-2.png"); // Loads the rock image
}
// The setup function to initialize the game
function setup() {
createCanvas(750, 775).parent("canavs-container"); // Size of the game canvas
openButton = createButton("Connect Arduino")
.position(410, 20)
.style("background-color", "rgba(244,238,238,0.1)(255, 50)")
.style("border-radius", "70px")
.style("padding", "10px 20px")
.style("font-size", "15px")
.style("color", "white")
.mousePressed(openSerialPort); // Click to open serial port
player = new Player(); // Initializes the player object
textAlign(CENTER, CENTER); // Setting text alignment for drawing text
textFont("arial");
}
function openSerialPort() {
port = createSerial(); // Initialize the serial port instance
if (port && typeof port.open === "function") {
port.open("Arduino", 9600); // Open with a predefined preset
// Set up polling to check for serial data every 100ms
serialInterval = setInterval(readSerialData, 100); // Poll for data
} else {
console.error("Failed to initialize the serial port.");
}
}
function readSerialData() {
if (port && port.available()) {
let rawData = port.readUntil("\n"); // Reads data till newline
if (rawData && rawData.length > 0) {
let values = rawData.split(","); // Splits by commas
if (values.length === 2) {
gyroData.x = parseFloat(values[0]); // Parse X value
gyroData.y = parseFloat(values[1]); // Parse Y value
} else {
console.error("Unexpected data format:", rawData); // Error handling
}
}
}
}
// Draw function called repeatedly to render the game
function draw() {
// Displays the space background image only during menu and game over states but displays a different image during gameplay
if (gameState === "PLAYING") {
background("#060C15");
noStroke();
fill(255);
for (let star of stars) {
circle(star.x, star.y, star.r);
star.y += star.yv;
if (star.y > height + 5) star.y = -5;
}
}
// Handles game state transitions
if (gameState === "MENU") {
drawMenu();
} else if (gameState === "PLAYING") {
if (!bgMusic.isPlaying()) {
bgMusic.loop(); // Looping the background music during gameplay
}
playGame();
} else if (gameState === "GAME OVER") {
bgMusic.stop(); // Stops the music on game over
drawGameOver();
} else {
//info
drawInfo();
}
}
function drawInfo() {
background("#060C15");
noStroke();
fill(255);
for (let star of stars) {
circle(star.x, star.y, star.r);
star.y += star.yv;
if (star.y > height + 5) star.y = -5;
}
textSize(16);
text(
"Connect the Arduino\n Wear the glove\n, Control the UFO through tilting your hand \nLeft, Right, Up and Down.",
width / 2,
height / 2
);
stroke(255);
Button("MENU", width / 2, height - 100, 100, 40);
}
// ---------------Function to display the game menu
function drawMenu() {
background(menuImage);
fill(200, 100, 100);
stroke(200, 100, 100);
textSize(62);
strokeWeight(2);
text("UFO ESCAPE", width / 2, 140);
fill(255);
text(
"UFO ESCAPE",
width / 2 - map(mouseX, 0, width, -5, 5),
135 - map(mouseY, 0, height, -2, 2)
);
let x, y;
if (frameCount % 40 < 20) {
x = width / 2 - map(frameCount % 40, 0, 20, -5, 5);
y = 400;
} else {
x = width / 2 - map(frameCount % 40, 20, 40, 5, -5);
y = 400;
}
Ufo(x, y, 60, 30);
//startButton
Button("START", width / 2, 550);
//info page
Button("INFO", width - 115, 40, 100, 40);
}
// Function to handle gameplay logic
function playGame() {
fill(255);
textSize(25);
text(`Score: ${score}`, width / 2, 50);
player.show(); // Displays the player
player.move(); // Moves the player based on key inputs
// Adding a new obstacle at intervals
if (frameCount % 120 == 0) {
obstacles.push(new Obstacle());
}
// Updates and displays obstacles
for (let i = obstacles.length - 1; i >= 0; i--) {
obstacles[i].show();
obstacles[i].update();
// Checks for collisions
if (player.collidesWith(obstacles[i])) {
gameOverImage = get();
gameState = "GAME OVER";
}
// Removes obstacles that have moved off the screen and increment score
if (obstacles[i].y > height) {
obstacles.splice(i, 1);
i--;
score++;
}
}
}
// Function to display the game over screen
function drawGameOver() {
if (gameOverImage) image(gameOverImage, 0, 0);
fill(200, 100, 100);
stroke(200, 100, 100);
textSize(46);
text("GAME OVER", width / 2, height / 2 + 50);
fill(255);
text(score, width / 2, height / 2 - 50);
Button("RESTART", width / 2, 550);
}
// Function to reset the game to its initial state
function resetGame() {
obstacles = []; // Clear existing obstacles
score = 0; // Reset score
player = new Player(); // Reinitialize the player
stars = [];
for (let i = 0; i < 100; i++) {
let r = random(1, 3);
stars.push({
x: random(width),
y: random(height),
r: r,
yv: map(r, 1, 3, 0.01, 0.1),
});
}
}
// Player class
class Player {
constructor() {
this.width = 60; // Width of the UFO
this.height = 30; // Height of the UFO
this.x = width / 2; // Starting x position
this.y = height - 100; // Starting y position
this.gotoX = this.x;
this.gotoY = this.y;
this.speed = 5;
}
// Function to display the UFO
show() {
this.y = lerp(this.y, this.gotoY, 0.1);
this.x = lerp(this.x, this.gotoX, 0.1); //and change this.gotoX
stroke(200, 100, 100);
Ufo(this.x, this.y, this.width, this.height);
}
// Function to move the player based on arrow key inputs
move() {
let moveSpeed = 10; // Speed multiplier
this.gotoX += gyroData.x * moveSpeed;
this.gotoY += gyroData.y * moveSpeed;
this.gotoX = constrain(this.gotoX, 30, width - 30);
this.gotoY = constrain(this.gotoY, 30, height - 30);
this.x = lerp(this.x, this.gotoX, 0.1);
this.y = lerp(this.y, this.gotoY, 0.1);
if (frameCount % 100 === 0) {
obstacles.push(new Obstacle()); // Add new obstacles
}
this.gotoX = constrain(this.gotoX, 30, width - 30); // Keeps player within horizontal bounds
this.gotoY = constrain(this.gotoY, 30, height - 30); // Keep player within vertical bounds
// //This is only so the player cannot exist the canvas
}
// Function to detect collision with obstacles
collidesWith(obstacle) {
return (
dist(
obstacle.x + obstacle.radius,
obstacle.y + obstacle.radius,
this.x,
this.y
) < 45
);
}
}
// Obstacle class
class Obstacle {
constructor() {
this.radius = random(15, 30); // Random radius for obstacle
this.x = random(this.radius, width - this.radius); // Random x position
this.y = -this.radius; // Starts off-screen so it looks like its coming towards you
}
// Function to display the rocks
show() {
image(rockImage, this.x, this.y, this.radius * 2, this.radius * 2); // Draws them as a circle
}
// Function to update obstacle position
update() {
this.y += map(score, 0, 10, gameSpeed, gameSpeed + 5); // Moves obstacles down the screen
//this will make game harder as time grows
}
}
function Button(txt, x, y, w = 200, h = 60) {
fill(255, 50);
if (
mouseX > x - w / 2 &&
mouseX < x + w / 2 &&
mouseY > y - h / 2 &&
mouseY < y + h / 2
) {
fill(255, 80);
if (mouseIsPressed) {
mouseIsPressed = false; //so only one click happnes
action(txt);
}
}
rect(x, y, w, h, h / 2);
fill(255);
textSize(h / 2);
text(txt, x, y);
}
function action(txt) {
switch (txt) {
case "START":
gameState = "PLAYING";
resetGame();
break;
case "RESTART":
gameState = "MENU";
resetGame();
break;
case "INFO":
stars = [];
for (let i = 0; i < 100; i++) {
let r = random(1, 3);
stars.push({
x: random(width),
y: random(height),
r: r,
yv: map(r, 1, 3, 0.01, 0.1),
});
}
gameState = "INFO";
break;
case "MENU":
gameState = "MENU";
break;
}
}
function Ufo(x, y, w, h) {
fill(255); // Sets color to white
rectMode(CENTER);
rect(x, y, w, h, 20); // Draws the UFO's body
fill(20); // Sets the glass color to red
arc(x, y - h / 4, w / 2, h / 1, PI, 0, CHORD); // Draws the glass
stroke(255);
let a = map(x, 0, width, 0, PI / 4);
arc(x, y - h / 4, w / 2 - 5, h - 5, -PI / 4 - a, -PI / 6 - a);
for (let i = 1 + frameCount; i < 10 + frameCount; i++) {
let x_ = map(i % 10, -1, 10, -30, 30);
circle(x + x_, y, 5);
}
}
Arduino Code:
#include <Wire.h>
#include <Adafruit_L3GD20_U.h>
// Initialize the L3GD20 object
Adafruit_L3GD20_Unified gyro = Adafruit_L3GD20_Unified(20); // Sensor ID
void setup() {
Serial.begin(9600); // Start serial communication at 9600 baud
// Initialize the L3GD20 gyroscope
if (!gyro.begin()) {
Serial.println("Failed to find L3GD20 gyroscope");
while (1) {
delay(10); // Halt if initialization fails
}
}
}
void loop() {
sensors_event_t event;
gyro.getEvent(&event);
// Send the X and Y gyro data with a newline at the end
Serial.print(event.gyro.x, 4); // X-axis gyro data with 4 decimal places
Serial.print(","); // Comma separator
Serial.println(event.gyro.y, 4); // Y-axis gyro data with 4 decimal places
delay(100); // Adjust delay as needed
}
What are some aspects of the project that you’re particularly proud of?
To be honest, the whole final project makes me proud that I was able to create such a complicated code with actual use in the end. The glove prototypes are also something that I was proud of because I have never created a glove from scratch before. I am just glad that it all worked out in the end and that the cables stopped popping out of the gloves whenever someone would put them on.
What are some areas for future improvement?
I would say that expanding on the game itself and making different levels and maps for the UFO to fly around in would also exemplify the game. Also, creating a button in the glove to restart once you press it would make it easier for the player to control the whole game if they lose.
My game concept simulates driving safely through a hailstorm in AbuDhabi. As a player, you control a toy car, displayed on the screen, by physically moving it left or right to dodge falling hailstones. This game is designed to reflect the unpredictable nature of real hailstorms and incorporates real-time physical interaction through a toy car and Arduino sensors, offering a unique and engaging gameplay experience, also symbolizing the hailstorm in Abu Dhabi months ago.
Implementation Overview
I built the game using p5.js for the visual components and game logic, and Arduino for the physical interaction aspects. The Arduino setup uses a pushbutton to start or restart the game and an HC-SRO ultrasonic sensor to determine the position of the toy car, which translates into the car’s movement on the screen. These inputs are sent to the p5.js application through serial communication, allowing the player’s physical movements to directly influence the gameplay.
Interaction Design Description
The interaction design focuses on tangible interaction, where the physical movement of the toy car (left or right) translates to the movement of the car on the screen. This method fosters more engaging and intuitive gameplay. An arcade button connected to the Arduino allows players to start or restart the game easily, making the interface user-friendly and accessible.
Arduino Code Explanation
In my Arduino code, I manage inputs from the ultrasonic sensor and the button. The ultrasonic sensor measures the distance of an object (the toy car) from the sensor, and this measurement is used to control the car’s position on the p5.js screen. The button input is debounced to avoid processing multiple unintended signals, used to start or restart the game, and toggles an LED for visual feedback. Serial communication sends the button press count and distance measurement to the p5.js application.
p5.js Code Explanation
My p5.js code is responsible for creating the visual representation of the game—rendering the car, hailstones, and other visual elements on the screen. It also handles the game logic, such as detecting collisions between the car and hailstones, updating the game state based on Arduino inputs, and managing game timers and scores.
The game has 3 screens:
The Main menu where the player is told the directions to play, the winning screen which comes up after 10 seconds of the user playing, and the game over screen which pops up if the car collides with the hailstones.
Communication Between Arduino and p5.js
I achieve communication between the Arduino and p5.js through serial communication. The Arduino continuously sends data from the button and the ultrasonic sensor to the p5.js application, which reads this data to update the game state accordingly. The p5.js listens for serial data, parses it, and uses these inputs to control the car’s movements and manage game controls like start and restart.
Important Images
Aspects I’m Particularly Proud Of
I am proud of several key accomplishments in this project:
Successfully integrating physical components with a digital interface, which enhanced the interactive gaming experience.
Overcoming the challenges associated with serial communication between Arduino and p5.js, a complex aspect of hardware-software integration.
Completing the project within a limited timeframe and being able to innovate with a unique approach to game design and interaction.
As well as my setup with the street and box; I really enjoyed making the box and the street, adding an extra layer of creativity
Future Improvement Areas
For future enhancements, I could consider:
Implementing sound and music for the game, I initially had music in the game but I decided to remove it, since the exhibition is already quite chaotic and the music won’t be heard
Enhancing the game’s visual and sound effects to create a more immersive experience.
Implementing additional gameplay features, such as different levels of difficulty or various weather conditions affecting gameplay.
Exploring different sensors or refining the calibration of the HC-SRO4 sensor. As it was slightly glitchy at first, but I managed to fix it.
Here is my Arduino code:
const int trigPin = 9;
const int echoPin = 10;
// Arduino code for button, which detects the counts
const int buttonPin = 2; // the number of the pushbutton pin
const int ledPin = 3; // the number of the LED pin
// variables will change:
int buttonState = 0; // variable for reading the pushbutton status
int lastButtonState = HIGH; // variable for reading the last pushbutton status
unsigned long lastDebounceTime = 0; // the last time the output pin was toggled
unsigned long debounceDelay = 50; // the debounce time; increase if the output flickers
int pressCount = 0; // count of button presses
void setup() {
pinMode(trigPin, OUTPUT);
pinMode(echoPin, INPUT);
pinMode(ledPin, OUTPUT); // initialize the LED pin as an output
pinMode(buttonPin, INPUT_PULLUP); // initialize the pushbutton pin as an input with internal pull-up resistor
Serial.begin(9600);
}
void loop() {
float distance = getDistanceCm(); // Get the distance in cm
int reading = digitalRead(buttonPin);
// check if the button state has changed from the last reading
if (reading != lastButtonState) {
// reset the debouncing timer
lastDebounceTime = millis();
}
if ((millis() - lastDebounceTime) > debounceDelay) {
// if the button state has changed:
if (reading != buttonState) {
buttonState = reading;
// only toggle the LED if the new button state is LOW
if (buttonState == LOW) {
digitalWrite(ledPin, HIGH);
pressCount++; // increment the press count
} else {
digitalWrite(ledPin, LOW);
}
}
}
// save the reading. Next time through the loop, it will be the lastButtonState:
lastButtonState = reading;
Serial.print(pressCount); // print the count to the serial monitor
Serial.print(",");
Serial.println(distance); // Print the distance to the Serial monitor
delay(100); // Short delay before next measurement
}
float getDistanceCm() {
// Trigger the measurement
digitalWrite(trigPin, LOW);
delayMicroseconds(2);
digitalWrite(trigPin, HIGH);
delayMicroseconds(10);
digitalWrite(trigPin, LOW);
// Calculate the distance based on the time of echo
float duration = pulseIn(echoPin, HIGH);
float distance = (duration * 0.0343) / 2;
return distance;
}
And this is the code from P5:
//Add this in index.html
// <!-- Load the web-serial library -->
// <script src="p5.web-serial.js"></script>
// Go download the web serial at https://github.com/Pi-31415/Intro-To-IM/blob/main/p5.web-serial.js
// Declare a variable to hold the smoothed value
let smoothedDistance = 0;
let gameMode = 0; // Variable to store the current game mode
var landscape; // Variable to store the landscape graphics
var car_diameter = 15; // Diameter of the ball
var bomb_diameter = 10; // Diameter of the bombs
var cardistancex;
var ypoint;
var zapperwidth = 6; // Width of the zapper
var numofbombs = 3; // Number of bombs
var bombposX = []; // Array to store X positions of bombs
var bombposY = []; // Array to store Y positions of bombs
var bombacceleration = []; // Array to store acceleration of each bomb
var bombvelocity = []; // Array to store velocity of each bomb
var time = 0; // Variable to track time, usage context not provided
var timeperiod = 0; // Variable to store a time period, usage not clear without further context
//var score = 0; // Variable to store the current score
var posX; // X position, usage context not provided
var inMainMenu = true; // Boolean to check if the game is in the main menu
//var prevScore = 0; // Variable to store the previous score
let font; // Variable to store font, usage context not provided
let introgif;
let gameovergif;
let gif3;
let survivedgif;
let countdownTimer = 10; // Countdown timer starting from 30 seconds
let serial; // Declare a serial port object
let latestData = "waiting for data"; // Latest data received
let gameovernow = false;
//CONNECTION
let clickCount = 0;
let previousClickCount = 0; // Store the previous click count
let distanceReal = 255;
let ignorefirstclick = false;
let gameovergifLarge;
function preload() {
introgif = createImg(
"https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExNTA4MG42MWlhdWV3Y2cyZ3U1cTFqZHhpbHp1amcweDhjYzhkcHBkYyZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/35cSlj5ELlzGSg0ZYM/giphy.gif"
);
introgif.hide();
survivedgif = createImg(
"https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExdm1yeWt3aGlteWZkcHB3czk3Ym81YWtrZTVtb29pMng2NW83bnF4bCZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/LY3JbLuhmjDVfCcCxh/giphy.gif"
);
survivedgif.hide();
gameovergif = createImg(
"https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExZHo1dmNrdzQ5NnYycWdvMjBqOGt1Zmg0MTdxZHQ4eHAyZGZrMDZtbCZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9cw/hp9wzCTbGeGfIkXE6a/giphy.gif"
);
gameovergif.hide();
gameovergifLarge = createImg("https://intro-to-im.vercel.app/afra/gameoer.gif");
gameovergifLarge.hide();
//Temp
gif3 = loadImage("https://intro-to-im.vercel.app/afra/bggif.gif");
font = loadFont("fonts/Inconsolata_Condensed-Light.ttf");
car = loadImage("car.png");
car2 = loadImage("car2.png");
}
// Cloud class starts
class Cloud {
constructor(x, y, speed, stepSpeed, scale) {
this.x = x;
this.y = y;
this.scale = scale; // Add scale property
this.speed = speed;
this.stepSpeed = stepSpeed;
this.step = 0;
this.facingRight = false; // Initially moving to the left
this.animationTimer = null;
}
move() {
if (this.facingRight) {
this.x += this.speed;
}
}
display() {
push();
if (!this.facingRight) {
scale(-this.scale, this.scale); // Apply scale with horizontal flip
image(oneDimensionarray[this.step], -this.x, this.y);
} else {
scale(this.scale, this.scale); // Apply scale
image(oneDimensionarray[this.step], this.x, this.y);
}
pop();
}
advanceStep() {
this.step = (this.step + 1) % 8;
}
startAnimation() {
this.facingRight = true;
clearInterval(this.animationTimer);
this.animationTimer = setInterval(() => this.advanceStep(), this.stepSpeed);
}
stopAnimation() {
this.facingRight = false;
clearInterval(this.animationTimer);
}
}
let clouds = [];
// Cloud class ends
// Define a maximum boundary for the distance
const maxDistance = 600; // Set this to whatever maximum value makes sense in your context
function mapDistance(distanceReal) {
// Define the smoothing factor (alpha). Smaller values make the motion smoother but less responsive.
const alpha = 0.2;
// Calculate the target position without smoothing
const targetPosition = (640 * (distanceReal - 3)) / 17;
// Apply exponential smoothing
smoothedDistance = alpha * targetPosition + (1 - alpha) * smoothedDistance;
// Ensure the smoothed distance does not exceed the maximum allowed distance
if (smoothedDistance > maxDistance) {
smoothedDistance = maxDistance;
}
return smoothedDistance;
}
function setup() {
createCanvas(640, 480);
textAlign(CENTER);
gif3.resize(640 * 2, 480 * 2);
var temp00 = 0,
temp01 = -20;
// A while loop that increments temp01 based on temp00 until temp01 is less than the canvas height
while (temp01 < height) {
temp00 += 0.02; // Increment temp00 by 0.02 in each loop iteration
temp01 += temp00; // Increment temp01 by the current value of temp00
timeperiod++; // Increment timeperiod in each iteration
}
// Calculate the initial position of posX based on zapperwidth and car_diameter
posX = zapperwidth + 0.5 * car_diameter - 2;
// Set cardistancex and ypoint relative to the width and height of the canvas
cardistancex = 0.7 * width; // Set cardistancex to 70% of the canvas width
ypoint = height - 0.5 * car_diameter + 1; // Set ypoint based on the canvas height and car_diameter
initbombpos(); // Call the initbombpos function (presumably initializes bomb positions)
imageMode(CENTER); // Set the image mode to CENTER for drawing images centered at coordinates
// Initialize variables for width and height based on
// Create 3 clouds with horizontal offsets, different speeds and scales
clouds.push(new Cloud(width / 8, height / 9, 0, 100, 0.9)); // First cloud
clouds.push(new Cloud((2 * width) / 5, height / 9, 0, 100, 1.2)); // Second cloud
clouds.push(new Cloud((2 * width) / 2, height / 9, 0, 200, 1.0)); // Third cloud
}
//Serial Read
function readSerial(data) {
////////////////////////////////////
//READ FROM ARDUINO HERE
////////////////////////////////////
if (data != null) {
// make sure there is actually a message
// split the message
let fromArduino = split(trim(data), ",");
// if the right length, then proceed
if (fromArduino.length == 2) {
// only store values here
// do everything with those values in the main draw loop
// We take the string we get from Arduino and explicitly
// convert it to a number by using int()
// e.g. "103" becomes 103
clickCount = int(fromArduino[0]);
distanceReal = parseFloat(fromArduino[1]);
}
}
}
function draw() {
clear();
//Establish Serial
if (!serialActive) {
} else {
text("Connected", 20, 30);
// Print the current values
console.log("clickCount = " + str(clickCount), 20, 50);
text("distanceReal = " + str(distanceReal), 20, 70);
}
// Check if clickCount has increased
if (clickCount > previousClickCount) {
if (ignorefirstclick) {
simulateMouseClick(); // Call your function that simulates a mouse click
}
ignorefirstclick = true;
}
previousClickCount = clickCount; // Update previousClickCount
// background(gif3);
// displayTimer();
// updateTimer();
if (gameMode == 0) {
//Main Menu
textFont(font);
textSize(50); // Larger text size for the game title
textAlign(CENTER, CENTER); // Align text to be centered
text("HAILSTORM HAVOC", width / 2, height / 2 - 40);
textSize(16); // Smaller text size for the directions
text(
"DIRECTIONS:\n click mouse to dodge hail\n the longer the press the further right\n the car will go\n\n AVOID the red line - crossing it means game over",
width / 2,
height / 2 + 50
);
textSize(20);
text("Click to start!", width / 2, height / 2 + 140);
introgif.show();
introgif.position(0, 0);
introgif.size(width, height);
} else if (gameMode == 1) {
//Actual game
// gif3.show();
image(gif3, 20, 20);
displayTimer();
updateTimer();
introgif.hide();
survivedgif.hide();
gameovergifLarge.hide();
// fill(239, 58, 38);
// rect(0, 0, zapperwidth, height);
//scoreUpdate();
fill(255);
noStroke();
for (var i = 0; i < numofbombs; i++) {
ellipse(bombposX[i], bombposY[i], bomb_diameter, bomb_diameter);
}
updatebombpos();
// ellipse(cardistancex, ypoint, car_diameter, car_diameter);
//Betwen 0 and 640
let cardistancex = mapDistance(distanceReal);
image(car, cardistancex, ypoint - 30, car_diameter * 5, car_diameter * 5);
if (cardistancex <= posX || bombCollistonTest()) {
//gameover(); // Call the gameover function if either condition is true
gameMode = 3;
}
time += 1;
// if (frameCount % 60 == 0) {
// score++; // Increase score by 1
// }
checkGameOver();
// gif3.show();
// gif3.position(0, 0);
// gif3.size(width, height);
} else if (gameMode == 2) {
//Survive
survivedgif.show();
survivedgif.position(0, 0);
survivedgif.size(width, height);
restartGame();
// displayWin();
} else if (gameMode ==3){
//GameOver
gameovergifLarge.show();
gameovergifLarge.position(0, 0);
gameovergifLarge.size(width, height);
restartGame();
}
}
function displayTimer() {
if (font) {
textFont(font); // Set the loaded font for displaying text
}
fill(255, 255, 0); // Set the text color to white for visibility
textSize(30); // Set the text size
textAlign(RIGHT, TOP); // Align text to the center top
textStyle(BOLD);
text("Time: " + countdownTimer, width - 10, 10); // Display the timer on the canvas
}
function updateTimer() {
if (frameCount % 60 == 0 && countdownTimer > 0) {
countdownTimer--; // Decrease timer by 1 each second
}
}
function updatebombpos() {
// Iterate over each bomb
for (var i = 0; i < numofbombs; i++) {
bombvelocity[i] += bombacceleration[i]; // Update the velocity of the bomb by adding its acceleration
bombposY[i] += bombvelocity[i]; // Update the Y position of the bomb based on its velocity
}
if (time > timeperiod) {
initbombpos(); // Reinitialize the positions of the bombs by calling the initbombpos function
time = 0;
}
}
function initbombpos() {
for (var i = 0; i < numofbombs; i++) {
bombacceleration[i] = random(0.02, 0.03); // Randomize the acceleration
bombvelocity[i] = random(0, 5); // Randomize the initial velocity
bombposX[i] = random(zapperwidth + 0.5 * car_diameter, width); // Randomize the X position within playable area
bombposY[i] = -bomb_diameter; // Start bombs just above the top of the canvas
}
}
function bombCollistonTest() {
// Define the car's bounding box
let carLeft = cardistancex - car_diameter * 2.5;
let carRight = cardistancex + car_diameter * 2.5;
let carTop = ypoint - 20 - car_diameter * 2.5;
let carBottom = ypoint - 20 + car_diameter * 2.5;
// Iterate over each bomb to check for a collision
for (var i = 0; i < numofbombs; i++) {
// Check if bomb is within the bounding box of the car
if (
bombposX[i] >= carLeft &&
bombposX[i] <= carRight &&
bombposY[i] >= carTop &&
bombposY[i] <= carBottom
) {
return true; // Collision detected
}
}
return false; // No collision
} //This function checks for collisions between the player and each bomb by comparing the distance between them to a threshold. If any bomb is too close (within the threshold), it returns true (collision detected). Otherwise, it returns false.
function gameover() {
gameovernow = true;
let cardistancex = mapDistance(distanceReal);
image(car2, cardistancex, ypoint - 30, car_diameter * 5, car_diameter * 5);
gameovergif.show();
gameovergif.position(0, 0);
gameovergif.size(width, height);
}
function keyPressed() {
if (key == "a") {
// important to have in order to start the serial con nection!!
setUpSerial();
}
}
function simulateMouseClick() {
console.log("Mouse clicked via Arduino"); // Log or perform actions here
// You can call any functions here that you would have called in mouseClicked()
if (gameMode == 0 || gameMode == 2 || gameMode == 3) {
gameMode = 1;
}
//just flipping between modes 0 and 1
clouds.forEach((cloud) => cloud.startAnimation());
}
function mousePressed() {
//No mouse press
}
function mouseReleased() {
clouds.forEach((cloud) => cloud.stopAnimation());
}
function checkGameOver() {
if (countdownTimer <= 0) {
gameMode = 2;
}
}
function restartGame() {
// Reset all game variables to their initial values
gameovernow = false;
gameovergif.hide();
time = 0;
//score = 0;
countdownTimer = 5;
posX = zapperwidth + 0.5 * car_diameter - 2;
cardistancex = 0.5 * width;
ypoint = height - 0.5 * car_diameter + 1;
initbombpos();
// Restart the game loop
loop();
}
//This function resets the game environment and variables to their initial state, essentially restarting the game. It resumes background music, pauses any game over , resets score and time, repositions the player and bombs, and restarts the game loop.
Overall, I’m very proud of my project and I was very happy to see users play my game in the exhibition.
The main concept of “I See Sound” is to create an immersive experience with users and music by visualizing their favorite songs. By using 2 sensors, users are able to switch shapes and colors giving them a stylistic hand in the visualization. The aim is to put I See Sound in settings where upbeat music is played, for example in concerts, musical installations, parties, etc.
Arduino and p5.js scripts work together to create a dynamic audio-visual experience that responds to audio inputs via a photocell and a button. The Arduino script continuously monitors the environment through a photocell and a button. The photocell measures the light level and this data is read and printed. This value is then printed and sent to P5.js via serial communication. Additionally, the Arduino script checks for the state of a button. When pressed, it sends a specific command (“changeShape”) to the p5.js script. The combination of these sensors act as the main communication medium between users and the visualization.
The Arduino sends data values that include light levels and control commands, which the p5.js script reads and implements in different functions. This script is designed to respond to these inputs by altering visual and audio outputs accordingly. For example, higher light levels can result in brighter visuals, while pressing the button changes the visual form, demonstrating a real-time interactionbetween the user’s physical environment and the digital representation.
Arduino Code:
int photocellPin = 0;
int photocellReading;
int buttonPin = 2;
int buttonState = 0;
void setup() {
Serial.begin(9600);
pinMode(buttonPin, INPUT);
}
void loop() {
photocellReading = analogRead(photocellPin);
photocellReading = 1023 - photocellReading;
Serial.println(photocellReading);
buttonState = digitalRead(buttonPin);
if (buttonState == HIGH) {
Serial.println("changeShape");
delay(200);
}
delay(100);
}
p5.js Code:
let dj =0;
let sound, amplitude;
let currentShape = 'ellipse';
let currentState = 'startScreen';
let photocellData = 0;
function preload() {
sound = loadSound('sounds/aroundme.mp3');
startScreenImage = loadImage('P5 DJ.gif');
}
function setup() {
let cnv = createCanvas(700, 600);
amplitude = new p5.Amplitude();
noiseSeed(millis());
}
function draw() {
if (currentState == 'startScreen') {
displayStartScreen();
} else if (currentState == 'running') {
runVisualization();
}
}
function displayStartScreen() {
background(startScreenImage);
}
function runVisualization(){
let level = amplitude.getLevel();
photocellBackground();
let numShapes = int(map(level, 0, 5, 15, 30));
for (let i = 0; i < numShapes; i++) {
let angleOffset = TWO_PI / numShapes * i;
let x = width / 2 + 4 * (sin(frameCount * 0.02 + angleOffset) * 100 * noise(0.001 * frameCount + i));
let y = height / 2 + 4 * (cos(frameCount * 0.02 + angleOffset) * 100 * noise(0.001 * frameCount + 100 + i));
let size1 = map(sin(frameCount * 0.1 + angleOffset), -1, 1, 10, 100);
let myColor = color(255 * noise(i), 255 * noise(i + 10), 255 * noise(i + 20), 200);
fill(myColor);
let colors = ['red', 'blue', 'green', 'purple', 'maroon'];
let chosenColor = random(colors)
stroke(chosenColor);
strokeWeight(map(level, 0, 1, 10, 100));
switch (currentShape) {
case 'ellipse':
ellipse(x, y, size1, size1);
break;
case 'rectangle':
rect(x, y, size1, size1);
break;
case 'triangle':
triangle(x - size1 * 0.4, y + size1 * 0.4, x, y - size1 * 0.4, x + size1, y + size1 * 0.4);
break;
case 'star':
drawStar(x, y, 5, size1 * 0.8, size1 * 0.4);
break;
case 'spiral':
drawSpiral(x, y, size1 * 0.8);
break;
}
}
}
function photocellBackground() {
background(map(photocellData, 0, 1023, 0, 255));
}
function togglePlay() {
if (sound.isPlaying()) {
sound.pause();
} else {
sound.loop();
amplitude.setInput(sound);
}
}
function changeShape() {
const shapes = ['ellipse', 'rectangle', 'triangle', 'star', 'spiral'];
let index = shapes.indexOf(currentShape);
currentShape = shapes[(index + 1) % shapes.length];
}
function drawStar(x, y, points, radius1, radius2) {
let angle = TWO_PI / points;
let halfAngle = angle / 2.0;
beginShape();
for (let a = 0; a < TWO_PI; a += angle) {
let sx = x + cos(a) * radius2;
let sy = y + sin(a) * radius2;
vertex(sx, sy);
sx = x + cos(a + halfAngle) * radius1;
sy = y + sin(a + halfAngle) * radius1;
vertex(sx, sy);
}
endShape(CLOSE);
}
function drawSpiral(x, y, maxRadius) {
let angle = 0;
let endRadius = 0;
beginShape();
while (endRadius < maxRadius) {
let sx = x + cos(angle) * endRadius;
let sy = y + sin(angle) * endRadius;
vertex(sx, sy);
angle += 0.1;
endRadius += 0.5;
}
endShape();
}
function keyPressed() {
if (key == " " && currentState == 'startScreen') {
setUpSerial();
waitForSerial();
}
}
function waitForSerial() {
if (serialActive) {
currentState = 'running';
togglePlay();
} else {
console.log("Waiting for serial connection.. Press Space to Connect.");
setTimeout(waitForSerial, 10);
}
}
////////////////////////////////////
//READ FROM ARDUINO HERE
////////////////////////////////////
function readSerial(data){
if (data != null) {
let fromArduino = data;
if (fromArduino.length >= 1) {
dj = int(data);
print(dj)
// Echo = int(fromArduino[1]);
console.log(data); // Print data for debugging
if (data.trim() === "changeShape") {
changeShape(); // Change the shape if the correct command is received
}
if (data.trim() === "photocellReading")
photocellBackground();
}
// //////////////////////////////////
// //SEND TO ARDUINO HERE (handshake)
// //////////////////////////////////
let sendToArduino = -1;
writeSerial(sendToArduino);
}
}
Overall, I am particularly proud of myself because I really did try my best, as a beginner I found it hard to even wrap my head around most of the stuff and I am honestly very proud of myself. During the process, I didn’t feel as confident at some point however I seeked out different resources such as Berney Codes, asked some of my classmates for help (Thank You Sarah), and used ChatGPT to help me understand serial communication and revise errors in codes, as well as in creating shapes. In the future, however, I would definitely work on the interaction part more, as well as stylistic choices. I did adapt this project from the p5.Amplitude library.
During user testing, while understanding the visualization was easy, users found it hard to figure out how to interact with the visualization.
The point of error was that there was no clear indication of what to do, a major design aspect I’ve overlooked. Eventually, however, the users understand that in the arduino circuit, the photocell is able to switch the colors of the beat in accordance with the light. And, to some I had to explain exactly what the sensor did since it isn’t clear when you first try it.
The p5.Amplitude library enabled the visualization consistency with any audio input, my friends who have tested this asked to add their own music and the code worked very well 99% of the time, the remaining 1% is that this only works well with upbeat music with heavy drums.
There is definitely more area for improvement in a number of things. It would be nice if the interactivity experience was elevated to a couple of sensors which would make the experience more inclusive to the users. Also another aspect would be the color palette for the overall look, while the rainbow is definitely beautiful a color palette that is consistent would be calmer and less disturbing for the eye. Additionally, I am still in the process of designing a cover page rather than the instructions in the top left corners.
For my final project, I delved into graphic design, culminating in the creation of a cooking game called “Mansaf Teta.” This game is a symbol of Palestinian culture, centered around the preparation of a traditional dish. Named after the colloquial term for “Grandma’s Mansaf,
” it aims to encapsulate the essence of Palestinian culinary heritage.
The gameplay involves a series of interactive cooking actions, each tied
to a physical component controlled by an Arduino board. The first step is cooking the lamb, where players gauge the ideal temperature using a potentiometer. Next, stirring the “laban” (fermented dried yogurt) is simulated using one button to create a stirring motion. Finally, the addition of garnish is triggered by another button, offering players a holistic cooking experience.
My favorite part of the code involves programming the potentiometer and working with images to create a visualization of the stove, shown below:
if (typeof val1 !== 'undefined') {
//console.log("val1:", val1);
// Respond to temperature
if (val1 < 500) {
image(toocold, 0, 0, w, h);
print("checking here: "+valInRangeTime)
valInRangeTime += deltaTime; // Add the time elapsed since the last frame to the timer
if (valInRangeTime >= 2000) { // Check if val1 remains in range for more than 3 seconds
print("going to next image")
background(next1); // Display the image
if (mouseIsPressed && mouseIsPressed && mouseX > w / 4 && mouseY > h / 2) {
gameState = 3; //go to first part
}
}
} else {
// If val1 goes out of range, reset the timer
valInRangeTime = 0;
}
}
}
Throughout the game-making process, I encountered challenges, especially with establishing a stable serial connection and some game logic. However, with help and guidance from professor and problem-solving skills, I managed to overcome these obstacles and bring my thoughts into a final product.
Looking ahead, I would add more to this project. This includes expanding the variety of sensors on the Arduino board to introduce more nuanced interactions and incorporating additional audio effects to heighten immersion.
Despite these challenges, I take pride in the design and color palette of the game, which I believe enhances the overall experience for players, immersing them in the rich tapestry of Palestinian culinary culture.
It was easy to come up with an overall abstract direction for the final project as I had set my mind early on to revisit the butterfly motif in different ways throughout the course. As for the actual concept behind the final, I wanted to experiment with something more centered around interaction and visuals in an installation-like manner and explore a new mode of design that I have not tapped into yet. After iterations and iterations of deliberations and conversations with Professor Aaron, I settled on creating a small piece centered around a mechanical butterfly that flutters when touched. The butterfly would be mounted atop a physical canvas, onto which p5-generated animations would be mapped and projected. The idea was to create a cohesive piece, with the hardware and the software working hand-in-hand to bring some life to a butterfly.
The mechanical butterfly is constructed out of two servo motors, with one moving at an angle supplementary to that of the other. The butterfly wings are printed on paper, laminated, cut, and attached to the servo motor blades. The butterfly “senses” touch through its antennas. My mechanical butterfly’s antennas are made of wires stripped, twisted to shape, and connected to a touch capacitive sensor. I used a box, which I wrapped with multiple layers of white paper and decorated with flowers (to look like the butterfly is in a flower field), with an opening for the Arduino and the circuit.
Interaction Design
For this piece, I wanted to emphasize designing a relatively simple interaction optimally well. The name I chose for the piece, “Pet-A-Butterfly” would be displayed to the user and would act as a signifier to touch the butterfly. The placement of the butterfly antennas opposite the user is intentional to maximize the probability that a user strokes the wires in the chance that they do not realize the antennas are to be touched. The user can interact with the piece by touching the butterfly antennas. Once touched, the butterfly wings flap, and a kaleidoscope of small p5-generated/projected butterflies emerge from beneath the butterfly and move outward in a synergistic, spiral motion.
Implementation
Arduino
The Arduino program gets the input from the sensor through the touched()method, which returns an 8-bit value representing the touch state of all pins, and sends it to the p5 sketch through serial communication. The program also gets the current status of the butterfly movement from the p5 sketch program. If the status is 1 (the butterfly is moving), the servo motor positions are updated every interval seconds. The angles of the motors are constrained to the range [25,50] and the direction of each motor’s movement alternates after each range span to achieve the flapping movement. The Arduino program also sends the current servo position to the p5 sketch to ensure the sketch only stops the butterfly animation if the servos are in the maximum angle position, ensuring the flapping stops when the wings are maximally spread.
Below is the full Arduino sketch:
/***************************************************
This is a library for the CAP1188 I2C/SPI 8-chan Capacitive Sensor
Designed specifically to work with the CAP1188 sensor from Adafruit
----> https://www.adafruit.com/products/1602
These sensors use I2C/SPI to communicate, 2+ pins are required to
interface
Adafruit invests time and resources providing this open source code,
please support Adafruit and open-source hardware by purchasing
products from Adafruit!
Written by Limor Fried/Ladyada for Adafruit Industries.
BSD license, all text above must be included in any redistribution
****************************************************/
#include <Wire.h>
#include <SPI.h>
#include <Adafruit_CAP1188.h>
#include <Servo.h>
// Reset Pin is used for I2C or SPI
#define CAP1188_RESET 9
// CS pin is used for software or hardware SPI
#define CAP1188_CS 10
// These are defined for software SPI, for hardware SPI, check your
// board's SPI pins in the Arduino documentation
#define CAP1188_MOSI 11
#define CAP1188_MISO 12
#define CAP1188_CLK 13
#define CAP1188_SENSITIVITY 0x1F
// For I2C, connect SDA to your Arduino's SDA pin, SCL to SCL pin
// On UNO/Duemilanove/etc, SDA == Analog 4, SCL == Analog 5
// On Leonardo/Micro, SDA == Digital 2, SCL == Digital 3
// On Mega/ADK/Due, SDA == Digital 20, SCL == Digital 21
// Use I2C, no reset pin!
Adafruit_CAP1188 cap = Adafruit_CAP1188();
// Or...Use I2C, with reset pin
//Adafruit_CAP1188 cap = Adafruit_CAP1188(CAP1188_RESET);
// Or... Hardware SPI, CS pin & reset pin
// Adafruit_CAP1188 cap = Adafruit_CAP1188(CAP1188_CS, CAP1188_RESET);
// Or.. Software SPI: clock, miso, mosi, cs, reset
//Adafruit_CAP1188 cap = Adafruit_CAP1188(CAP1188_CLK, CAP1188_MISO, CAP1188_MOSI, CAP1188_CS, CAP1188_RESET);
// make a servo object
Servo servoRight;
Servo servoLeft;
// servo pposition
int position=50;
// direction of wing movement
boolean direction = true;
unsigned long previousMillis = 0;
const long interval = 100; // interval between each wing flap in milliseconds
void setup() {
Serial.begin(9600);
Serial.println("CAP1188 test!");
// Initialize the sensor, if using i2c you can pass in the i2c address
if (!cap.begin(0x28)) {
if (!cap.begin()) {
while (1);
}
cap.writeRegister(CAP1188_SENSITIVITY, 0x5F);
// attach the servo to pin 9
servoRight.attach(11);
servoLeft.attach(5);
// write the position
servoRight.write(180- position);
servoLeft.write(position);
// // start the handshake
while (Serial.available() <= 0) {
digitalWrite(LED_BUILTIN, HIGH); // on/blink while waiting for serial data
Serial.println("0"); // send a starting message
delay(300); // wait 1/3 second
digitalWrite(LED_BUILTIN, LOW);
delay(50);
}
}
}
void loop() {
// wait for data from p5 before doing something
while (Serial.available()) {
uint8_t touched = cap.touched();
int isMoving = Serial.parseInt(); // check if butterfly is still moving
Serial.print(touched);
Serial.print(',');
if (isMoving == 1) {
unsigned long currentMillis = millis();
// check if it's time to update the wing position
if (currentMillis - previousMillis >= interval) {
// move servos to simulate wing flapping motion
if (direction) {
position += 10;
if (position >= 50) { // flip direction twhen max angle is reached
direction = false;
}
} else {
position -= 10;
if (position <= 25) {
direction = true;
}
}
// move servos in opposite directions
servoRight.write(180-position);
servoLeft.write(position);
previousMillis = currentMillis;
}
};
Serial.println(position); // send servc position to p5 sketch
}
digitalWrite(LED_BUILTIN, LOW);
}
P5
The p5 sketch is mainly responsible for triggering the animation of the smaller butterflies and for performing projection mapping which is essential for ensuring that the canvas of the sketch can always be calibrated to fit the surface of the physical box. For the latter, I made use of the p5.mapper library to create a quad map that could be calibrated to match the aspect ratios of the box’s surface dynamically. By pressing the ‘c’ key, the map’s points can be toggled and moved appropriately. This eliminated the challenge of having to align the projector height consistently across locations and manually configuring the sketch’s canvas dimensions to match the surface. After calibrating the map, the p5 program can save the map in a json file to be loaded with every program run by pressing the ‘s’ key. This code snippet of the setup()function shows how to initialize a map object and load an existing map configuration.
function setup() {
createCanvas(windowWidth, windowHeight, WEBGL);
// create mapper object
pMapper = createProjectionMapper(this);
quadMap = pMapper.createQuadMap(mapWidth, mapHeight);
// loads calibration in the "maps" directory
pMapper.load("maps/map.json");
// initialize objects
bigButterfly = new Butterfly(
centerX,
centerY,
null,
null,
null,
null,
null,
false,
false,
null,
null,
false
); // dummy butterfly object simulating the state of the physical butterfly
interaction = new Interaction(); // an interaction object that handles all interaction-related animations
// play background music in loop
backgroundMusic.loop();
}
To implement the animation, I created an Interaction class that would start and display the animation of the butterflies in a method called play(). This method would be the argument to a function of the pMapper object called displaySketch that would handle displaying the sketch only within the map’s bounds.
// class that controls the animation trigger by the interaction
class Interaction {
constructor() {
this.bigButterfly = bigButterfly; // the butterfly object containing information about the physical butterfly in the center
this.smallButterflies = []; // array that stores the smaller butterflies whose animation is triggered and displayed when signal is received from arduion
this.numButterflies = 100; // number of small butterflies
this.inTheCenter = this.numButterflies; // number of butterflies in the center
// initialize randomly colored butterfly objects and append to the smallButterflies array
let randomNum;
for (let i = 0; i < this.numButterflies; i++) {
randomNum = random([1, 2, 3]);
if (randomNum == 1) {
this.smallButterflies.push(
new SmallButterfly(
centerX,
centerY,
smallButterflySpritesheet2,
4,
10,
0,
3,
true,
false,
null,
null,
false
)
);
}
else if (randomNum == 2){
this.smallButterflies.push(
new SmallButterfly(
centerX,
centerY,
smallButterflySpritesheet1,
4,
10,
0,
5,
true,
false,
null,
null,
false
)
);
}
else if (randomNum == 3){
this.smallButterflies.push(
new SmallButterfly(
centerX,
centerY,
smallButterflySpritesheet3,
4,
10,
0,
13,
true,
false,
null,
null,
false
)
);
}
}
}
play(pg) {
/* function that controls that controls the sketch
display -> passed to mappper object's displaySketch function
*/
pg.clear();
pg.push();
pg.background(color("#B2D2A2"));
// display instructions text only before connecting to serial
if (textShow){
pg.push()
pg.fill(color("#2c4c3b"))
pg.textFont(font);
pg.textAlign(CENTER);
pg.textSize(16)
pg.text(textString, centerX+20, centerY+150);
pg.pop()
}
// display butterflies
for (let i = 0; i < interaction.numButterflies; i++) {
pg.push();
let angle = radians(180);
pg.translate(
interaction.smallButterflies[i].x,
interaction.smallButterflies[i].y
);
pg.rotate(angle); // rotate butterflies 180 degrees --> better visibility for the user
if (interaction.smallButterflies[i].moving) { // display the small butterfly if it's moving
pg.image(interaction.smallButterflies[i].show(), 0, 0, 40, 40);
interaction.smallButterflies[i].move(); // update movement of butterflies
}
pg.pop();
}
pg.push();
// ellipse enclosing projected surface area of the physical butterfly
pg.fill(color("#B2D2A4"));
// pg.fill(color("black"))
pg.noStroke();
// pg.ellipse(215, 180, butterflyWidth, butterflyHeight)
pg.pop();
// stop butterfly from moving after a set time has elapsed and only if the
// position of the servo is in the right direction
if (millis() - movementTime >= interval && servoPos == 50) {
bigButterfly.moving = false;
}
}
}
The movement of the butterflies follows a spiral-like path, originating outward and around the physical butterfly. It is implemented in a method of thesmallButterflyclass which inherits from a parent Butterflyclass. Here is a code snippet showing the implementation of the path movement in the smallButterflyclass :
move() {
// update the step of the animation
if (frameCount % this.animationSpeed == 0) {
this.step = (this.step + this.animationDir * 1) % this.numSpritesCol;
}
// control the direction of the sprite movement as spritesheet must be traversed back and forth to display correct movement
if (this.step == 0) {
this.animationDir = 1;
} else if (this.step == this.numSpritesCol - 1) {
this.animationDir = -1;
}
// update the x and y positions based on the current angle and radius
this.x = centerX + cos(this.angle)* this.radius + random(-0.5,0.5);
this.y = centerY + sin(this.angle)* this.radius + random(-0.5,0.5);
this.angle += this.angleSpeed; // increment angle to move the butterfly along a circular path
this.radius += this.radiusSpeed; // increment the radius to move the butterfly outward
// move back to center if butterfly exceeds the bounds
if (this.x < minX || this.y < minY || this.x > maxX || this.y > maxY) {
this.x = centerX;
this.y = centerY;
interaction.inTheCenter += 1; // butterfly is now counted as being in the center
this.moving = false; // stop butterfly from moving
// update angle and radius speed parameters to random values
this.angleSpeed = random(-0.02, 0.02);
this.radiusSpeed = random(0.5,1.2);
this.angle = 0;
this.radius = 0;
}
// flip butterfly direction depending on location in the sketch
if (this.x < centerX && this.sprites.length > 1) {
this.dir = 1;
} else {
this.dir = 0;
}
}
When the p5 sketch receives the touch state and servo position from Arduino, it sets the moving attribute of both the butterfly object simulating the physical butterfly in the sketch and the small butterflies to true. It also starts the timer, as the physical butterfly should only stop moving after 6 seconds have elapsed and if the servos are in the right position:
function readSerial(data) {
////////////////////////////////////
//READ FROM ARDUINO HERE
////////////////////////////////////
if (data != null) {
// make sure there is actually a message
let fromArduino = data;
// if the right length, then proceed
if (fromArduino.length > 0) {
// get value only when data sent from arduino is greater than 0
fromArduino = split(trim(fromArduino), ",");
touchSensorVal = int(fromArduino[0]); // get touch sensor val
servoPos = int(fromArduino[1]); // get servo pos
if (touchSensorVal >= 1) { // if sensor is touched, set the bigButterfly moving attribut to true
interaction.bigButterfly.moving = true;
movementTime = millis(); // record starting movement time
interaction.inTheCenter = 0;
// move smaller butterflies
for (let i = 0; i < interaction.numButterflies; i++) {
interaction.smallButterflies[i].moving = true;
}
}
}
//////////////////////////////////
//SEND TO ARDUINO HERE (handshake)
//////////////////////////////////
let sendToArduino;
if (interaction.bigButterfly.moving == true) {
sendToArduino = 1 + "\n"; // send 1 to Arduino if the butterfly is moving
} else {
sendToArduino = 0 + "\n"; // send 0 to Arduino if the butterfly is done with its animation
}
writeSerial(sendToArduino);
}
}
Here is an embedding of the full sketch (you can press the ‘d’ key to play the animation without the signal from Arduino):
Reflections and Parts I am Proud of
My biggest concern going into this, especially as I was going to employ projection mapping, was that I would be unable to align the p5 sketch and the physical butterfly together in a cohesive manner that still looks visually pleasing. I am, thus, proud that the final product resembles what I had envisioned. I also spent a lot of time thinking of the proper mechanism to automate the wing flapping motion and where/how to place the wings. I experimented with a lot of methods, such as attaching a vertical straw/wooden stick from the middle of the wings to the servo blades, and tugging on the wings when moving down to move the wings up and down. When that proved to be unhelpful, I switched to simply attaching each wing to a blade, which, in hindsight, should have been what I experimented with first. I also love the detail of having the connection between the butterfly and the sensor be through antenna-looking sensors, resembling the sense mechanisms of an actual butterfly (thanks to Professor Aaron for pointing this out). Finally, I am proud that I managed to properly calibrate the sensitivity of the touch sensor, as it initially was too sensitive, sometimes even detecting signals even when untouched. Keeping the sensitivity in check was a major challenge that I thankfully was able to overcome to keep the interaction consistent.
Areas for Future Improvements
I think the project could definitely be enhanced in a lot of ways. Because I spent a lot of time putting the interface together, an area of future improvement could be the p5-generated animation itself. I could have different path movements triggered with every touch, for example. I had initially wanted to map an actual animated butterfly from p5 onto a blank silhouette cutout of a butterfly, controlled by the servos in the same way. Because of difficulties in mapping the software animations to the movement of the hardware, I decided to pivot toward having the central butterfly be completely in hardware form. One improvement to explore is going in that direction, where I effectively add physical objects, like flowers, on the surface of the box and map simpler, more limited animations onto them.
