Honestly, at this point, I’m still not sure what exactly I want to do for my final project. I know I want to create an immersive and interactive game-like experience, and I have some broad ideas in my mind.
Inspired by our first Arduino assignment (the creative switch) and one of our class readings which talked about body-as-cursor interactions, I know I want my project to be a piece that uses something besides the hands to be played. Two things that immediately come to mind are story-telling through interaction in the real world, such as using the ultrasonic sensor to determine distance and having a p5.js program that responds to it. It could either be a fun dance game where different types of sounds play, or, a story-telling experience where different elements of the story pop up as the user appears closer.
I enjoyed this week’s reading because accessibility is a point that is always brought up when we speak of design and interactivity, whether it was last week’s readings about pictures-under-glass or a broader discussion about the simplicity of design. In this day and age, technological advancements are designed to be so simple that even a toddler could use them. I think instead of simplicity, the foremost thought should be accessibility. In modern design, accessibility is always an afterthought, a sort of ‘add-on’ to an existing design, that often looks unwieldy and takes away from the appearance of the original design. Another reason I strongly believe in everything being accessible instead of simply having accessible alternatives is that I have seen many disabled voices online speak of how having to use ‘alternatives’ is one of the many ways they feel other-ed by society. If everyone, abled or disabled, used the same interfaces and designs, we would be one step closer to a truly inclusive community.
I also love the discussion about the intersection of fashion and accessibility. I, personally, am unsure where I stand on this – I agree in the sense that good design is simple, and that simplicity goes hand-in-hand with universality. However, as someone who is fond of all things camp, a part of me doesn’t agree that all design should direct towards being simple. In fact, I believe that d in design can still be for disability, whilst still exploring aesthetics to their fullest potential.
For this exercise, I just repurposed the code from class. I kept the Arduino IDE code the same. I used the alpha value (which was the reading from the photoresistor) and added these three lines of code to move the ellipse along the horizontal axis.
let x = map(alpha, 0, 1023, 0, width);
fill(0, 0, 0);
ellipse(x, height/2, 20, 20);
2:
For this, I modified the Arduino code slightly.
// Week 11.2 Example of bidirectional serial communication
// Inputs:
// - A0 - sensor connected as voltage divider (e.g. potentiometer or light sensor)
// - A1 - sensor connect
pinMode(LED_BUILTIN, OUTPUT);
// Outputs on these pins
pinMode(leftLedPin, OUTPUT);
pinMode(rightLedPin, OUTPUT);
// Blink them so we can check the wiring
digitalWrite(leftLedPin, HIGH);
digitalWrite(rightLedPin, HIGH);
delay(200);
digitalWrite(leftLedPin, LOW);
digitalWrite(rightLedPin, LOW);
// start the handshake
while (Serial.available() <= 0) {
// digitalWrite(LED_BUILTIN, HIGH); // on/blink while waiting for serial data
Serial.println("0,0"); // send a starting message
delay(300); // wait 1/3 second
// digitalWrite(LED_BUILTIN, LOW);
delay(50);
}
}
void loop() {
// wait for data from p5 before doing something
while (Serial.available()) {
digitalWrite(LED_BUILTIN, HIGH); // led on while receiving data
int left = Serial.parseInt();
int right = Serial.parseInt();
if (Serial.read() == '\n') {
brightness = map(left, 0, 1023, 255, 0);
analogWrite(rightLedPin, brightness);
int sensor = analogRead(A0);
delay(5); // delay bc consecutive analog reads might make some noise
int sensor2 = analogRead(A1);
delay(5);
Serial.print(sensor);
Serial.print(',');
Serial.println(sensor2);
}
}
digitalWrite(LED_BUILTIN, LOW);
}
And here is the p5.js code:
let rVal = 0;
let alpha = 255;
let left = 0; // True (1) if mouse is being clicked on left side of screen
let right = 0; // True (1) if mouse is being clicked on right side of screen
let slider;
function setup() {
createCanvas(640, 480);
textSize(18);
slider = createSlider(0, 255, 0);
}
function draw() {
// one value from Arduino controls the background's red color
background(map(rVal, 0, 1023, 0, 255), 255, 255);
// the other value controls the text's transparency value
fill(255, 0, 255, map(alpha, 0, 1023, 0, 255));
if (!serialActive) {
text("Press Space Bar to select Serial Port", 20, 30);
} else {
text("Connected", 20, 30);
// Print the current values
text('rVal = ' + str(rVal), 20, 50);
text('alpha = ' + str(alpha), 20, 70);
text('brightness = ' + str(slider.value()), 20, 90)
}
}
function keyPressed() {
if (key == " ") {
// important to have in order to start the serial connection!!
setUpSerial();
}
}
// This function will be called by the web-serial library
// with each new *line* of data. The serial library reads
// the data until the newline and then gives it to us through
// this callback function
function readSerial(data) {
////////////////////////////////////
//READ FROM ARDUINO HERE
////////////////////////////////////
if (data != null) {
// make sure there is actually a message
// split the message
let fromArduino = split(trim(data), ",");
// if the right length, then proceed
if (fromArduino.length == 2) {
// only store values here
// do everything with those values in the main draw loop
// We take the string we get from Arduino and explicitly
// convert it to a number by using int()
// e.g. "103" becomes 103
rVal = int(fromArduino[0]);
alpha = int(fromArduino[1]);
}
//////////////////////////////////
//SEND TO ARDUINO HERE (handshake)
//////////////////////////////////
left = slider.value();
right = slider.value();
let sendToArduino = left + "," + right + "\n";
writeSerial(sendToArduino);
}
}
I created a slide that when the user moves back and forth, they can adjust the brightness of the blue LED (I used the blue LED because I used the exact same schematic, and the blue LED was connected to the digital PWM).
3:
The Arduino code I used for the third exercise was also very similar to the initial one provided:
// Week 11.2 Example of bidirectional serial communication
// Inputs:
// - A0 - sensor connected as voltage divider (e.g. potentiometer or light sensor)
// - A1 - sensor connected as voltage divider
//
// Outputs:
// - 2 - LED
// - 5 - LED
int leftLedPin = 2;
int rightLedPin = 5;
int brightness = 0;
void setup() {
// Start serial communication so we can send data
// over the USB connection to our p5js sketch
Serial.begin(9600);
// We'll use the builtin LED as a status output.
// We can't use the serial monitor since the serial connection is
// used to communicate to p5js and only one application on the computer
// can use a serial port at once.
pinMode(LED_BUILTIN, OUTPUT);
// Outputs on these pins
pinMode(leftLedPin, OUTPUT);
pinMode(rightLedPin, OUTPUT);
// Blink them so we can check the wiring
digitalWrite(leftLedPin, HIGH);
digitalWrite(rightLedPin, HIGH);
delay(200);
digitalWrite(leftLedPin, LOW);
digitalWrite(rightLedPin, LOW);
// start the handshake
while (Serial.available() <= 0) {
// digitalWrite(LED_BUILTIN, HIGH); // on/blink while waiting for serial data
Serial.println("0,0"); // send a starting message
delay(300); // wait 1/3 second
// digitalWrite(LED_BUILTIN, LOW);
delay(50);
}
}
void loop() {
// wait for data from p5 before doing something
while (Serial.available()) {
digitalWrite(LED_BUILTIN, HIGH); // led on while receiving data
int left = Serial.parseInt();
int right = Serial.parseInt();
if (Serial.read() == '\n') {
// brightness = map(left, 0, 1023, 255, 0);
digitalWrite(leftLedPin, left);
digitalWrite(rightLedPin, right);
// if (right == )
int sensor = analogRead(A0);
delay(5); // delay bc consecutive analog reads might make some noise
int sensor2 = analogRead(A1);
delay(5);
Serial.print(sensor);
Serial.print(',');
Serial.println(sensor2);
}
}
digitalWrite(LED_BUILTIN, LOW);
}
and this is the p5.js code:
let velocity;
let gravity;
let position;
let acceleration;
let wind;
let drag = 0.99;
let mass = 50;
let ledVal = 0;
function setup() {
createCanvas(640, 360);
noFill();
position = createVector(width / 2, 0);
velocity = createVector(0, 0);
acceleration = createVector(0, 0);
gravity = createVector(0, 0.5 * mass);
wind = createVector(0, 0);
}
function draw() {
background(255);
if (!serialActive) {
print("click to select Serial Port");
fill(0);
text("click to select Serial Port", 20, 30);
} else {
applyForce(wind);
applyForce(gravity);
velocity.add(acceleration);
velocity.mult(drag);
position.add(velocity);
acceleration.mult(0);
ledVal = 0;
ellipse(position.x, position.y, mass, mass);
if (position.y > height - mass / 2) {
velocity.y *= -0.9; // A little dampening when hitting the bottom
position.y = height - mass / 2;
ledVal = 1;
}
}
}
function applyForce(force) {
// Newton's 2nd law: F = M * A
// or A = F / M
let f = p5.Vector.div(force, mass);
acceleration.add(f);
}
function keyPressed() {
if (key == " ") {
mass = random(15, 80);
position.y = -mass;
velocity.mult(0);
}
}
function mousePressed() {
if (!serialActive) {
// important to have in order to start the serial connection!!
setUpSerial();
}
}
function readSerial(data) {
if (data != null) {
let fromArduino = split(trim(data), ",");
// if the right length, then proceed
if (fromArduino.length == 2) {
let val = int(fromArduino[1]);
wind.x = map(val, 0, 1023, -1, 1);
print("wind", wind.x);
}
//////////////////////////////////
//SEND TO ARDUINO HERE (handshake)
//////////////////////////////////
// left = slider.value();
// right = slider.value();
// console.log(position.y);
let sendToArduino = ledVal + "," + ledVal + "\n";
writeSerial(sendToArduino);
}
}
The modifications in the p5.js code make it so that the user clicks on the screen to set up the serial. Once the game is started, the reading from the potentiometer is used to adjust the wind, and everytime the ball hits the ground the LED lights up. One thing that I struggled with was how to stop the LED from being perpetually lit up while the ball was just on the ground.
Concept: For this project, we knew we wanted to implement something piano-esque, i.e. pressing buttons and implementing different sounds. Essentially, it is a simple Arduino circuit that uses three buttons and a buzzer, playing a note every time a button is pressed. However, we wanted to make the concept a little more fun, so we decided to add a photocell (it was also a cool way to add more notes than just 3). When the photocell is uncovered, the buttons play higher frequency notes (A flat, G, and F) and when it’s covered, it plays lower frequency sounds (D, C, and B). The idea is in daylight the keys play more upbeat-sounding tones, while at night they make more sombre-sounding noises.
Video & Implementation
The code for this project was relatively straightforward. The frequency equivalent in Arduino for the notes were taken from this article.
int buzzPin = 8;
int keyOne = A1;
int keyTwo = A2;
int keyThree = A5;
int lightPin = A0;
int brightness = 0;
int keyOneState, keyTwoState, keyThreeState;
int flexOneValue;
void setup() {
Serial.begin(9600);
}
void loop() {
brightness = analogRead(lightPin);
keyOneState = digitalRead(keyOne);
keyTwoState = digitalRead(keyTwo);
keyThreeState = digitalRead(keyThree);
Serial.println(brightness);
if (brightness < 45) {
if (keyOneState == HIGH) {
// tone(buzzPin, 262);
playTone(587); //D flat
} else if (keyTwoState == HIGH) {
playTone(523); //C
} else if (keyThreeState == HIGH) {
playTone(494);
}
} else {
if (keyOneState == HIGH) {
// tone(buzzPin, 1000, 400);
playTone(831); //A
} else if (keyTwoState == HIGH) {
playTone(784); //G
} else if (keyThreeState == HIGH) {
playTone(698);
}
}
}
void playTone(int frequency) {
int duration = 200;
tone(buzzPin, frequency, duration);
delay(duration);
noTone(buzzPin);
}
Future Highlights and Improvements
One thing that we both thought would be cool to implement on a longer form of this project would be to have different levels of brightness play different notes, rather than just having two states. It would also be cool to incorporate different forms of input, such as the flex sensor (we tried to use it but it was a bit buggy so we scrapped the idea).
Last semester, I took a core class called Touch that discussed the very phenomenon Bret Victor is discussing in A Brief Rant on the Future of Interactive Design. In the class, we took time to understand the sheer capability of human touch and the richness of sensory experiences. Inevitably, we also discussed how modern technology is a disservice to the human tactile and proprioceptive experiences, and could even be contributing to the dullness of our sensory capabilities. Needless to say, I believe Victor’s annoyance at the ‘Pictures Under Glass’ world is very understandable and even justified. This might sound like a claim coming from an angry old boomer, but I genuinely believe most people in today’s world (unless they make the effort to seek it) have extremely limited haptic experiences. We live in an era where the current technology renders our haptic system obsolete. I’d even say that this lack is felt to some capacity in individuals, as you often find people gravitating to objects that are old-fashioned, or even cumbersome, just for the vibes or the aesthetic. Mechanical keyboards, polaroid and film cameras, and even flip-phones. This is because humans were inherently made to touch and feel.
In his follow up, Victor also brings up a point about how using touch-screens is easy because they have been dumbed down enough to be used by toddlers. He says that adults deserve “so much more”, implying that multimodal experiences need to be made for adults. However, I find it important to point out that any sort of primitive interactive can be performed by a child – it’s just that in this day and age, pictures under the glass are all we have. Children deserve more as well, and there should be research on technology as such.
I believe that both of these readings highlight two important facets of physical computing.
Physical Computing’s Greatest Hits (and Misses) is an informative piece that aims to categorize (and explain) some of the most notable genres of physical computing projects. In the preamble to the piece, in which the author states that despite certain themes recurring, it doesn’t mean that the projects that use those ideas are unoriginal. In fact, they go on to assert that these forms of interaction allow for a lot of innovation and creativity. In my opinion, this is incredibly important, as it reminds us of a key principle in physical computing: the sensory input need not aim to be unique, but what we do with that input should.
Making Interactive Art: Set the Stage, Then Shut Up and Listen highlights a second key component of physical computing. Physical computing is inherently an interactive paradigm and necessitates a user-computer interaction. It might be tempting to provide clear instructions or an elaborate backstory to the project, but part of the beauty of interactive artworks is the limitless ways one can play around with them. Letting the user explore the project itself is the only way to truly realized the potential of a physical computing project.
For this assignment, I decided to make a simple circuit using three LEDS, a switch and a potentiometer.
The idea is that it functions as a way to communicate three moods: anger, sadness and happiness. The mood is decided through the potentiometer, which maps the analog reading to a number within the range 0 – 90. If it’s in between 0-30, the green LED lights up, if it’s between 30 – 60, the blue LED lights up, and anything after 60 up till 90 makes the red LED light up. The LEDs blink in morse code – the red LED spells ANGRY, the blue LED spells SAD, and the green LED spells HAPPY. The digital switch in the circuit is used to turn the circuit on or off, with the LEDs only blinking if the switch is pressed.
code highlights:
const int greenLED = 12;
const int redLED = 11;
const int blueLED = 10;
const int btn = A2;
const int pot = A1;
int value;
int currLED = 0;
void flashDot() {
digitalWrite(currLED, HIGH);
delay(250);
digitalWrite(currLED, LOW);
delay(250);
}
void flashDash() {
digitalWrite(currLED, HIGH);
delay(1000);
digitalWrite(currLED, LOW);
delay(250);
}
void setup() {
pinMode(redLED, OUTPUT);
pinMode(greenLED, OUTPUT);
pinMode(blueLED, OUTPUT);
Serial.begin(9600);
}
void loop() {
int btnState = digitalRead(btn);
int potVal = analogRead(pot);
value = map(potVal, 0, 1023, 0, 90);
Serial.println(currLED);
Serial.println(value);
if (btnState == LOW) {
digitalWrite(greenLED, LOW);
digitalWrite(redLED, LOW);
digitalWrite(blueLED, LOW);
// currLED = 0;
}
if (value <= 30) {
currLED = greenLED;
// digitalWrite(redLED, LOW);
// digitalWrite(blueLED, LOW);
} else if (value > 30 && value <= 60) {
currLED = blueLED;
// digitalWrite(greenLED, LOW);
// digitalWrite(redLED, LOW);
} else {
currLED = redLED;
// digitalWrite(greenLED, LOW);
// digitalWrite(blueLED, LOW);
}
if (btnState == HIGH) {
// digitalWrite(greenLED, LOW);
// digitalWrite(redLED, LOW);
// digitalWrite(blueLED, LOW);
// } else if (btnState == LOW) {
if (currLED == greenLED) {
digitalWrite(blueLED, LOW);
digitalWrite(redLED, LOW);
flashDot(); // H
flashDot();
flashDot();
flashDot();
delay(1000); // Gap between letters
flashDot(); // A
flashDash();
delay(1000); // Gap between letters
flashDot(); // P
flashDash();
flashDot();
flashDot();
delay(1000); // Gap between letters
flashDot(); // P
flashDash();
flashDot();
flashDot();
delay(1000); // Gap between letters
flashDash(); // Y
flashDot();
flashDash();
flashDash();
delay(1000); // Gap between words
} else if (currLED == blueLED) {
digitalWrite(greenLED, LOW);
digitalWrite(redLED, LOW);
flashDot(); // S
flashDot();
flashDot();
delay(1000); // Gap between letters
flashDot(); // A
flashDash();
delay(1000); // Gap between letters
flashDash(); // D
flashDot();
flashDot();
delay(1000); // Gap between words
} else if (currLED == redLED) {
digitalWrite(blueLED, LOW);
digitalWrite(greenLED, LOW);
flashDot(); // A
flashDash();
delay(1000); // Gap between letters
flashDash(); // N
flashDot();
delay(1000); // Gap between letters
flashDash(); // G
flashDot();
flashDot();
delay(1000); // Gap between letters
flashDot(); // R
flashDash();
flashDot();
delay(1000); // Gap between letters
flashDash(); // Y
flashDash();
delay(1000); // Gap between words
}
}
}
The code is pretty simple and straightforward. One thing I like about my code is using the functions flashDash and flashDot, as it made it much easier to translate the morse code into blinking.
reflections:
One thing I struggled with and couldn’t really fix and/or understand why it was happening was the delayed transition between states, i.e. it takes a while to go from green to red etc., or even to turn off (as seen in the video). In the future, I’d want to be able to assess the root cause of this issue as it could be very problematic in other sorts of circuits where timing is very necessary.
These days, there is this board in D2 with cardboard cutouts that mention ways to eat sustainably. I coincidentally happened to pay attention to the board right after our last IM class, which gave me an idea – a switch that lights up when the cutlery touches the plate. The premise is that if you get a portion proportional to your appetite, and are then able to finish your food (the cutlery touching the plate), the LED lights up.
It is a simple series circuit consisting of an LED and a resistor. I used aluminum foil on the base of the plate and the spoon so that a larger surface area is conductive. whenever the spoon makes contact with the plate, the circuit completes and the LED lights up. I used a green LED to indicate successful completion of a meal.
reflections:
There is certainly room for improvement. I didn’t try it out with actual food, so I’m not sure if the food content would actually affect the way the switch works. Also, the foil doesn’t actually cover the length of the plate. Because of this, even though it technically *works* it doesn’t capture the essence of the concept fully. In addition, I’d also love to improve the visuals of the project, because the aluminum foil and tape do make it look a bit tacky.
“Emotion and Design: Attractive Things Work Better” by Donald A. Norman
I really enjoyed reading Norman’s work, because it perfectly encapsulates many discussions we have been having in class about design and usability. My overall takeaway from the piece was that good design exists at the intersection of aesthetics and functionality, though there is a place in the world for highly functional but ugly or highly aesthetic/creative but unusable items.
Norman’s piece, however, brought an interesting dimension to the conversation that I hadn’t heard of before: the link between design (or more accurately, perceived attractiveness) and emotion. I really agree with his sentiment that in neutral situations, “Attractive things work better”. In my opinion, in everyday objects, the more aesthetically pleasing an object is, it is not only more likely to be purchased but also more likely to be held on to for longer (i.e. it’ll be a long while before it ends up at some random Goodwill). It’ll also be more likely to be treated with care – amongst myself and my peers, I’ve noticed the more aesthetically pleasing an object looks, the more likely we are to be able to take care of it. For neutral objects, in most situations, the efficiency cannot outweigh the ugly – and the moment the user feels no need for the tool, it is more likely to be gone or treated carelessly.
“Her Code Got Humans on the Moon—And Invented Software Itself” by Robert McMillan
I thought this article was quite informative (I didn’t know the inventor of modern software was a woman!) and I didn’t have much thoughts on it, but one point that was really interesting to me was the lack of emphasis on error-checking mechanisms at the time. The idea that higher-ups at NASA believed it was unnecessary the astronauts were “perfect” is so unbelievable to me, not only because the stakes were so high, but also because the fallibility of users is one of the main considerations in software engineering today! I think this reflects a shift in the mindset that has occurred between then and now. Their inability to believe that the astronauts could make a mistake reflects that they believed that user was smarter than the tool, whereas in modern computing we always believe the tool has to be smarter than the user.
(for some reason the initial start screen music doesn’t play on full-screen)
concept:
I knew from the beginning that I wanted to make a game. The aesthetics I had in mind for my project were always there, too – I’m a huge fan of 16-bit and 8-bit pixel art, and I knew I wanted to incorporate that art style into my work. I was also torn between the type of game I wanted to make; I could make it a complex story told through an RPG game format, or I could lean towards something more simple but with more gameplay. In the end, I ended up doing the latter, because it held one’s focus on the project for much longer. The gameplay for my project was inspired by simple platform games like Google’s T-Rex runner game or even Flappy Bird. Essentially, the player stays in one position throughout the game but has to avoid obstacles and/or hurdles throughout the way. My game has collectibles, which the player gathers for points, and obstacles which the user has to jump/avoid or else they lose points.
The main thing that I thought about was what I wanted the setting of the game to be. This semester, I’m taking a core class called Hindsight which focuses a lot on autobiographical story. So naturally, my mind went towards something similar: different life phases for different levels. Initially I was making it my own, but then I realized it would be infinitely more interesting if I loosely based the levels around my father’s life. He’s lived through much more different ‘settings’ (I’ve been in a city for my entire life). The game isn’t super focused on his life story, but the general inspiration for each level can be seen below.
chapter 1: the simple life
My dad grew up in a village, and on a farm, so the setting of this level is inspired by that! You also collect mangoes because that’s his favorite fruit.
chapter 2: the concrete jungle
In his late teens/early twenties, he moved from his village to Karachi, a large metropolitan city. It was his first time being in such an environment, so the level is based on the novelty of that. Cars, crows and rats are things one often finds in big cities like this which is why they’re the obstacles.
chapter 3: setting sail
After completing training at his cadet college (which was why he moved in the first place), my dad officially joined the Navy. He spent a lot of time at sea in his early years, which is why the last level he’s on a boat. At first I wanted there to be fish or something, but that seemed a bit difficult so I just settled on making the obstacles birds.
how it works
The functionality is relatively straightforward. There is a player object, which actually stays in place the entire game. The only movement from the user is that it can jump (and double-jump!) using the spacebar. The player object is given the illusion of moving because everything else – the obstacles, the collectibles, the background – move towards the player. The collectables and obstacles essentially have speeds at which they move towards the user, and if the player collides with a collectible points are added and deducted for obstacles. The placement of the obstacles and the collectables are random. Once the user has passed all obstacles in the game, the game advances to the next level.
One of my favorite things to do when I’m working on a technical creative project is figuring out the ambiance/aesthetics of the piece. Sometimes, I spend more time on that than the actual code. So needless to say, I’m the most proud of the aesthetics of this project. Not only does it it looks very in line with my initial vision of a retro game project, but I’m proud of how I was able to create the atmosphere for every setting. The background graphics, the obstacles, the collectibles, and the music. I think it all ties together very well and makes the game super immersive.
improvements
While I’m happy with how my project turned out, there is definitely room for improvements, or small changes that could completely elevate the level of my work.
Canvas size! I think because I was thinking of platform games on small screens, I kept the canvas size small. But I now wish it was bigger – unfortunately my code wasn’t typed in the most adaptable way, so it became a bit difficult to change the size towards the end.
Small features like being able to turn off the music would help a lot as well as I know the game is a bit heavy on the sound, which can be a bit disruptive.
Making the scoring system much more refined, i.e. adding points based on how elevated the collectable object is.
These changes seem relatively straightforward to implement, but I couldn’t mostly because of time restrictions.
