Author: Shota Matsumoto

Final Project Concept:

“Tokyo Flight” is an interactive flying game that is inspired by one of Ghibli movies called The Wind Rises. The player controls a cardboard airplane equipped with an accelerometer and a button, both of which are connected to Arduino. The way the game works is that the user is going to tilt the airplane and the airplane on p5 will follow the movement of the tilt. The goal of the game is to pass through as many floating rings as possible without touching them or going out of the canvas. If users press the button on the airplane, then the speed of the airplane on p5 will boost up to help reach difficult rings. If the player collides with a ring or flies out of the screen, the flight ends. 

 

Arduino Components:

• • Accelerometer (MPU6050) inside the airplane: It measures the tilt forward and backward. Possibly left or right tilt as well.
  • Push buttons: It triggers a temporary speed boost.
  • LED lights: Light up when the boost is active

 

What Arduino does:

1. 1. Read sensor values: 
      • It continuously read acceleration + gyroscope data from the MPU 6050
      • We smooth the data to remove the noise 
      • We convert the tilt into a single value (from -90 degrees to 90)
   2. Read button state: 
      • We check if the button is pressed or not
      • Sends a signal (0 or 1) 
   3. We send data to p5.js through webserial 
   4. Additional feature: 
      • We receive signals from p5 to blink LED when boost is available 
      • Vibrate the motor when the user crashes 

 

What p5.js game will do:

1. 1. Connect to arduino via webserial 
      • We parse incoming serial strings and extract tilt angle and button state
   2. Control the on-screen airplane
      • We map tilt to the airplane’s vertical position. 
      • We also map horizontal tilt to slight horizontal drift
      • If speed boost button is pressed, we increase forward movement temporarily 
   3. Generate game objects: 
      • Rings appear from the right and move left 
      • Rings vary in size and vertical position 
      • We generate a wind to affect the airplane’s movement 
   4. Collision detection: 
      • If the airplane touches ring boundary, it is game over 
      • If the airplane goes beyond the canvas, it is game over. 
   5. Scoring System 
      • +5 for passing through small rings
      • +2 for passing through big rings 
   6. Visual and Audio design: 
      • We will have a soft blue sky with clouds and a green glassfield at the bottom to represent Wind Rises.
      • We will also play Ghibli audio while the game is playing 
   7. Send feedback to Arduino
      • Arduino flashes LED lights when the airplane crashes
      • Arduino keeps lights on when the airplane is on boost mode 
      • Arduino vibrates when the airplane goes through the rings.

 

Main Concept

The main concept for my final project is the “Lock In Buddy.” It basically never lets you stop studying. Your laptop acts as the monitor, and the system uses Arduino sensors and p5 to track your behavior while you are studying. If it detects your phone, it triggers a super loud alarm. If it senses that you’re leaving the table using a distance sensor, it activates the speaker and says, “Come back to the desk. You gotta lock in!” I also want to incorporate the Pomodoro technique, where you study for 25 minutes and take a 5-minute break. A student can press a physical button to start the timer, and press it again to stop it. At the end, the system will give you a score based on how distracted you were by environmental variables, like looking at people walking by, eating, or any other interruptions. Essentially, it becomes your study buddy until you graduate.

 

Lock In Buddy will have these features: 

• • Phone detection alarm 

It will detect when the user picks up a phone, uses, or brings their phone to a nearby desk. The monitor is always watching the user to make sure that they don’t use their phone. 

• • Leaving the desk detection 

The distance sensor detects when the user moves away from the study area. It plays a warning voice message saying “you gotta lock in buddy. Or you will become homeless”. 

• • Real time distraction detection 

The system detects whether you’re being distracted by environmental conditions. Those include looking at people passing by, eating, chair movement, or noise.

• • Pomodoro timer with physical buttons 

Users can press the button to start, pause and stop the Pomodoro session. Also, it will record how long you have been studying for, which is going to be recorded for the scoring system as well. 

• • Focus score generation 

P5 will calculate the number of distractions, phone usage count, time away from desk, noise levels, and the number of successful Pomodoro sessions.

 

How to integrate p5 and Arduino

Phone detection + alarm: 

• • p5.js + Webcam + ML5 COCO-SSD 

Or 

• • Arduino light sensor + phone jail box

• • Start playing the warning alarm saying “Lock in” in p5

 

Eating or talking: 

• • Sound sensor

 

Pomodoro Timer with physical buttons: 

• • Buttons with arduino

 

Leaving desk: 

• • Ultrasonic distance sensor 

 

Focus Score generation: 

• • Arduino continuously reads 
    • Light sensor → phone removed 
    • Ultrasonic sensor → left desk 
    • Sound sensor → eating/talking 
    • Button → start/stop Pomodoro
  • Send those data to p5 and calculate the score 

This reading gave me a new perspective on the intersection between disability and technology. I didn’t know that eyeglasses were initially made for the medical purposes only, so it was really interesting to learn how societal perceptions and stigma can change so dramatically over the course of time. I realized how powerful designers are in shaping these perceptions, as discussed in the reading. This is definitely something that engineers cannot achieve by themselves. As a prospective software engineer, I learned that my goal should not be just to make things useful, but to make them as simple and usable as possible for everyone in the world, like AirPods, irrespective of nationality and background. I’ve personally experienced how complexity can mess up the development process and even the essential purpose of a product.

 

For instance, I’m currently developing my mobile app that turns TikTok into studying. Initially, I wanted to integrate a social media feature where users can compete with each other in terms of how many questions they can answer in a row. But that feature turned out to be super complex, and spending too much time on implementing that single feature prevented the essential purpose of my app, which is to make studying addictive for students in the world. That is the reason why I decided to focus sorely on the scrolling system, since it is the simplest and most essential feature of TikTok that makes it so addictive. Through this reading, I was able to understand how important it is to make a product as simple and usable as possible for all the users in the world.

EXERCISE 01: ARDUINO TO P5 COMMUNICATION

Concept:

For the first exercise we used a photoresistor as an analog sensor on Arduino to make our ellipse shape move on the horizontal axis in p5. For our code, we adjust it so that with every touch on the photoresistor, the ellipse would move on the canvas.

VIDEO:

video1

Code we’re proud of:

When it comes to the first exercise, the part we are most proud of is the block of code that controls the x-position of the ellipse using the light sensor value from the Arduino. We read the sensor value through the serial port that is connected to the Arduino and map it from 0 to the width of the canvas so that it never goes out of the dimension of the screen.

//read the data coming to the port until the newline character
   let data = port.readUntil("\n");
   //if there is a data
   if (data.length > 0) {
     //convert the clean data into integer
     rValue = int(trim(data));


     //store data of left and right into msg
     let msg = left + "," + right + "\n";
     //send the msg back to the Arduino
     port.write(msg);
   }


   text("rValue = " + rValue, 20, 50);
 }


 //move xpos depending on the sensor value (max = 640)
 let xpos = map(rValue, 550, 900, 0, width);
 fill(0, 0, 255);
 ellipse(xpos, height / 2, 100, 50);

 

EXERCISE 02: P5 TO ARDUINO COMMUNICATION
Make something that controls the LED brightness from p5

Concept:

For the second exercise we used the createSlide function to create a slider on the canvas of p5. This way, every time we moved the slider to the right, the brightness of the LED on the Arduino would increase, and every time the slider was moved to the left, the brightness would decrease.

VIDEO:

video2

Code we’re proud of:

For the second exercise, this block of code combines what we learned during the class about p5.js (createSlider) and Arduino. Since we remembered using a slider to control a value or game score, we decided to use it to control the brightness of the LED. We take the slider value, store it in a variable called brightness, and send it to the Arduino through the serial port, as shown in the block of the code below.

//read the slider value
 brightnessValue = brightnessSlider.value();


 //if the port is open
 if (port.opened()) {
   //store the brightness value from the slider
   let msg = brightnessValue + "\n";
   //send it to Arduino
   port.write(msg);
 }

 

EXERCISE 03: BI-DIRECTIONAL COMMUNICATION

Concept:

For the third exercise, we made use of a potentiometer ( analog sensor ) on Arduino and a LED light, connecting it on the corresponding pins and then connecting Arduino UNO with P5. After adding the necessary codes on P5, we adjusted our “ball” so that every time it bounced against the floor, the LED would light up, and once it stopped bouncing the LED turned off. With the potentiometer, we were able to control the “wind”, causing the ball to be pushed from side to side, depending on the direction in which we turned the potentiometer.

VIDEO:

video3

Code we’re proud of:

For the third exercise, we’re proud of the block of code below. It basically captures the main objective of this exercise. When the ball reaches the ground and the LED state is 0, meaning the LED is currently off, we turn it on by sending a 1 to the Arduino through the serial port. It is simple but we needed to understand the basics of what is going on in the vectors. 

//if the ball reached the floor and the led is currently off
 if (isBounced && ledStatus === 0) {
   //turn the LED on
   port.write("1");
   //set the state to 1
   ledStatus = 1;
 }
 else if (!isBounced && ledStatus === 1) {
   //turn the led off
   port.write("0");
   ledState = 0;
 }

 

Challenges and Further Improvements:

Fortunately, we were able to complete all the assignments successfully, and managed to achieve all the necessary responses of P5 and Arduino. For each exercise, we started by planning out which tools we would use, how to arrange them on our breadboard, and once this was done, our greatest challenges consisted of arranging and adding the codes required to make each response function. On the third assignment we especially struggled writing the codes required to make the potentiometer affect the “wind” and consequently the direction in which the ball was pushed. For future improvements, we would like to grasp a better understanding of the exchanges between P5 and Arduino in order to make more complex and interactive responses that can also satisfy the assignment’s requirements.

 

Github Link:

exercise1 Arduino

exercise1 p5

exercise2 arduino

exercise2 p5

exercise3 arduino

exercise3 p5

Reference:

We used ChatGPT to help us understand the bouncing ball template code since we hadn’t yet learned about the vector library in class. After understanding the foundation of it, we were able to integrate the isBounced logic into the template code to detect whether the ball was bouncing or not. Thus, we’re glad we were able to learn a lot about vectors library. We also used ChatGPT to learn about DOM library methods such as .position() and .style(). Although we had covered the basics of createSlider, we didn’t delve deeper into it during the class, so we needed to learn how to adjust its position and CSS styling of the ball.

After reading this article, I was reminded of Mark Zuckerberg’s quote where he said that the next biggest innovation is going to be VR. The author emphasized that the incorporation of more senses into these technological devices allows people to really interact with them in a much deeper sense, whereas smartphones are just composed of screens and we just scroll on the glass, not getting any feedback. At first, I thought we still get tactile feedback from the screen. For instance, when we are playing games and get attacked by someone, we feel the vibration. But that does not foster any intimate interaction between users and the device itself. What makes an experience more unique and immersive is stimulating as many human senses as possible, something that I learned in the class I took when I was a sophomore, called Immersive Experiences. I think for my final project, I want to do something that can impact people’s thoughts and make them reflect on their past experiences through immersive experiences. Through these two articles, I learned that we are shifting to a world where the distance between technology and humans is getting much closer. I am somewhat worried about it because there are already cases where people get confused about their identities since they spend most of their time in a digital world. I think it is best to create a boundary between us and technology to remain humanities in this world.

Main Concept:

The main concept for our group project is a DJ, since we wanted to experience what it feels like to be one. A DJ needs to handle many sounds and instruments using their unique artistic skills to create music that makes people happy and excited. Thus, we crafted this device called “NYUAD DJ Kit.” By using it, you can choose different kinds of songs with various speeds and a base sound produced by a wooden stick. This is a unique way to compose new kinds of songs as a DJ. 

Demonstration Video

 

Schematic:

 

Code we’re particularly proud of:

The parts of the code we’re most proud of are the one shown below. The if else statement allows us to move to the next song and play it. When the button is pressed, meaning the pin is low, we set buttonPressed to true and noteIndex to 0 so that the song plays from the beginning. We also used the modulo operator to ensure that we always go back to the first song after the last one. The else if statement resets the buttonPressed state to false, so that the next time we press the button, it plays the next song. 

//handle music switching using modulo
if (digitalRead(BUTTON_PIN) == LOW && !buttonPressed) {
    buttonPressed = true;
    //move to the next song
    currentSong = (currentSong + 1) % 3; 
    //set note to 0 so that a song plays from the beginning
    noteIndex = 0; 
    isPlaying = false;
    noTone(BUZZER_PIN);
    delay(250); //delay for 250 milliseconds
  } else if (digitalRead(BUTTON_PIN) == HIGH) {
    //set buttonPressed to false to play next song 
    buttonPressed = false;
  }

 

The second snippet of code allows the servo to move every servoDelay milliseconds, controlling its speed and angle. We applied the concept we learned in class called “non-blocking” to ensure that this operation does not affect the rest of the program. Inside the if statement, we use the write() function from the Servo library to change the servo’s angle each time it runs. This way, the servo continues changing its angle until it reaches either 180° or 0°, incrementing by a step each time servoDelay milliseconds have passed. We’re happy that we were able to apply multiple concepts we learned in class, such as non-blocking and modulo, to our DJ project. As references, we used the example file ToneMelody from the Digital section and the Knob example from the Servo library. We also used ChatGPT to help us figure out how to apply the non-blocking concept so that the melody can move to the next one without affecting the rest of the program, which allows the servo to continue moving smoothly.

//Use non-blocking to not affect the rest of the program
if (currentTime - lastServoUpdate >= servoDelay) { //if servoDelay mills has passed
    lastServoUpdate = currentTime;
    //Change the angle of the servo by servoPos 
    myservo.write(servoPos);
    servoPos += servoStep;
    //Start decrementing if servo reaches 0 or 180 degrees
    if (servoPos >= 180 || servoPos <= 0) servoStep = -servoStep;
  }

 

Link to GitHub: https://github.com/KimShota/Intro-to-IM/blob/13b494508781fc36c9b95d3b46e5145d18c06808/nyuad_dj.ino

Reflections & Future Improvements:

In terms of reflections, we struggled a lot to make the base work because we needed to attach the wooden stick to the servo and it was not stable at all at first. The way we attached it was by using tape, which was the primary cause of why it was unstable. As a result, every time we ran the servo fast, the stick fell off or the servo stopped working for some reason. We eventually managed to make the stick and servo stable by placing some weight on top of the setup so that no matter how fast the servo moved, the stick remained stable. 

As for future improvements, we want to enhance the quality of the base because right now we’re just using a wooden stick, and it doesn’t produce a loud enough sound for a party situation. Furthermore, as the stick swings faster, its swing range becomes smaller, so we need to move the bottle manually to allow the stick to reach it. We believe this happens because the servoDelay becomes too small, reaching about 1 ms, so the servo can’t physically keep up. Therefore, next time we should use constrain() on the mapped value to prevent electrical noise from going out of range. This way, we can allow the servo to catch up with the speed that we want.

Thinking about the artwork “Remote Hugs”, I believe current physical computing devices are incapable of understanding and replicating human emotions due to their complexity. Emotions are constructed through a series of events that occur throughout one’s life, and we need to comprehend each event to truly understand and simulate one’s emotions. Since Remote Hug does not have the capability to analyze people comprehensively, I think it cannot replicate human warmth. Even the author mentioned that they were not able to understand the motivation behind it. Thus, I believe we need to integrate deep learning or machine learning mechanisms into such affective computing devices to better understand one’s emotional states.

In terms of the second reading, I disagree with the author in the sense that we should not completely let the audience interpret the meaning and purpose of an artwork on their own. I used to be a big fan of this idea, but that idea changed after I went to teamLab. When I visited teamLab, I experienced all the immersive artworks there but I was confused about what they were trying to convey and what kinds of experiences they wanted me to have. At the end of the day, I was completely lost and feeling dizzy heading back to campus. I think the essence of an artwork is to guide the audience through the experience, allowing them to connect it with their own personal thoughts and emotions. It goes back to the discussion we had about how we can incorporate randomness into artwork. As an artist, I think it is important to have a clear motivation and purpose behind the artwork and to guide the audience in how to interpret it.

Main Concept:

I was always curious about using a temperature sensor because I could never imagine how such a tiny device can detect human temperature and convert it into volts. Therefore, I used this homework to experiment with the temperature sensor. I also decided to use the slider switch because I love the feeling of opening and closing it. I learned how to use the temperature sensor through an online tutorial (tutorial) since we have not yet covered it in class. It works quite simply. We supply electricity through the +VS pin to measure temperature internally, and the measured analog voltage is then sent back to the Arduino through the GND pin.

 

Figure 1 : temperature sensor + slide switch

 

Schematic

Full Video of Demonstration

 

Code I’m proud of:

This is the part of the code that I’m most proud of. First, I read the sensor value from pin A0 and convert it into millivolts since the temperature sensor calculates the corresponding voltage in millivolts. Then, I crafted this formula based on what I learned from the tutorial: each degree Celsius corresponds to 10 millivolts. Because the output voltage increases by 10 millivolts per degree Celsius, starting from 500 millivolts at 0°C, I subtracted 500 from the measured millivolts and divided the result by 10 to map the temperature to its corresponding voltage value. Converting Celsius to Fahrenheit was quite easy since I had already learned that in chemistry class.

//read the volatge from analogPin A0
  sensorValue = analogRead(A0);
  //convert digital numbers (0 to 1023) into voltage values 
  volts = sensorValue * maxVoltage / 1023.0;
  //convert volts into millivolts 
  millivolts = 1000 * volts;
  //convert into temperature in celsius
  celsius = (millivolts - 500)/10;
  //convert into temperature in fahrenheit 
  fahrenheit = (celsius * 9/5) + 32;

 

Reflections & Future Improvements:

Overall, I loved learning how to use the temperature sensor since I have always been fascinated by how such a tiny component can capture environmental variables. I was especially curious about why it has three pins, and I was able to understand the internal mechanism where we supply electricity to measure temperature, convert it into analog voltage, and send that voltage back to the Arduino. It was a bit challenging to find the right thresholds for each LED light to turn on and off smoothly. Sometimes I set the threshold of the last LED too high, so it never turned on. However, I was able to adjust them correctly through multiple experiments. For future improvements, I would like to combine the slider switch and temperature sensor so that they work together to control shared LED lights seamlessly. I tried doing this, but I had so many wires and LEDs that I struggled to find a proper way to connect them. Next time, I will spend more time figuring this out.

Attractive Things Work Better

It was really interesting for me to learn how human cognition and emotion are intertwined and how that relationship can determine which design is the best. I personally agree with the author’s idea that more aesthetic and pleasing designs can enhance people’s creativity and allow them to accept even small mistakes or glitches. However, I think that even if a design is aesthetic, if it is too complicated and difficult to use, it will still make me feel frustrated. 

Thus, I believe it’s all about finding the perfect balance between usability, aesthetics, and the emotional impact we gain from the design. In terms of the emotional impact, it is very hard to determine which design works best for people in both pleasant and stressful situations. However, I think it’s always important to make designs both aesthetic and simple at the same time. For instance, Notion is a very simple yet visually pleasing website, and I think that was the key to its success. I know where the templates are, so I can easily choose which template I want to use for my daily diaries and so on. As a software engineer, it is always hard for me to find the right balance between simplicity and aesthetics. However, getting as much user feedback as possible makes it easier for me to find any improvements and add features that make people’s lives easier.

 

Her Code Got Humans On The Moon

As someone who wants to become a software engineer after graduation, I totally agree that the way Hamilton approached designing software that handles crucial tasks and resolves real-time errors is very important, especially in today’s society. Nowadays, anyone can build almost anything from scratch due to technological advancements without actually knowing how to code at all. However, the real challenge that we, as software engineers, need to tackle is handling critical errors when something goes wrong. If we do not understand the fundamental principles of how software interacts with other components, such as databases, everything in this world will start to malfunction. 

After reading this article, I realized that I want to become a software engineer like Hamilton who can prioritize customers’ needs in  the first place and handle errors in real time. I believe these two skills are fundamental parts of a software engineer’s job in today’s world.

Main Concept:

In Japan, the way you bow is really important when it comes to apologizing to someone older than you, and even some teachers in middle and high school teach their students how to bow properly. It is considered that bowing to 90 degrees is the most polite and respectful way to apologize to someone older than you. When I was coming up with the idea for this project, the first thing that came to mind was to teach people how to bow correctly to 90 degrees. The main concept is that the light can only be turned off if you bow to 90 degrees.

The link below is the video of me demonstrating my proper bowing detector.

My Proper Bowing Detector

Highlight of the code:

This is the code that I wrote. I wanted to use different pin numbers instead of pin 13 or different analog pins, such as  A1 or A3, but after changing them, my circuit suddenly stopped working. Thus, I chose to stick with pin 13 and A2. Inside the code, I used variables for pin numbers so that I can easily change them in the future. However, I want to figure out with my professor why my circuit suddenly stopped working just by changing the pin numbers.

int ledInput = 13; //variable to control LED
int sensorInput = A2; //variable to control sensor

void setup(){
  pinMode(13, OUTPUT); //send singnals from pin 13
  pinMode(A2, INPUT); //receive signals from A2
}

void loop(){
  //read signals from A2
  int buttonState = digitalRead(A2); 

  if(buttonState == HIGH){
    digitalWrite(13, LOW); //turn off LED if my forehead touches the foil 
  } else {
    digitalWrite(13, HIGH); //turn on LED
  }
}

 

Reflection and Future Improvements:

Since it was my first time building hardware using an Arduino, it took me a tremendous amount of time to understand how things work, especially the switch example we did in class. Thus, I spent a lot of time going through the slides and experimented for several hours to figure out which side of the jumper wire I can connect to the aluminum foil to make this project work. I really struggled with turning off the LED by placing my head on the A5-sized foil paper on the table for several reasons. First of all, the jumper wire was not long enough, so I had to connect multiple wires, but some of them got unplugged during the trials. Furthermore, since the tape is not conductive, when I covered the tip of the wire, the LED didn’t turn off. I tried using conductive tape because I thought it would allow electricity to go through the wire, but that didn’t work as well. At the end of the day, I made the foil paper on my forehead as large as possible to increase the surface area and only applied the tape to the sides of the foil so that it wouldn’t cover the tip of the wire. 

As for future improvements, I would like to use a sensor instead of a foil paper so that my users don’t have to put the wire on their forehead, which would be inconvenient to use. I also want to create a sensor that can precisely measure 90 degrees to detect the proper Japanese bowing style.