Category: F2023 – Shiloh

Concept:

This project follows my initial idea of making a digital petting zoo. The name of this zoo is PixelPals. In this petting zoo, you are able to approach different animals and make different interactions with them based on their needs and personalities. The theme and design is quite adorable, aiming to create an atmosphere of comfort and fun. The soundtrack of the background music is from the Nintendo Game, Animal Crossing, something very similar to this project. This project also includes a dedicated control box.

https://editor.p5js.org/Yupuuu/full/2L2O-pC8e

Challenges and Highlights:

The most difficult challenge I had was the serial communication. It took me sometime understand how the serial communication truly works so that I could send more complicated codes. Designing levels of the experience is also hard: it’s important to decide which levels one animal uses, and how to differentiate among all these animals, etc. This is especially difficult because I had to deal with two ends of the communication at the same time. Therefore, it took me a lot of time to figure these things out. And also, sometimes the circuit itself did not work and I had to try some methods such as resetting the Arduino or rewiring things to make it work.

Another problem I ran into the wiring the wires and making the control panel. The control panel is smaller than I thought when I was using the laser cutter to make it. This gave me a hard time putting everything into this enclosed box. In fact, I am still not able to put all wires into it. But one advantage of keeping things outside is that I can easily spot the problem if anything happens. However, this is still a lesson learnt: always make the box bigger than I think.

I am especially proud of using many different features in this single project, including state machine, spritesheet, colliders, etc. All these parts work smoothly with each other to make this project possible. And I am also glad how I used fours buttons to complete several different tasks. Allocating these buttons is also important for the implementation of this project.

Arduino code: https://drive.google.com/drive/folders/1tKOEiyYHFteIk_WYt9-tBES8K4-c9bhL?usp=sharing

VIDEO: IMG_1755

Control Panel: Arduino  Mega,  LCD  Display,  Buttons,  Light  Sensor

User Testing

The current control panel is a result of user testing. Originally, there were no instructions at all. Especially when it came to the light sensor, people who did not know what it was had no idea what it could do and how it could relate to the owl. Therefore, I decided to add a shape of the sun to indicate that this can represent a sun and you need to cover this sun for the owl to sleep. Following this, I added instructions to the control panel. However, as a relaxing experience, I do not want the player to spend a lot of time reading the instructions. Therefore, I ended up using some simple illustrations and words to guide the player.

(After the showcase) It was interesting that almost nobody followed the instructions before I told them to. They liked pushing the button even when it was not lit up. And they never read what was on the LCD screen. The only persons who read them were kids. Probably this game was meant for kids. This really shows me the gap between the designed experience and the actual experience people have. It actually reflects on one of our readings that the interactive media artists should let the audience explore on themselves. Obviously, my project is not such a good example in which audience should be allowed to freely explore it as it might break my project… However, overall, people could understand what my project was about and made a laugh when they actually read the texts on the LCD screen!

Reflection & Improvements

This project is very interesting. It prompted me to utilize all the skills I learnt in this class with fabrication skills make this project. It is very satisfactory to see the final product. Reflecting on this project, I realized the power of physical computing and how the hardware and softwire can work together to create unique experiences. However, one of my initial ideas was to create an animal like controller that can act like an animal. However, that would require much more physical constructions, which I did not have enough time for this time. In this future improvement, this could be added, and more interactions could be added to more animals in this zoo. Maybe some customization functions could also be added to make this experience more interesting and personal.

Concept:

My concept for the final presentation is the Interactive Glove: a completely different way of interacting with the computer. The idea behind this project comes from “A Brief Rant on the Future of Interaction Design”, where interaction with digital media is discussed and criticized. After thinking about it thoroughly, I imagined that the best way to innovate in the interaction field would be to use a medium that we commonly use, our hands, but in a way that will utilize the limbs differently.

In this case, we can imagine the interactive glove as a binary system: we have 4 sensors in the left hand and 4 pins on the right hand that will turn on a sensor on contact. This allows for the user to have versatility in terms of how they want to use the glove. Some users will prefer to use one finger for all 4 sensors, while others might require 4 fingers for 4 sensors. Because it is a binary system, the glove can do anything: It can become a mouse, a keyboard, a switch, a controller, etc…

To show the versatility of this input method, we will use p5.js alongside some small games and experiences in order to show how the glove inputs values.

Implementation:

P5.js Sketch: https://editor.p5js.org/ff2185/sketches/A4mnwlg4h

Arduino code:

/*
  DigitalReadSerial

  Reads a digital input on pin 2, prints the result to the Serial Monitor

  This example code is in the public domain.

  https://www.arduino.cc/en/Tutorial/BuiltInExamples/DigitalReadSerial
*/

// digital pin 2 has a pushbutton attached to it. Give it a name:
int inputA = 2;
int inputB = 3;
int inputC = 4;
int inputD = 5;

// the setup routine runs once when you press reset:
void setup() {
  // initialize serial communication at 9600 bits per second:
  Serial.begin(9600);
  pinMode(LED_BUILTIN, OUTPUT);
  pinMode(inputA, INPUT);
  pinMode(inputB, INPUT);
  pinMode(inputC, INPUT);
  pinMode(inputD, INPUT);
  while (Serial.available() <= 0) {
    digitalWrite(LED_BUILTIN, HIGH);  // on/blink while waiting for serial data
    Serial.println("0,0");            // send a starting message
    delay(300);                       // wait 1/3 second
    digitalWrite(LED_BUILTIN, LOW);
    delay(50);
  }
}

// the loop routine runs over and over again forever:
void loop() {
  // print out the state of the button:// wait for data from p5 before doing something
  while (Serial.available()) {
    digitalWrite(LED_BUILTIN, HIGH);  // led on while receiving data
    if (Serial.read() == '\n') {
      // read the input pin:
      int stateA = digitalRead(inputA);
      int stateB = digitalRead(inputB);
      int stateC = digitalRead(inputC);
      int stateD = digitalRead(inputD);
      delay(5);
      int sensor2 = analogRead(A1);
      delay(5);
      Serial.print(stateA);
      Serial.print(',');
      Serial.print(stateB);
      Serial.print(',');
      Serial.print(stateC);
      Serial.print(',');
      Serial.println(stateD);
    }
  }
  digitalWrite(LED_BUILTIN, LOW);
  delay(1);  // delay in between reads for stability
}

 

Arduino:

The Arduino Design is a simple switch system with 4 resistors connected to the ground, each connected to its proper pin and a cable which will be the connection to the 5v source. On the other side, we connected 4 cables to the 5v strip. This allows us to create a circuit in which any time 5v touches a cable that is on the other side, it will set the digitalInput of the pin to 1.

To cover the circuit, I handcrafted a “sandwich” shaped box that will allow the cables to move freely in the vertical axis. The biggest problem was to create an enclosure that would not disturb the cable movement. My idea, to not utilize the classic open top box, was to do this tringular enclosure that allows cables to move up and down freely. Due to the pure nature of the circuit itself, horizontal movement has to be limited in order to not disturb the functioning, so that is why movement is only constrained in the vertical axis

P5.js:

The P5.js sketch is a combination of past projects, examples provided by p5.js and other sketches I have gathered during my research. Each one of the experiences/games is a representation of different inputs styles: For example, the piano will behave differently based on the combination of fingers we press, while the rocket moves from left to right with the index and ring finger.

The available experiences are the following:

• Piano: The piano experience is a recreation of a previous project. Back then, I utilized the keyboard in order to press the piano keys and generate the sound. This time, the gloves will do that function. Every key is mapped to a certain combination. For example. the “DO” note corresponds to touching just the index finger sensor. The following one is mapped to the middle finger, and so on until you will need to use two fingers. The input method shown here is converting the glove into a set of 2^4 inputs. In other words, a binary input.
• Rocket: The Rocket Game involves long pressing a single sensor in order to fly the rocket up, go to the left or go to the right. There is no score, no timer, just you and a single fuel bar. Careful, if you go too low you will crash. Try your best to stay on air!
  This game simulates what it would be to use it as a mouse (the mouse clicks) to control the rocket.
• Disco: The Disco experience is simple. Out of the 4 buttons you press, each of them will show a different ever-changing background. Try all of them and focus on the colors!
  This shows the glove working as 4 different buttons independent of each other.
• Snake: The Snake Game is all about trying to eat the fruits that will randomly spawn in the map. The 4 sensors in the glove represent the 4 directions that the snake can go to: left, right, top, down. This makes the glove resemble the keyboard arrows system.

Code:

One piece of code I am proud of is the state management. A seemingly trivial problem becomes a major challenge when you realize every single experience needs to have a different setup. More over, the different frame rate, strokes, fill, background and other setting that vary in each screen make it especially specific to manage.

if (state == "intro") {
      scoreElem.html("");
      drawIntro();
    } else if (state == "piano") {
      scoreElem.html("");
      drawPiano();
      if (frameCount % 60 == 0) {
        for (var i = 0; i < 10; i++) {
          rSide[i].white();
          black[i].white();
          mid[i].white();
          lSide[i].white();
        }
      }
    } else if (state == "rocket") {
      rectMode(CENTER);
      scoreElem.html("");
      drawRocket();
    } else if (state == "disco") {
      scoreElem.html("");
      drawDisco();
    } else if (state == "snake") {
      scoreElem.html("Score = 0");
      frameRate(15);
      stroke(255);
      strokeWeight(10);
      drawSnake();
    }else if(state == "tutorial"){
      stroke(255);
      scoreElem.html("");
      drawTutorial();
    }

Another code I especially like is the code that allows the snake to move:

function updateSnakeCoordinates() {
  for (let i = 0; i < numSegments - 1; i++) {
    xCor[i] = xCor[i + 1];
    yCor[i] = yCor[i + 1];
  }
  switch (direction) {
    case "right":
      xCor[numSegments - 1] = xCor[numSegments - 2] + diff;
      yCor[numSegments - 1] = yCor[numSegments - 2];
      break;
    case "up":
      xCor[numSegments - 1] = xCor[numSegments - 2];
      yCor[numSegments - 1] = yCor[numSegments - 2] - diff;
      break;
    case "left":
      xCor[numSegments - 1] = xCor[numSegments - 2] - diff;
      yCor[numSegments - 1] = yCor[numSegments - 2];
      break;
    case "down":
      xCor[numSegments - 1] = xCor[numSegments - 2];
      yCor[numSegments - 1] = yCor[numSegments - 2] + diff;
      break;
  }
}

Learnings and Improvements

For this final project, I feel that I have learned a lot from the Arduino development field. Small things that we usually omit during the process of creating these types of projects become bigger problems when it comes to user usability and versatility. For example, the contacts for the glove are really hard to match due to the lesser amount of copper I had available. Moreover, the cables were extremely hard to position properly in order to fix them but also allow movement of the whole arm.

In general, the construction process for the cables and circuit was the hardest part of this project. I would like, in future iterations to improve the cable management and the outside box, maybe ideate some other way to construct a box that will allow the movement of the glove but also easy access if needed (open box).

To conclude, I have really enjoyed every step I took in this class. I have learned ways of expressing my ideas and thoughts in manners I would have never imagined. I have grown to love the idea of exploring new horizons and implementing new designs, approaches, methods for my projects.

Thank you and see you soon.

What if Space had Sound?

• Concept

For my final project, I decided to go with my first idea since I’m interested in gaining more experience in the scientific field and data visualization with a creative twist. My idea is to deliver to the audience some fun facts about space and its exploration through the method of sonification , which is transforming data from electromagnetic waves to sound and musical rhythms.

• Design and description of what your Arduino program will do.

For the Arduino program, I would use the infrared distance measuring sensor and the light sensor. The user would have control over the notes (sound) using both sensors together but each would have different note sound effects, however, they would have similar pitches depending on distance and light, representing electromagnetic waves from pulsar stars. For example, when there is a short distance, there will be high pitch representing stronger radiation, and more light would also mean a higher pitch, representing stronger radiation from a star.

Input – light and distance values /Output – sound and visual changes of wave patterns.

• Design and description of what P5 program will do.

The P5js would represent a starting page, where the user can click either for information about the project with sounds playing in the background from data in a csv file or click “Begin Experience” to head to the space themed page with wave patterns where they use controls from arduino to change the notes while an ambience background sound plays. It will be receiving values from distance measuring sensor and light sensor.

 

Concept

Considering the time limitations, upon deep contemplation of my two pervious ideas, I decided to focus on creating a single game with a unique controller in the form of a glove. Below a brief plan for how the game interface will look, the setups, and what the controls will be is shown.

 

As of now, there are a few more things that I will most likely decide on as I will be constructing the final project very shortly. First of all, I will need to create the game in 5p and make sure it works with no input from the Arduino. Simultaneously, I will work on developing the controlling glove with the Arduino. One issue that I have noticed is that the flex sensors are not extremely accurate, and thus, I will have to benchmark every sensor and determine the specific value ranges that will work well for it (0-1023). I am still unsure if I would want to use the amount of flex to control the speed since that would complicate the process significantly. Therefore, I will most likely create certain conditions in the Arduino code that would send numerical values through the serial communication, which would respond to the different actions in the p5 sketch. Additionally, I will add a few LEDs to the physical controller board to indicate the health points remaining of the player. Another feature I would like to try to implement is allowing users to save their high scores.

Arduino Inputs/Outputs

The Arduino will collect measurements from 5 flex sensors and at least 3 digital inputs (buttons). The data from the flex sensors will be processed within the Arduino code. Consequently, this will determine if certain conditions are met and will send these values to p5.

The Arduino will receive data from p5, which will correspondingly light up LEDs for the remaining health (3 LEDs).

P5.JS Inputs/Outputs

The P5 sketch will receive the data from the Arduino and process it to the corresponding actions to be performed in the game. The P5 sketch will keep sending data about the health points remaining to the Arduino.

In Bret Victor’s “A Brief Rant on the Future of Interaction Design,” he advocates for a more comprehensive approach to technology design, one that surpasses the conventional focus on visual stimuli. He highlights the necessity of integrating multiple sensory inputs, such as touch, sound, and perhaps even smell, to create a more immersive and human-centric experience. This perspective challenges the current trend in technology that heavily relies on visual elements, often neglecting other senses that could enrich the user’s interaction with digital environments.

Victor’s call for a balance in sensory engagement in technology design underlines the importance of harmony in user experience. By harmonizing different sensory inputs, designers can create more intuitive and natural interfaces that resonate with the user’s innate human capabilities. This balance would not only enhance usability but also foster a deeper connection between users and their technological tools, making technology more accessible and enjoyable.

Moreover, the pursuit of this balance in design philosophies raises several pertinent questions. How can designers effectively integrate different sensory modalities without overwhelming the user? What are the implications of such a balanced approach for users with sensory impairments? How will this shift impact the future trajectory of technology design, particularly in fields like virtual reality, gaming, and educational tools?

Victor’s ideas, although possibly viewed as idealistic, serve as a crucial reminder of the untapped potential in interaction design. They encourage designers and technologists to think beyond the current paradigms and explore innovative ways to make technology more human, not just in function but in experience. This reflection not only broadens our understanding of what interaction design could be but also inspires a more thoughtful and inclusive approach to technology development.

Concept:

Finding a concept for this assignment was complicated. We had a couple of cool ideas at the beginning, like using soda cans as drums, but unfortunately, their implementation didn’t go as planned.  We decided  in the end to use our favorite part of the kit: the ultrasonic sensor, and get a concept out of it.

Both our previous assignments used ultrasonic sensors as distance detectors. So we thought to base this assignment on the same concept.

We were inspired by the Accordion instrument when designing this musical instrument (with a twist):

 

With a plastic spiral, we mimicked the same movement of an accordion to produce different notes. The different notes are being produced by the distance between the hand and the sensor. We have assigned a different note to each range of distances. The instrument has a switch on the circuit to turn it on or off.

Here is a video of it in action to understand better:

IMG_8161-2

Code:

Again, assembling the circuit was a challenging task, but we managed to make it work! The code is straightforward and includes an ultrasonic sensor and a switch and loop function repeatedly used. The ultrasonic sensor to measure the distance to an object then converts this distance to centimeters based on the speed of sound. Depending on these values, we determine whether to play a musical note. If the distance is outside our defined range or the force is below our certain threshold, we make sure to stop playing the note. Otherwise, if enough force is applied then the distance is matched to the value in an array of musical notes based on distance and plays it using a piezo buzzer.

 

int trig = 10;
int echo = 11;
long duration;
long distance;
int force;

void setup() {
  pinMode(echo, INPUT);

  pinMode(trig, OUTPUT);

  Serial.begin(9600);
}

void loop() {
  digitalWrite(trig, LOW); //triggers on/off and then reads data
  delayMicroseconds(2);
  digitalWrite(trig, HIGH);
  delayMicroseconds(10);
  digitalWrite(trig, LOW);
  duration = pulseIn(echo, HIGH);
  distance = (duration / 2) * .0344;    //344 m/s = speed of sound. We're converting into cm



  int notes[7] = {261, 294, 329, 349, 392, 440, 494}; //Putting several notes in an array
  //          mid C  D   E   F   G   A   B

  force = analogRead(A0); //defining force as FSR data


  if (distance < 0 || distance > 50 || force < 100) { //if not presed and not in front

    noTone(12); //dont play music

  }

  else if ((force > 100)) {  //if pressed

    int sound = map(distance, 0, 50, 0, 6);  //map distance to the array of notes
    tone(12, notes[sound]);  //call a certain note depending on distance

  }


}

Reflection:

In future iteration i would love to make it look more like an accordion and also sound like it more, since for now the notes don’t sound as musical as I would like. However in the end i love this simple version of it that does the job really well and looks very cute.

Concept: 

For the final project, I decided to go with the running experience. For this project, I will use a potentiometer, wires, and a band to create a circuit that will receive data from the user (motion) and translate it into a visible visual to P5js. 

I have been trying to collect data and learn more about the potentiometer and how to make the project a reality. 

Arduino Design: 

For the circuit, I will connect one end of the potentiometer to 5v, another to GND, and the third to an analog pin and a band attached to the user.

In this case, the input is the Potentiometer Data, which presents the running speed, and the output will transfer the values to the computer for further manipulation. 

I will also try to make the band and the potentiometer appealing to the user and maximize safety while using. 

P5js Design:

I have to create a start button for users to start the experience. Then, I have to constantly update the 

canvas with the speed obtained from Arduino. Finally, I have to use the data to estimate the calories burned probably using the Harris-Benedict equation. 

https://www.omnicalculator.com/health/bmr-harris-benedict-equation

P5 will receive data from Arduino. Then it will visually display the results in real-time. I am still not sure if I will make a character that would run at the same speed as the user, or an object, or maybe translate the data into an artwork for the user to have. 

After the person is done they will have the amount of calories burnt during the the running time. This way the project would be helpful but also fun for the user. 

Concept

For my final project, I aim to create an engaging and personalized experience that will allow our students and workers to design their own NYUAD ID cards.

This project would address the issue of dissatisfaction with ID card photos. Individuals will be able to take their own photographs, through the feature of capturing their images in real-time.

Additionally, recognizing the importance of chosen names, I intend to include a space for users to input their preferred names, which will then appear on their ID cards.

To enhance individuality, the program will offer options to customize the photos with accessories like hats. People will also be able to choose their preferred animal to be displayed in the corner of the ID card.

This project seeks to not only address practical concerns but also provide a unique and enjoyable experience for users.

Design of the ID card (3 options) :

 

JJR (Jump Jump Revolution) is a remake of the famous video games series DDR (Dance Dance Revolution) but with a twist. This time, the user won’t dance on the platform, but will play a game instead. They won’t control a character in the game, they will be the character!

I want to create an interactive game utilizing a physical platform reminiscent of Dance Dance Revolution (DDR) for user control.

This project integrates Arduino for the physical interface and P5JS for the game itself. The user will navigate through the game by stepping on different tiles of a wooden platform, triggering corresponding actions in the game.

Psychic Hotline Fortune Teller

My final project will be a “fortune teller” machine, where users can inquire into 3 categories: relationships, the future, and yes/no questions. The fortune teller will resemble an landline telephone, where users can dial various phone numbers to receive different fortunes on a slot-machine-like cardboard display. The concept is basically that of a Magic 8 ball, but with more specific predictions and a “psychic” aesthetic.

This is how I visualize the project to work:

Another option is to make the display and phone interface separately, to allow for more flexibility. The phone could still be powered by the Adafruit Trellis keypad, or if I have the time and resources, Professor Shiloh’s rotary phone could be connected to Arduino to add a novel tactile experience for the user.

 

To fully utilize servo motors’ 180-degree rotation and the limited physical space I will have to write the fortunes, I was inspired by Professor Shiloh’s suggestion to have sentence beginnings and endings. These will mix and match to result in a total of 16 possible fortunes for “relationship” inquires, 16 possible fortunes for “future” inquires, and 4 answers to yes/no questions. I now realize that it might not be possible to have 4 answers, because for yes/no questions one of the rolls must be blank, to avoid the display showing “As I see it, most likely Yes, Definitely!”, which would be confusing. I used ChatGPT to help me generate some sentence beginnings and endings, and picked my favorites:

The input to Arduino will be the number sequence the user punches in on the keypad, and the output will be a randomized combination of 2 servo rotation angles to produce 1 sentence.

*Note: my project does not have P5 integration because Professor Shiloh approved an exception.