Author: Isabella Hernandez

Undergraduate Student Major: Visual Arts Minors: Interactive Media & Creative Writing

Final Project_Wild Rune

1. Concept + How it Works

Inspired by Season 2 of the Netflix series Arcane, this project offers an interactive experience that incorporating subtle references to the show. Found in the game page (state is playing)  is a colorful sphere,  floating in a dark space, inspired by the “Wild Rune”, introduced as a seed of destruction and chaos.. In the series, this is an unstable, magical object that is constantly distorting and causing chaos around it. For this reason, in my project, this sphere will be open to drastic changes. While it is in a constant state of rotating flux, the player must adjust its size, growing it and shrinking it, until it reaches the “ideal size” in order to fix the stability of the rune,  all while some of the most popular songs from the season’s soundtrack play.

Goal of the game: “Stabilize the Wild Rune”

2.  Images of project

Arcade box painted by myself
Arcade box with button and potentiometer. Final Project completed

3. Schematic

4. User Testing videos

User testing 1: https://youtu.be/KjYdtzcL7hw

User testing 2: https://youtu.be/kOqq3b2YJ9E

5. How does the implementation work?

5.1 Description of interaction design

  • The game begins with the Homepage, a combination of a screenshot taken from the series with illustrations made by myself. Once the players clicks on the lower “s” key on the laptop, they get access to the introduction page, which explains how the game works. Once they click “s” again, they are able to access the game playing state.The sphere begins at the runeSize, or the initial size. Since the sphere was set in a completely black background and the sphere was layered with a thick stroke, this gives the initial illusion that the sphere is not present. Once the user twists the potentiometer, they are able to make the sphere grow larger or smaller according to the potentiometer’s position. If the user grows or shrinks the sphere past the allowed sizes, a red LED turns on only as a warning.Once they find the “winning” size, the green LED flicks for a moment before the player is transferred to end screen and congratulated for their efforts. This page also contains important information regarding the following steps that must be taken, which is clicking on any key to return to the homepage, and returning the potentiometer back to the initial position so the next user can play.

5.2 Description of Arduino code + code snippets + add link to Github full code


Github: finalProject_Wild_Rune: https://github.com/imh9299-sudo/Intro-to-IM.git

ARDUINO UNO

In the Arduino Uno, the main interactions consist of sending information to p5 from the potentiometer, and the button. At the same time, it introduces essential information from the LEDs, such as their pin location in the Arduino board, or microcontroller board, the pinMode, and digitalWrite that allow P5 to understand how to proceed when each of the LEDs is on, off, and how they must begin the interaction once while(Serial.available()= is executed.

Other important snippets from the code include the “if statement” that contain information regarding the state of the button, whether it is off or on, and what was the last button state.

int lastButtonState = 0; // LOW or GND when pressed

int winLedPin = 7;
int loseLedPin = 4;

// lose = red LED
//win = green LED

void setup() { // setup code 

// Start serial communication so I can send data over the USB connection to my p5js sketch
  Serial.begin(9600); 
  pinMode(button1, INPUT_PULLUP);
  pinMode(LED_BUILTIN,OUTPUT);

  pinMode(winLedPin,OUTPUT);
  pinMode(loseLedPin,OUTPUT);

}

void loop() {

    while(Serial.available()){
      digitalWrite(LED_BUILTIN,HIGH);

      int win = Serial.parseInt();
      int lose = Serial.parseInt();

      if(Serial.read() == '\n'){

       digitalWrite(winLedPin, win);
        digitalWrite(loseLedPin, lose);
      
      
        // digitalWrite(loseLedPin, lose);
      int buttonState = digitalRead(button1);
     delay(5);
      if (buttonState == LOW && lastButtonState == 0) {
      lastButtonState = 1; // BY Gemini. To make sure the code doesn't miss when one presses the button
      
      }
      else if
        (buttonState == LOW && lastButtonState == 1){
          lastButtonState = 0;
        }

 

5.3 Description of p5.js code + code snippets + embedded sketch

Full Screen P5 Sketch: https://editor.p5js.org/imh9299/full/Jtg_IviWp

(Use google chrome)

While every line of code matters in programming and is an essential component that contributes to the success of a project, I found these blocks of code to be the most challenging yet crucial fragments of my game.

The first block shown below contains the functions, variables, and statements necessary for the game’s main objective. The “if statement” was built so that: 1) if the rune size  (sphere) was larger than 275, the red LED would turn on as a warning, as well was any value under 100. This function also connected with the Arduino, with the “lose” and “win” acting as the states of the LED. Meaning, if lose = 1, the red LED turns on, the green remains off, and will only change if the size is adjusted. On the other hand, if the player “wins”, the red LED remains off, and the green LED turns on, then turns off again when this is no longer true, or to be more specific, when the player is on the ending screen. Lastly, if neither of these statement are true, if the player is not winning or is not above or below the limit sizes, then the LEDs will remain off , otherwise known as the “safe zone”.

 

The second block was essential to receive the information from Arduino in order to make the button change the song playing upon clicking. By using an “if” statement contains the lastButtonState, playNextSong, and introducing the sections in which the songs would play (when “playing” mode is activated), this allowed for a smooth processing of data and making sure the songs would change only after clicking the button.

 

The third block is the initial foundation of the game, a Class made to design the sphere and facilitate its use in the main sketch. By implementing all the layers from the shaders in separate files (normalShader.frag, and normalShader.vert), and introducing all the lines of codes I learned from the tutorials, I made the sphere rotate on its own, added a large number to the strokeWeight of the lines around the sphere to give the illusion of multiple fragments making up the object, and set the size open to transform only when the player twists the potentiometer.

 

  if (rune.size > 375 || rune.size < 100) {
    lose = 1; // RED LED turns on
    win = 0; // GREEN LED is off

    let sendToArduino = win + "," + lose + "\n";
    port.write(sendToArduino);
  }

  // sphere the perfect size between 290 and 300 "win"
  else if (
    rune.size >= 295 &&
    rune.size <= 300 &&
    runeSize >= 295 &&
    runeSize <= 300
  ) {
    lose = 0; // RED LED is off
    win = 1; //GREEN LED turns on
    wonGame = true;

    let sendToArduino = win + "," + lose + "\n";
    port.write(sendToArduino);
    //print (sendToArduino);

    gameState = "end";
  }

  // Safe zone: not winning or losing
  else {
    lose = 0; // RED LED is off
    win = 0; // GREEN LED is off

    let sendToArduino = win + "," + lose + "\n";
    port.write(sendToArduino);
    //print (sendToArduino);
  }
} // END OF ELSE STATEMENT
let fromArduino = split(trim(data), ",");
      if (fromArduino.length == 2) {
        sensorValue = int(fromArduino[0]); // Keeps this updated always
        buttonState = int(fromArduino[1]);

        // play songs if state is in playing or in instructions
        if (
          buttonState === 1 &&
          lastButtonState === 0 &&
          (gameState === "playing" || gameState === "instructions")
        ) {
          playNextSong();
        }

        lastButtonState = buttonState;
      }
class WildRune {
  constructor() {
    this.size = 130;
  }
  
  display() {
    shader(normalShader);
    normalShader.setUniform("uBrightness", brightness);

    push();
    rotateX(angle);
    rotateY(angle);
    rotateZ(angle);


    stroke(0);
    strokeWeight(25);
    sphere(this.size);  
    pop();

    resetShader();
  }
}

 

 

P5 Embedded Sketch:

5.4 Description of communication between Arduino and p5.js

From Arduino to P5

  • The arcade button allows the users to change the songs played in the background while the “game state” is active. Found in an array, each song plays on infinite loop while it runs in the background, and will only change when the player clicks the button.
  • The Potentiometer allows the users to change the size of the sphere. After connecting Arduino UNO on p5, once the user presses “start”, the user is able to increment or decrease the size of the sphere by twisting the potentiometer.

From P5 to Arduino

  • If the player grows the sphere past a specific radius on the p5 canvas (above 375),  a red LED light will turn on.  Once the sphere shrinks back to the “SAFE ZONE”  or return to the home page, the LED will turn off.
  • Similarly, if the player shrinks the sphere past a specific radius on the canvas (below 100),  the red LED light will turn on.  Once they grow the sphere back to the “SAFE ZONE” or return to the home page, the LED will turn off.
  • When the player keeps the sphere at a certain size (between 295 and 300) for a few seconds, a green LED light turns on and takes the user to the “winning” page. The LED then turns off.

 6. What are some aspects of the project that you’re particularly proud of?

One of the main aspects of the project that I am proud of is the p5 to Arduino bidirectional communication. Once the game starts running, without the data of the potentiometer on Arduino to p5, adjusting the size with just a p5 slider would be far too simple. However, the potentiometer adds a whole new layer of value that engages the user through a hands-on practice. Only once this transfer of information was made, it was possible to make the LEDs react to what was happening to the sphere. I found this process fascinating, how an action happening on the screen triggered by an action on a physical, adjustable object (potentiometer), could cause a reaction back on another tangible object (the LEDs). Doing this offered me a glimpse to how the logic behind the arcade games I used to love as a kid  work, and reinforces the pride of being a human who can accomplish so much with a. few circuits and programming platforms. While it is not the most advanced game system, I am very happy with the outcome, and making this bidirectional exchange after having no previous experience in coding makes me feel proud of my progress,

 7. Links to resources used

Main Inspiration: Arcane Season 2 Clip
https://youtu.be/8PU2iKx0YtQ?si=qtSexmSyrLLJLWcc

Background Songs:

Coloring 3D

Spheres

Rotating 3D objects

Brightness and shades over sphere

Growing and shrinking 

Other student’s work inspiration:

8.  AI Tools Reference (How they were used and where)

 

function preload() {
//... rest of code
  normalShader = loadShader("normalShader.vert", "normalShader.frag"); //Allows lights, shaders, textures, in 3D objects.

}

///////////////////////////////////////////////////////////
function restartGame() {
//... rest of code

  sensorValue = 0;
  runeSize =130;

}

/////////////////////////////////////////////////////////////
function draw() {
  background(51);

  readArduino();
  //... rest of code

} // END of draw()

//------- Arduino + Sphere Game -----------------
function drawGame() {
  background(5);
  
  // resetMatrix(); by ChatGTP so that the arduino button will appear despite the differences of the WEBGL
  if (!port.opened()) {
    push();
    resetMatrix();
    //... rest of code

    
    }
  } //... rest of code
}
////////////////////////////////////////////////////////////
function drawHome() {
  resetShader(); // disable 3D shader so text can be drawn
  noLights();
  //runeSize=130;

  push(); // Drawing the fullscreen cover image
  translate(-width / 2, -height / 2); // move image to top-left for WEBGL
  translate(0, 0, -30);
}

////////////////////////////////////////////////////////////
function drawEndScreen() {
  if (port.opened()) {
    port.write("0,0\n");
  }

//... rest of code
  }
}

///////------ TO PLAY NEXT SONG----------------////////////////
function playNextSong() {
  // Stop all songs
  for (let s of songList) {
    if (s.isPlaying()) s.stop();
  }

  // Go to next track
  currentSong = (currentSong + 1) % songList.length;

  // Loop the new track
  songList[currentSong].loop();
}

//////////////////////////////////////////////////////

function readArduino() {
  //... rest of code
}

 

The use of ChatGTP and Gemini was used specifically to assist me organizing my code as I advanced in my project, fix a few problems I was facing that prevented the code from running or affected some aspect of the project, and adding a few lines of codes when there was no other resource that could fix an issue. 

 

Shaders for 3D Shapes

To create the colors shown in the sphere, originally I used this tutorial https://www.youtube.com/watch?v=GYFiu7nLTak as reference and from there I started to read all the instructions in the p5 3D object instructions,  and information about types of shaders. Initially I was going for a normalMaterial();  shader to create the colorful layer (check the references section to see all the links). However, I also wanted the sphere to have changes in its brightness to add more realism, just as the Wild Rune appears in the Arcane series. This is when I shifted my focus to the baseMaterialShader, since in one of the examples provided https://p5js.org/reference/p5/baseMaterialShader/ it contained all the properties I wanted for my own sphere. Nevertheless, I faced two problems with this:

  1. I wanted these elements from the shaders to apply in a Class, since I wanted the main focus on the sketch to be the Arduino and P5 exchange.
  2. Given that we had only learned how to work with 2D objects, layers, and other variables in class, I struggled to understand  how to apply the variables present in the baseMaterialShader into a smooth sphere, combined with the color patterns from the normalMaterial(); from the tutorial.

This is why I asked ChatGTP to help me combine the foundation functions for the sphere, and the shaders and integrate them into Classes (normalShader.vert) and (normalShader.frag). Basically, the codes in normalShader.frag are elements from GLSL fundamentals, introduced in the Introduction to Shaders of p5. By following the instructions on how to adjust the properties of brightness over the sphere,   this Class simplifies some of the fundamental lines of code from the Shaders, such as precision mediump float; which is needed to calculate the values of the floating points, varying vec3 vNormal; which calculates the color in this case, the Uniform makes the values the same for every pixel, and the uBrightness naturally controls the brightness cast on the sphere.

And for normalShader.vert, this Class focuses more on the general aspects of the sphere, with the aPosition focusing on the position of every vertex in the 3D space in x,y, and z positions, the aNormal which focuses on direction, and the attribute that assigns each vertex its own value.

Winning State and Resetting Sphere Values

After I achieved to make every state of the game function effectively (Homepage, Instructions, drawGame, and end page/winning), I realized a new problem had emerged, one that took me a long time to resolve. After growing the sphere to the “winning” size and ending the game, I noticed that every time I wanted to start a new round, the sphere wouldn’t return to its original size, even after returning the potentiometer to the starting position. This caused the sphere to lock itself on the winning position, and no matter how many times I was taken back to the homepage to restart the game, I would be immediately “win”, unable to change the size of the sphere again. After much trial and error, this was the process towards solving this issue.

  1. In the restartGame() function, Gemini suggested to add the  sensorValue = 0; runeSize =130; } . With the sensorValue= 0; and the runeSize = 130; the sphere would return back to the original size for the next player, and ensure the value of the potentiometer was at 0 as well.
  2. It also suggested to add the function   if (port.opened()) { port.write(“0,0\n”); under drawEndScreen() to make sure the port to the Arduino was functioning and reading the new values after the game ended.
  3. When these changes didn’t work, Gemini pointed out that the Arduino code was only running during the drawGame (the state of the game is “playing”). This meant that the Arduino was no longer reading the new values and was stuck in the last position the sphere was on. For this reason, I added the function readArduino(), containing the lines of code in which P5  would read the information from Arduino UNO that we learned in class: (if (port.opened()) { let data = port.readUntil(“\n”); ….etc).

After this, I wrote down readArduino(); on top of the data in the draw() function so that it would read the information at any state of the experience:

readArduino();

if (gameState == “start”) {
drawHome();
// rest of code

Fortunately this allowed p5 to reset the game’s data and fix the original problem.

Text on WEBGL Canvas

After learning how to create a sphere and how to hold it in a WEBGL as part of the canvas size, I discovered that the text that I had for my home page was not showing, nor the button to connect p5 with Arduino. Despite watching numerous tutorials and doing research on similar 3D works, I couldn’t find the reason of why the text wouldn’t appear. And without understanding what was happening, I couldn’t search for a solution. After reading every line of code and analyzing its effects on the game, I learned thanks to ChatGTP that the text was designed to be layered on a 2D plane, and not a 3D plane which was only meant to sustain the sphere and other 3D objects. By using the resetShader();  and the noLights(); and translating the background images with negative values, I was able to fix the layer for the introduction, the endGame, and the HomePage so that the text would show, all without affecting the “playingState”. These functions allowed to: 1) Reset the shader for the sphere so that it wouldn’t affect the previously mentioned pages, 2) translate the background image behind the text, and adapt the data in spite of using WEBGL.

Changing Songs with Arduino Button

Lastly, after compiling all the songs that would play, while the game was running, into an array (songList), and following the basic structure to play a sound file (https://p5js.org/reference/p5.sound/p5.SoundFile/), my objective was to have this list of songs ready to play, so that every time the user clicked on the Arduino button, they would be able to change the song. However, doing this required a complicated logic.

  1. The songs had to play in an infinite loop, so that every time the song playing ended, it would reset itself and play from the beginning.
  2. The moment the player clicked the button, the old song would stop playing
  3. After the old song stopped playing, the new song would start playing on loop.
  4. All of this had to be executed through the Arduino button, and not through p5

So I followed the instructions from the p5 sound loop https://p5js.org/reference/p5.sound/p5.SoundLoop/?utm_source=chatgpt.com and adjusted it accordingly in the lines:

function playNextSong()

{ // Stop all songs for (let s of songList)

{ if (s.isPlaying()) s.stop(); }

// Go to next track currentSong = (currentSong + 1) % songList.length; // Loop the new track songList[currentSong].loop(); }

And Chat GTP helped me understand the logic behind each line. First, I had to create a function (playNextSong) to play each song, stop the current song if the user clicked the button, look back to the song list and pick the next song in line, play  the song and make sure the old song is no longer plating. Once the current song was playing, this would go on a loop.

After doing this, it was just matter of introducing this function into the gameState “playing” so that it would run while the game was running and everything was reset when the game was set to restart, as shown in the block of code below inside fuctionKeyPressed()

 

function keyPressed()

{ //... rest of code

// Start first song on loop currentSong = 0;

songList[currentSong].loop(); }

else if (gameState === "end") { restartGame();

// stop ALL songs for (let s of songList)

{ if (s.isPlaying()) s.stop(); } } }

///////------ TO PLAY NEXT SONG----------------//////////////// function playNextSong() {

 

9. Challenges faced and how I overcome them

As previously mentioned, the main challenges I faced were understanding the the logic of Shaders for 3D Shapes, how to fix the Winning State and Resetting Sphere Values, adding text on a WEBGL Canvas, and changing the songs with the Arduino Button.  Every time I encountered an obstacle, I tried to first resort to tutorials or other sources in the internet to find a solution. If this didn’t work, I would ask my peers or approach Professor Aya. While I wish I hadn’t resort to using AI platforms, when I did, I made sure that they would always provide long explanations, step by step, and simplify the information so I could understand the logic behind every line of code.


WEBGL ChatGTP aidSongs loop aid from Gemini

Shaders aid from ChatGTP

Shaders 2 aid from ChatGTP

LEDs and P5 to Arduino Exchange

Another challenge I struggled with was with the p5 to Arduino interaction. Initially the intention was for a red LED to turn on when the sphere grew past the size of 400 or under 50. However, despite following the template we learned in class to send the information of the LEDs to p5 on Arduino UNO and applying it to work with my “winning” and “losing” format, the LED’s were not turning on once I inserted them inside the code in which p5 received the information from Arduino.

// Start serial communication so I can send data over the USB connection to my p5js sketch
  Serial.begin(9600); 
  pinMode(button1, INPUT_PULLUP);
  pinMode(LED_BUILTIN,OUTPUT);

  pinMode(winLedPin,OUTPUT);
  pinMode(loseLedPin,OUTPUT);
 
}

void loop() {

    while(Serial.available()){
      digitalWrite(LED_BUILTIN,HIGH);

      int win = Serial.parseInt();
      int lose = Serial.parseInt();

After a long session with Professor Aya, however, I finally understood my mistake.

In p5, I set the minimum size of the sphere to be 50, and the maximum size to be 400. However, my bidirectional exchange was stating that the red LEDs would turn on only if the sphere grew under and above these sizes. Naturally it didn’t make sense for the maximum size to also be the size which the sphere could not grow above of. After realizing the mistake in my logic, we changed the maximum size to be 700, and kept the minimum as 50. On the P5 and Arduino exchange, however, we changed the data so that if the sphere was instead above of 375 or under 100, the red LED would turn on. And the green LED would only turn on when the sphere reached a size between 295 and 300.  Once this was fixed along smaller adjustments in Arduino UNO, the program ran successfully.

 

10. What are some areas for future improvement?

For future improvements, I would explore adding more interactive elements to the game, as the current version feels too simple for my liking. While I did integrate music into the project, one suggestion I received and would definitely implement with more time is the addition of sound effects that indicate whether the player is getting closer to or farther from the correct size, similar to the “warm and cold” childhood games. This would not only guide the player more effectively, but would also make the experience more engaging. I would also consider adding more layers of complexity, such as different difficulty levels or challenges where the player must find not only the correct size but also the correct rotation speed of the sphere. These variations could be presented as different rounds, helping to keep players more engaged over time. Another feature that could enhance the experience is a time-based element, where players attempt to beat the shortest completion time. At the end of the game, all records could be preserved and checked for future interest. Finally, I would like to introduce more interactive controls that allow the player to change additional aspects of the sphere, such as its rotation speed and color, in a way that complements the game’s overall aesthetic.

11. IM show documentation, images, videos, people interacting with your project

IM Showcase “Wild Rune” video: https://youtu.be/fethsawtmDY

IM Showcase “Wild Rune” video part 2: https://youtu.be/057bjR7LJNM

Conducting User Testing Week 14

User testing Video
Have people try your project without giving them any
prompts/instructions and see how they use it

https://youtu.be/KjYdtzcL7hw

Sidenote: I advice to lower the volume of the video since there were people being loud nearby

• Are they able to figure it out? Where do they get confused and
why?

At first, I forgot to mention in the instructions page what the potentiometer was, which is why the person who tried my game couldn’t figure out what was the tool that controlled the size of the sphere. After this, although it didn’t take her long to understand how the potentiometer worked, she commented that it would be helpful to have a small description of how it is used.

Do they understand the mapping between the controls and
what happens in the experience?

Fortunately, they did understand what the potentiometer was doing to the sphere and expressed her interest in the interaction.

• What parts of the experience are working well? What areas could be improved?

Although so far everything in my sketch and Arduino works as I wanted

  • The sphere shrinks and grows in response to the potentiometer
  • the songs change in response to the button
  • the red LED turns on when the sphere reaches the maximum or minimum size
  • the green LED turns on when the sphere reaches a small range (between 290 and 300)
  • the “winning” size triggers the end page and then the user can go back to the homepage
  • there is a homepage and an introduction page

I feel like I could add a last component to make it more interactive, which would be the spinning of the sphere with another potentiometer. I think I could also make the growth and shrinking of the sphere far more smoother, and maybe add some exciting element for the winning page. While I am aware that on the surface this interactive project seems simple, it has forced me to push myself far outside my comfort zone, and I am happy with my progress so far.

• What parts of your project did you feel the need to explain?

Like I previously mentioned, I realized I failed to explain in the instructions how the potentiometer works and how it can be identified by those who don’t know what it is. I also realized I need to explain what happens when the spheres reaches the incorrect sizes that trigger the red LEDs and what triggers the “win”.

  • How could you make these areas more clear to someone that isexperiencing your project for the first time?

I believe it would be best to introduce this information in the instructions page since this is the most efficient way to inform the user while also keeping their focus on the information before splitting their attention to the p5 sketch/screen  and the Arcade/arduino game box.

Wild Runes – Final Project Proposal

Concept + How it Works

Inspired by Season 2 of the Netflix series Arcane, this project prioritize the experience of the concept while incorporating subtle interactive responses. On the first page, there will be a brief instruction of how the experience works. In simple terms, there will be an animated object floating in the screen once the user clicks one of the keys in the computer. This floating object, inspired by the “Wild Rune” introduced as a seed of destruction and chaos, is a colorful sphere floating in a dark space. In the series, this is an unstable, magical object that is constantly distorting and causing chaos around it. For this reason, in my project, this sphere will be open to drastic changes: From changes in size (growing and shrinking), to changes in speed, all while some of the most popular songs from the season’s soundtrack play.

Goal of the game: “Stabilize the Wild Rune”

From Arduino to P5

For the interaction,  I aim to have p5 respond to two potentiometers and one button.

  • Button 1 will allow the users to change the songs played in the background while the “game state” is active. After doing an array, each song will play on infinite loop while it runs in the background, and will only change when the player presses the button.
soundFormats("mp3", "ogg"); // Loading the sound file.
songList = [
  loadSound("Woodkid_To_Ashes_and_Blood.mp3"),
  loadSound("Blood_Sweat_&_Tears.mp3"),
  loadSound("Imagine Dragons_Enemy.mp3"),
  loadSound("Stromae, Pomme_Ma_Meilleure_Ennemie.mp3"),

 

  • Potentiometer 1 will allow the users to change the size of the sphere. After connecting Arduino UNO on p5, once the user presses “start” they will be able to increment or decrease the size of the sphere by twisting the potentiometer. (As shown in the VIDEO)
  • Potentiometer 2 will allow the user to change the speed at which the sphere is rotating. Just as the previous analog sensor, the second potentiometer will be accessible as soon as the Arduino UNO is connected to the p5.
// -------------------------
   // READ FROM ARDUINO
   // -------------------------
   let data = port.readUntil("\n");
   data = trim(data);

   if (data.length > 0) {
     // ---- POTENTIOMETER DATA ----
     if (data.startsWith("P:")) {
       let potValue = int(data.substring(2));
       if (!isNaN(potValue)) {
         sensorValue = potValue;
       }
     }

     // ---- BUTTON PRESSED ----
     else if (data === "BTN" && gameState === "playing") {
       playNextSong();
     }
   }
 }
   // Smoothly transition sphere size
   rune.size = lerp(rune.size, runeSize, 0.1);

   // Draw the sphere
   rune.display();

 

From P5 to Arduino

  • If the player grows the sphere past a specific radius on the canvas,  a red LED light will turn on.  Once they fix the size or return to the home page, the LED will turn off.
  • Similarly, if the player shrinks the sphere past a specific radius on the canvas,  a red LED light will turn on.  Once they fix the size or return to the home page, the LED will turn off.
  • When the player keeps the sphere at a certain size, a green LED light will turn on and take the user to the “winning” page. The LED will turn off.

Background Songs:

Imagine Dragons feat. JID – “Enemy (Opening Title Version)” (from Arcane) [Official Visualizer]

https://youtu.be/UqcE-IIevf0?si=1fEAqWTDhATXA7Av 

Woodkid – “To Ashes and Blood” (from Arcane Season 2) [Official Visualizer]

https://youtu.be/Gj-jmBi0aK8?si=6Yn53yDESy4y0R0K

Blood Sweat & Tears (from the series Arcane League of Legends)

https://youtu.be/ySbgRwQi3Rc?si=uf5mUQyy-rPUKWLz 

 

Objectives:

  • Add button to change music (in process)
  • Add potentiometer to adjust speed of spinning sphere
  • Fix Title (must show when the game begins (it currently doesn’t show since the text is used to showing on a 2D layer)
  • Make the size (too big and too small) cause RED LED to show
  • Make the right size cause the user to win (show GREEN LED)

Tasks completed so far:

  • Created a sphere in a WEBGL space for the three dimensional elements to work (shades, colors, among others)
  • Added potentiometer from p5 to Arduino to successfully grow and shrink the sphere
  • Defined a structure of p4 including: Introduction page, “playing state”, “winning page”.
  • Established the functions necessary for Arduino and p5 to exchange more than one piece of information (potentiometer + button 1)
  • Downloaded and uploaded GitHub files to p5 , learned in class, to stabilize bidirectional serial communication

VIDEO

Main Inspiration: Arcane Season 2 Clip 

https://youtu.be/8PU2iKx0YtQ?si=qtSexmSyrLLJLWcc

Chat GTP

Given the complexity of this project, I have used Chat GTP so far to assist me in organizing the code and guiding me step by step when something is missing or a mistake is preventing the code from running. For example, I struggled to make both the button and the potentiometer work at the same time, so I explained my situation to Chat GTP and it made me realize the error was in my code where the Arduino exchanges information with p5. It also helped me realize I hadn’t uploaded the files from Github, causing the port connection to malfunction.

Week 11 – Serial Communication With Shota

With Shota Matsumoto
EXERCISE 01: ARDUINO TO P5 COMMUNICATION

Concept

For the first exercise we used a photoresistor as an analog sensor on Arduino to make our ellipse shape move on the horizontal axis in p5. For our code, we adjust it so that with every touch on the photoresistor, the ellipse would move on the canvas.

VIDEO:

https://drive.google.com/file/d/12b1KzwTlo1IQLtDfleAJNJccZhL7J0Tb/view?usp=sharing

Code we’re proud of:

When it comes to the first exercise, the part we are most proud of is the block of code that controls the x-position of the ellipse using the light sensor value from the Arduino. We read the sensor value through the serial port that is connected to the Arduino and map it from 0 to the width of the canvas so that it never goes out of the dimension of the screen.

 

//read the data coming to the port until the newline character
   let data = port.readUntil("\n");
   //if there is a data
   if (data.length > 0) {
     //convert the clean data into integer
     rValue = int(trim(data));
     //store data of left and right into msg
     let msg = left + "," + right + "\n";
     //send the msg back to the Arduino
     port.write(msg);
   }
   text("rValue = " + rValue, 20, 50);
 }
 //move xpos depending on the sensor value (max = 640)
 let xpos = map(rValue, 550, 900, 0, width);
 fill(0, 0, 255);
 ellipse(xpos, height / 2, 100, 50);

 

EXERCISE 02: P5 TO ARDUINO COMMUNICATION
Make something that controls the LED brightness from p5.

Concept

For the second exercise we used the createSlide function to create a slider on the canvas of p5. This way, every time we moved the slider to the right, the brightness of the LED on the Arduino would increase, and every time the slider was moved to the left, the brightness would decrease.

VIDEO:

https://drive.google.com/file/d/12b1KzwTlo1IQLtDfleAJNJccZhL7J0Tb/view?usp=sharing

Code we’re proud of:

For the second exercise, this block of code combines what we learned during the class about p5.js (createSlider) and Arduino. Since we remembered using a slider to control a value or game score, we decided to use it to control the brightness of the LED. We take the slider value, store it in a variable called brightness, and send it to the Arduino through the serial port, as shown in the block of the code below.

//read the slider value
 brightnessValue = brightnessSlider.value();
 //if the port is open
 if (port.opened()) {
   //store the brightness value from the slider
   let msg = brightnessValue + "\n";
   //send it to Arduino
   port.write(msg);
 }

 

EXERCISE 03: BI-DIRECTIONAL COMMUNICATION

Concept

For the third exercise, we made use of a potentiometer ( analog sensor ) on Arduino and a LED light, connecting it on the corresponding pins and then connecting Arduino UNO with P5. After adding the necessary codes on P5, we adjusted our “ball” so that every time it bounced against the floor, the LED would light up, and once it stopped bouncing the LED turned off. With the potentiometer, we were able to control the “wind”, causing the ball to be pushed from side to side, depending on the direction in which we turned the potentiometer.

VIDEO:

https://drive.google.com/file/d/1YA0d_xkwM2zxTco6mqnZNbEIr9Ou60oq/view?usp=sharing

Code we’re proud of:

For the third exercise, we’re proud of the block of code below. It basically captures the main objective of this exercise. When the ball reaches the ground and the LED state is 0, meaning the LED is currently off, we turn it on by sending a 1 to the Arduino through the serial port. It is simple but we needed to understand the basics of what is going on in the vectors.

//if the ball reached the floor and the led is currently off
 if (isBounced && ledStatus === 0) {
   //turn the LED on
   port.write("1");
   //set the state to 1
   ledStatus = 1;
 }
 else if (!isBounced && ledStatus === 1) {
   //turn the led off
   port.write("0");
   ledState = 0;
 }

 

Github

Links:

exercise1 Arduino

exercise1 p5

exercise2 arduino

exercise2 p5

exercise3 arduino

exercise3 p5

Challenges and Further Improvements

Fortunately, we were able to complete all the assignments successfully, and managed to achieve all the necessary responses of P5 and Arduino. For each exercise, we started by planning out which tools we would use, how to arrange them on our breadboard, and once this was done, our greatest challenges consisted of arranging and adding the codes required to make each response function. On the third assignment we especially struggled writing the codes required to make the potentiometer affect the “wind” and consequently the direction in which the ball was pushed. For future improvements, we would like to grasp a better understanding of the exchanges between P5 and Arduino in order to make more complex and interactive responses that can also satisfy the assignment’s requirements.

 

References:

We used ChatGPT to help us understand the bouncing ball template code since we hadn’t yet learned about the vector library in class. After understanding the foundation of it, we were able to integrate the isBounced logic into the template code to detect whether the ball was bouncing or not. Thus, we’re glad we were able to learn a lot about vectors library. We also used ChatGPT to learn about DOM library methods such as .position() and .style(). Although we had covered the basics of createSlider, we didn’t delve deeper into it during the class, so we needed to learn how to adjust its position and CSS styling of the ball.

 

Final Project- Preliminary Concept Blogpost

Concept

One of my favorite shows of these two last years has been Arcane. The animation, the character designs, the music,  and the story are all beautiful elements combined into a masterpiece. For this reason, for my final project I want to create a reflection of my emotions towards these series.  My audience is intended for those who are fans of the series, but also those who have not watched it yet, and encourage them to do so. Moreover, I also intend to enhance a user experience in which the participant can absorb the parallel between music and visuals, demonstrating the power of these two elements when they are combined in an interactive medium.

“Stabilize the Wild Rune”

How does the project work 

My intentions are to make this project prioritize the experience of the concept while incorporating subtle interactive responses. On the first page, there will be a brief instruction of how the experience works. In simple terms, there will be an animated object floating in the screen once the user clicks one of the keys in the computer. This floating object, inspired by the “Wild Rune” introduced in Season 2 of Arcane, is a colorful sphere floating in a dark space. In the series, this is an unstable, magical object that is constantly distorting and causing chaos around it. For this reason, in my project, this sphere will be open to drastic changes: From changes in size (growing and shrinking).

From Arduino to P5

For the interaction,  I aim to make a series of buttons (three maximum), as part of the Arduino tactile “speaking”.

  • One of the buttons would be to play the music (if this doesn’t work, I would make it the start button instead)
  • To grow the sphere I would make a second button
  • To decrease size I would make a third button.

(Maybe Instead of buttons I could use a slider)

From P5 to Arduino

  • When the player grows the sphere past a certain point they lose and a RED LED light will turn on and turn off as they return to the home page.
  • When the player decreases the sphere past a certain point they lose and a RED LED light will turn on and turn off as they return to the home page.
  • When the player keeps the sphere at a certain size for a specific amount of time (for example 5 seconds), a GREEN LED light will turn on and turn off as they return to the home page.

sidetone: Maybe I give them candy if they win

If there is music, this one would play on its own and reset when it is done playing.

Background Song: (still in progress of decision)

  • To Ashes and Blood – Arcane, Woodkid

Questions to ask:

  • How can I make the buttons affect the size of the object?
  • Can I use the songs or not (because of copyright)
  • How can I start at a different size in the canvas
  • Will there be an end of game or not? If there is, how can I do it?
      • (For this scenario, I would probably follow the storyline where it is necessary to stabilize the wild rune, so if the user makes the sphere too big or too small it would be game over).

Fullscreen Mode Sketch 

Link to sketch: https://editor.p5js.org/imh9299/full/lv73qdahm

Main Inspiration: Arcane Season 2 Clip 

https://youtu.be/8PU2iKx0YtQ?si=qtSexmSyrLLJLWcc

 

Other student’s work inspiration:
https://intro.nyuadim.com/2025/05/08/week-14-final-project-2/ 

 

Reading reflection Week 11

Design Meets Disability

After reading this text,  I found myself agreeing with the author’s argument on fashion and designing products for disabilities. I’ve always been curious about the process behind designs for products meant to support people with disabilities, such as prosthetics, hearing aids, and glasses. Although at the beginning fashion was described as a form of reducing the size and enhancing the aesthetic aspect of products such as glasses, my thoughts about the author’s argument changed. But as I read further, I realized that reducing the size and making a product “fashionable” is not just about making it pretty, but also satisfying  the comfort of the client, and respecting their experience. 

The text reinforced some ideas I already had, especially the belief that good design is often thoughtful for problem-solving. I connected strongly with Charles Eames’ idea that design is guided by constraints, because it reminded me that although sometimes there will be limitations, this is just a first step towards understanding what the client wants and needs. At the same time, the reading challenged me to think about other  perspectives I thought about before, such as how making a design for disability discreet might unintentionally send a message that it should be hidden. Thus, it is important to always prioritize the comfort of the client, and take these different perspectives into consideration. 



Week 10 – Musical instrument (Group Project with Abdelrahman)

Concept

Mini Piano

Given that we had to make a musical instrument, we were inspired by a project we found in YouTube of what seemed like a piano. From this, we aimed to make a smaller scale of this project following the requirements for the assignment. The circuit consists of one piezo speaker, a potentiometer as an analog sensor, and a three buttons as digital sensors.

Schematic

Circuit Diagram

Figure 2 ( Circuit design with code and simulation on “Magnificent Jaiks” by Abdelrahman

 

Final Results

VIDEO

Arduino code

One aspect of this project we were  proud of is the octave multiplier implementation. Instead of having fixed notes, Abdelrahman used the potentiometer to create a continuous pitch control that multiplies the base frequencies by a factor between 0.5x and 2.0x.

float octaveMultiplier = map(potValue, 0, 1023, 50, 200) / 100.0;

This line of code transforms a basic three-note instrument into something much more expressive and fun to play with. Users can play the same melody in different registers, which makes the instrument feel more like a real musical tool rather than just a tech demo.

else if (digitalRead(button2) == LOW) {
  int freq = notes[1] * octaveMultiplier;
  tone(buzzerPin, freq);
  Serial.print("Playing ");
  Serial.print(noteNames[1]);
  Serial.print(" at ");
  Serial.print(freq);
  Serial.println(" Hz");
}
else if (digitalRead(button3) == LOW) {
  int freq = notes[2] * octaveMultiplier;
  tone(buzzerPin, freq);
  Serial.print("Playing ");
  Serial.print(noteNames[2]);
  Serial.print(" at ");
  Serial.print(freq);
  Serial.println(" Hz");
}

I also thought this snippet was interesting since it captures how the notes are played and how the frequencies adjust when the buttons are played and the potentiometer is twisted.

Github

https://github.com/imh9299-sudo/Intro-to-IM.git

Challenges and Further Improvements

Despite the complexity of the circuit, Abdelrahman successfully managed to produce a design in Magnificent Jaiks which I managed to follow without any issues. After arranging the jumper wires and all the other pieces, the final result was as we aimed for. One of the biggest challenges we faced was finding a way to divide the work as a result of the distance. Nevertheless, we managed to make a plan after meeting on zoom, and finished our circuit on time. For future projects, one thing I would like to improve on is developing an instrument of this kind with more buttons in order to have a larger version of the mini piano done for this assignment.

Reading Reflection Week 10

One of the examples I found most interesting in “A Brief Rant on the Future of Interaction Design” was the author’s take on voice, especially when he described a fictional situation in which Monet uses his voice to obtain the painting he wants. It may sound ridiculous but I really do agree and understand his argument, how active exploration is necessary in some cases, and the use of voice doesn’t apply. This challenged my own way of thinking, expanding to something essential that the author may be trying to express, whether consciously or not. This is, some forms of interactive will not apply to some cases, and this is why the designer or programmer must understand what is most relevant and effective for the experience they are making. For example, in an interactive experience designed to be engaged while standing up, it might not be the best idea to make certain responses triggered by having the user go click on something on the laptop every time as well. Allowing the user to focus on a few things at a time will improve their experience and help them understand the relevance of the interactive elements within the main objective of the project

 

Something that caught my attention in Bret Victor’s follow-up article was the neglected factor when it comes to tools that are meant to amplify human capabilities, and how hands play an important role in these. I think I speak for many when I say that we often take the skills of our hands for granted. Cooking, drawing, writing, building, organizing, some of the best activities done both for pleasure or to complete a task are done with our hands. And I agree with the author that there is a unique satisfaction that comes from doing these things with our hands. For example, I do love drawing digitally, as the result of my efforts match my intention to draw realistic yet slightly cartoonish objects, landscapes, and characters. However, this process doesn’t match the precision and delicacy of using a brush and applying paint to a canvas, stroke for stroke.

Despite this, I must disagree with the author on one thing. Even though the connection between what I am doing with my hands and the outcome on a screen might not be as strong as doing with another more tangible medium, this does not mean there is no connection at all. Before technology took over most of our ordinary activities such as writing, I enjoyed writing stories with a pen or pencil on my notebook. However, this process became exhausting whenever I had to erase or cross something, or my fingers started to ache from writing for so long. Now, even though I am staring at a screen instead of brushing my fingers over a piece of paper, and typing over a keyboard instead of using a pen, I still enjoy writing just as much as before, and the action has become more manageable for me.

 

Week 9: Analog sensor and Digital sensor

Concept

For this project, to create an analog sensor and one digital sensor (switch), and use this information to control at least two LEDs, I decided to make for the digital sensor a connection between a digital pin and a button to control one LED light. For the analog sensor I used the analog input and output to control the second LED light, turning it on and off depending on how much brightness the light sensor received.

Circuit Illustration
Figure 1

Final Results

oplus_0
oplus_0

VIDEO

Arduino code
oid loop() {

int sensorValue = analogRead(A3);
Serial.println(sensorValue);
// set the brightness of pin 11
sensorValue = constrain(sensorValue,500,900);
brightness = map(sensorValue, 500, 900, 0, 255);
analogWrite(11,sensorValue);
delay(30);


  int buttonState = digitalRead (A2);
  if (buttonState == HIGH){
    digitalWrite(13,LOW);
  } else{
      digitalWrite(13,HIGH);
  }

Although it isn’t the most creative or unique code, I was happy to find that my initial code was working to make both LED lights light up, and it was just a matter of adding information for the sensorValue and the brightness to make the LED light responding to the light sensor to show a greater difference in the changes of brightness.

Github

https://github.com/imh9299-sudo/Intro-to-IM.git

Challenges and Further Improvements

While I was able to successfully complete the assignment and create both digital and analog responses, I wish I could have found a more creative way of obtaining these responses. Moreover, I struggled to put the code together so both sensors would function simultaneously, as well as arranging the position of all the jumper wires, the resistors, and the LED lights themselves.  For my next project, I will seek help from the Lab assistants, and I will pay more attention to all the information on the slides we are provided in class to make sure I fully grasp the foundation behind every code and understand what each tool does. I would also like to explore the other tools found in the lab to make different responses in comparison to the tools we already have in our Arduino kit.

Week 9 Reading Reflection

,

In “Making Interactive Art: Set the Stage, Then Shut Up and Listen”,  the author emphasizes the importance of not defining an interactive work, since unlike a traditional art piece, an interactive piece is not just an expression but a medium that will inevitably be interpreted differently by the users. I found myself agreeing with this statement, especially since it does apply to traditional art in some cases as well. There is something exciting about being the viewer or the user, and being able to not only produce our own interpretations of a creative piece, but also being able to engage with it, whether it is through a computer program or something tangible. It also reminds me of when I write creative stories; sometimes it is better to leave certain details unsaid rather than reveal everything about a setting or a character, as readers often enjoy filling in those gaps with their imagination. Ultimately, like any creative work, I agree with the author that an interactive piece can resemble a performance more than a finished painting or sculpture, allowing something beautiful and engaging to emerge. 

Similarly, in “Physical Computing’s Greatest Hits (and Misses)”, I found it interesting how the author notes that when people learning about physical computing realize a particular ideas has been done before, they often give up on it because they believe it’s not worth it since it is no longer original. My first thought was that this happens in art and writing as well. When a writer wants to explore a familiar theme, like reincarnation, they might hesitate because it has been done many times before. Before reading this text, I hadn’t realized that the struggle for originality and authenticity is just as applicable in physical computing as it is in other creative or noncreative fields. Now I feel like I share much more in common with this field than I initially imagined.