Author: Ting-An Yang

Concept

The game is inspired by the history of Taiwanese rock, which developed not as a direct product of American cultural imperialism but as a manifestation of our people’s desire to preserve our cultural identity through these modern forms of music; many Taiwanese rock bands write their lyrics in Taiwanese Hokkien and are very involved in social movements outside of their music. These are elements of Taiwanese culture that were suppressed during the martial law era, and the vibrant pop punk/punk rock scene embodies the resilience of our people. Taiwanese rock is an amalgamation of local Taiwanese, Chinese, American, and Japanese influences that reflects our complex history as a colonial subject of many but also our present and future as a country.

The game’s title Formosa! refers to the name “Ilha Formosa,” which was given to Taiwan by Portuguese sailors in the 16th century, meaning “beautiful island.” The song that plays throughout the game is “人間條件 Human Condition” by Fire EX., a band that I consider a symbol of Taiwanese resilience due to their outspoken support of Taiwanese independence and other causes.

I intend for the game to be a light-hearted introduction to this aspect of Taiwan. In the spirit of the wacky, energetic aesthetic of the Scott Pilgrim franchise, the player controls a guitarist who is running late for a gig and has to get there as soon as possible, evading cops and picking up pirated music tapes to gain experience and recharge. My final project is a game centered around a guitar repurposed as a gamepad: each string, when strummed, triggers a specific movement for the protagonist.

How the game works

• 3 of the guitar’s frets are covered with red strips. These are buttons that players press to either start, jump or attack.
• Players will have to jump to collect power-ups in the form of vinyls and cassette tapes in order to build up an attack charge (red bar on the top right corner of the screen). The maximum charge value is three.
• When the attack charge is greater than zero, players can attack to destroy enemies (cops and cop cars). Cops take one attack to destroy while cars take two.
• When an enemy is destroyed, the score goes up. Contact with enemies will cause the score to go down.
• Each game lasts 60 seconds. The game automatically ends when the countdown timer reaches zero.

Arduino + p5.js Setup

When players press on a “button” (the red strips on the guitar neck), a circuit is closed between the metal guitar fret and its strings, triggering in-game actions such as jumping and attacking depending on which button is pressed.

The Arduino code simply takes input from the digital pins connected to the three guitar frets; the p5.js code takes this input (0 or 1 depending on whether the circuits are closed or not) and translates it into triggers for character actions.

Arduino code on Github 

The p5.js code has a separate function for each game state — start, instructions, main game and end screen — that is called on in an if-else statement in the main draw function.

The properties of characters and power-ups are set in their individual classes. The hit(entity) function in the Protag class set conditions that are then used to check if the Protag has come into contact with enemies or power-up collectibles in the handleCollisions() and handleAttacks() functions of the main sketch.

Problems Encountered + Solutions

Personally, I’m proud of managing to successfully set up the serial connection between p5.js and Arduino. For a good amount of time, the connection wasn’t working, and I was panicking until I realized that p5.js wasn’t responding as it should be because in all the “if” statements for each in-game action, I put the button state “1” as a string instead of an integer, as they were specified as in the readSerial() function. After simply removing the quotation marks, the serial connection was up and running successfully.

As for the hardware, I also had to figure out new solutions after initial configurations didn’t work as I expected. While i had originally planned to attach 4 wires to individual guitar strings and use a wired conductive guitar pick to create a controller in which strumming each string triggers a different in-game action, this failed to work as all the strings were connected to a metal saddle, which made all the strings one conductive body and therefore could not be utilized for individual actions. I ultimately decided to turn the guitar’s metal frets into the buttons of the controller. I covered three separate frets with copper tape for enhanced visibility and connected them to wires, doing the same for the metal saddle; when the player presses any string against the taped frets, a circuit is completed and a signal is sent from arduino to p5js to trigger 3 actions — start, jump, and attack — depending on which fret is pressed.

Showcase

Future Improvements

The game interface could definitely use a bit for polishing; if I had more time, I’d want to try making more advanced visual and sound effects for when an enemy is attacked or when the protagonist successfully collects a power-up so that players don’t have to keep checking their score and attack charge bar to keep track. It would also be interesting if I could somehow incorporate haptic feedback on the guitar controller– perhaps have the strings vibrate when an attack is used — so that the experience is more immersive and engaging.

Progress

Over the break, I made quite a few changes to the hardware component. While i had originally planned to attach 4 wires to individual guitar strings and use a wired conductive guitar pick to create a controller in which strumming each string triggers a different in-game action, this failed to work as all the strings were connected to a metal saddle, which made all the strings one conductive body and therefore could not be utilized for individual actions. I ultimately decided to turn the guitar’s metal frets into the buttons of the controller. I covered three separate frets with copper tape for enhanced visibility and connected them to wires, doing the same for the metal saddle; when the player presses any string against the taped frets, a circuit is completed and a signal is sent from arduino to p5js to trigger 3 actions — start, jump, and attack — depending on which fret is pressed.

The overall structure of the p5 game is now complete. The only work left to do is to finish the visuals of the game, as some of the power-ups and enemies are displayed as basic black placeholders. A system also has to be set up to keep track of high scores over the course of multiple games.

User Testing

At first, it was confusing for users to figure out how to operate the guitar controller, as I haven’t created any signage to attach to the frets and indicate their functions. It took them a few seconds of fiddling before getting to the actual gameplay. The gameplay also wasn’t as straightforward as I thought it was; users were confused as to how to increase their score (which is by collecting power-ups to build up attack charge and attack enemies). I will have to work on the visual components of the guitar controller (adding small labels, etc.) and adding an instruction menu before the game starts.

Concept

The game is inspired by the history of Taiwanese rock, which developed not as a direct product of American cultural imperialism but as a manifestation of our people’s desire to preserve our cultural identity through these modern forms of music; many Taiwanese rock bands write their lyrics in Taiwanese Hokkien and are very involved in social movements outside of their music. These are elements of Taiwanese culture that were suppressed during the martial law era, and the vibrant pop punk/punk rock scene embodies the resilience of our people. Taiwanese rock is an amalgamation of local Taiwanese, Chinese, American, and Japanese influences that reflects our complex history as a colonial subject of many but also our present and future as a country.

I intend for the game to be a light-hearted introduction to this aspect of Taiwan. In the spirit of the wacky, energetic aesthetic of the Scott Pilgrim franchise, the player controls a guitarist who is running late for a gig and has to get there as soon as possible, evading cops and picking up pirated music tapes to gain experience and recharge. My final project will be a game centered around a guitar repurposed as a gamepad: each string, when strummed, triggers a specific movement for the protagonist.

Progress

Assets from itch.io

As of now, I have a rough configuration of the guitar controller; 4 of the 6 strings are connected to the breadboard, on which a circuit will send input if the strings are strum by the metal guitar pick.

In my previous assignments, I wrapped a regular plastic guitar pick in tin foil and taped jumper wires to the foil, but it was rather flimsy. This time, I decided to use a flat metal nut as a guitar pick so that I don’t have to wrap additional material on it to make it conductive. Right now, the wires are temporarily taped to the nut, but I am considering soldering them directly so that they don’t shift around and affect connectivity.
As for the game itself, I have set up the moving background, as the game will be a side-scroller.

Exercise 1:

Make something that uses only one sensor on Arduino and makes the ellipse in p5 move on the horizontal axis, in the middle of the screen, and nothing on Arduino is controlled by p5.

The location of the x of ellipse is being altered by the potentiometer.

The value from the potentiometer is mapped between 0 and 640, the width of the canvas.

Utilizing a Potentiometer, the ellipse moves along the horizontal axis, while also changing colors by making changes to the B value of fill.

Arduino code

P5 code:

let ellipseX = 0; //x value of ellipse to be changed by potentiometer
let B =0;

function setup() {
  createCanvas(640, 480);
  ellipseMode(CENTER);
}

function draw() {
  clear();
  background(0)
  fill(255,0,B);
  ellipse(ellipseX, height/2, 50, 50);


  if (!serialActive) {
    text("Press Space Bar to select Serial Port", 20, 30);
  } else {
    text("Connected", 20, 30);  
    // Print the current values
    text('ellipseX = ' + str(ellipseX), 20, 50);
  }
}

function keyPressed() {
  if (key == " ") {
   
    setUpSerial(); //establish serial communication
  }
}

function readSerial(data) {  
  if (data) {    //run only if data is received
    data = data.trim(); // Remove any whitespace
    if (!isNaN(data)) { //check whether data is a number or not
      //debug: console.log("Received:", data);
      ellipseX = int(data);
    }
  }

Exercise 2:

Make something that controls the LED brightness from p5.

A slider is created and data from it is sent to the arduino. Based on the input from the p5 sketch, the LED’s brightness is adjusted accordingly.

Arduino code

P5 code:

let slider;
let brightness = 0;
function setup() {
  createCanvas(400, 400);
  // Create a brightness slider
  slider = createSlider(0, 255, 128);
  slider.position(width/2, height/2);
  slider.style('width', '100px');
}
function draw() {
  background(255);
  if (!serialActive) {
    textAlign(CENTER)
    text("Press Space Bar to select Serial Port", width/2, height/3);
  } else {
    text("Connected", width/2, height/3);
  }
  brightness = slider.value();
}
function keyPressed() {
  if (key == " ") {
    // important to have in order to start the serial connection!!
    setUpSerial();
  }
}
function readSerial(data) {
  console.log(data);
    let dataToSend = brightness + ", \n";
    writeSerial(dataToSend);  
}

Exercise 3:

Take the gravity wind example and make it so every time the ball bounces one led lights up and then turns off, and you can control the wind from one analog sensor.

Using a potentiometer mapped from -2 to 2, when the potentiometer is towards the left, the wind blows towards the left, and vice versa. The LED lights up every time the ball touches the bottom edge of the canvas.

A serial signal is sent to arduino everytime the ball touches the bottom of the canvas, resulting in the led to light up on the arduino. The potentiometer’s value will be reflected in the direction that the wind is blowing on the ball. I also added two walls on the right and left sides of the canvas, to prevent the wind from blowing the ball outside of the canvas.

Arduino code

P5 code:

let led;
let velocity;
let gravity;
let position;
let acceleration;
let wind;
let drag = 0.99;
let mass = 50;

function setup() {
  createCanvas(640, 360);
  noFill();
  position = createVector(width/2, 0);
  velocity = createVector(0,0);
  acceleration = createVector(0,0);
  gravity = createVector(0, 0.5*mass);
  wind = createVector(0,0);
}

function draw() {
  background(255);
  applyForce(wind);
  applyForce(gravity);
  velocity.add(acceleration);
  velocity.mult(drag);
  position.add(velocity);
  acceleration.mult(0);
  ellipse(position.x,position.y,mass,mass);
  if (position.y > height-mass/2) {
      velocity.y *= -0.9;  // A little dampening when hitting the bottom
      position.y = height-mass/2;
    
    
    if(serialActive){
      writeSerial("bounced\n");
    }
    }
  // Check for collisions with the left wall
  if (position.x < mass / 2) {
    velocity.x =0; // Reverse horizontal velocity (bounce)
    position.x = mass / 2; // Correct position
  }

  // Check for collisions with the right wall
  if (position.x > width - mass / 2) {
    velocity.x =0; // Reverse horizontal velocity (bounce)
    position.x = width - mass / 2; // Correct position
  }
}

function applyForce(force){
  // Newton's 2nd law: F = M * A
  // or A = F / M
  let f = p5.Vector.div(force, mass);
  acceleration.add(f);
}




function keyPressed() {
  if (key == " ") {
    // important to have in order to start the serial connection!!
    setUpSerial();
  }
  if (keyCode==DOWN_ARROW){
    //mass=random(15,80);
    position.y=-mass;
    velocity.mult(0);
  }
}


function readSerial(data){
  
  if (data != null){
    wind.x=int(data);
    console.log("working");

  }
}

 

For two of my assignments this semester, I incorporated my guitar in arduino as an unusual switch for turning on LED lights. I really enjoyed working with it, and I figured that I might as well expand upon it for my final project. The idea of an unusual switch reminded me of videos I’ve watched in which people try to beat video games using modified objects as controllers —  dance pads, bongos, and even bananas.

Combining my hobbies, I will create an interactive system where players will use a guitar as a gamepad of sorts to defeat enemies in a p5 game. I don’t have an idea for the actual game yet, but I expect it to have a quirky, cartoonish style inspired by the Scott Pilgrim franchise, in which the titular character is a bassist and finds himself in video-game-esque fights.

The author offers a really insightful perspective on the relationship between  fashion and practical design. We often assume that medical devices and aesthetic value are inherently incompatible, but the emotional is just as important as the physical; fashionable prosthetics can provide emotional value for its users by creating a positive image and de-stigmatizing disabilities without deliberately hiding the device.

This just goes to show how inspiration often comes from the most unexpected places, therefore we must keep an open mind in order to come up with good designs instead of setting too much limitations. It is only through interdisciplinary efforts that we can take each field to new heights. While I still do not have a concrete idea for my final project, this excerpt made me consider making something that enhances some kind of human activity adding an artistic touch to it, which is more of my strong suit.

A Brief Rant on the Future of Interactive Design

I believe that the author’s frustrations with the current trends/future of interactive design are completely justified. It is undeniable that many developers now seek to make things digital just for the sake of being digital, subjugating everything behind a glass panel for no substantive reason. In this pursuit of digitalization, we often forget that humans are in fact very dependent on tactility to interact with the world around us; to reduce everything to swipes on a flat, glassy screen is to gradually reduce our cognitive abilities to respond to different forms of input from our senses.

Of course, this is not to say that touch screens are devil’s spawn and must be retired, but to suggest that perhaps efforts in technological advancement are too focused on applications of touch screens and it would be much more worthwhile to explore how other human needs can be aided by other forms of interactive design.

Follow-up

It’s quite surprising how many of the responses to the rant seemed to misunderstand the author’s point. It’s unreasonable to expect the author to singlehandedly provide a solution to the problem, which seems to be what many readers expected him to do. The point of the rant was to point out a problem and hopefully direct future research to solve that problem.

I think analogy in the last section of the article sums up the author’s argument perfectly: “A child can’t understand Hamlet, but can understand Cat In The Hat. Yet, it’s Shakespeare, not Dr. Seuss, who is the centerpiece of our culture’s literature…A tool for adults should take full advantage of the adult capabilities of both mind and body. Tools that are dumbed down for children’s minds or children’s bodies are called ‘toys’.” Ultimately, it is important to keep in mind that useful, well-designed tools make things easier while helping us develop our existing skills instead of simply dumbing things down and impose homogeneity to every activity in the name of digitalization.

Concept

For this assignment, we were inspired by the toy pianos that we’ve all played with at least a few times as children. These toys were often quite limited, as they only had enough space to accommodate the keys for a single octave. We decided to create a miniature piano that could play the notes from C to A, with a knob that can be turned to change octaves.

Setup and Code

We set up a row of buttons that play a note when pressed, a potentiometer that changes the octave, and a switch that switches the notes to their respective sharp notes.

We created arrays that store the frequencies of the natural notes across seven octaves, doing the same for the sharp notes. The range of the potentiometer is mapped to a range of 0-6 and determines which octave the notes will be played in. When the switch is low, natural notes are played, and vice versa.

To prevent the notes from continuously playing, the noTone() function stops the buzzer playing any sound when no button is pressed.

Full code on Github

if (buttonState7 == LOW && buttonState6 == LOW && buttonState5 == LOW && 
    buttonState4 == LOW && buttonState3 == LOW && buttonState2 == LOW) {
  noTone(8);  // Stop any tone on pin 8
}

Demo

Reflection

Overall, this group project was a success as we managed to recreate our inspiration using Arduino. We were really satisfied with how we made sure to implement the natural notes as well as sharp notes in our project. A few struggles we faced was that the wires and resistors were really crowded on our small breadboards, making it a little difficult to reach the buttons. Ideally, we would want to get bigger breadboards in the future so that there would be more room for interacting with the product. Additionally, the bread board was just a little too small for us to add the 7th natural note, so we could only fit 6, as we’re missing a B note. 

In the future, we would like to add two different colored led lights that switch on and off along with the natural or sharp notes. This way, it would help users know if they are currently playing natural or sharp notes, as not everyone is familiar with musical notes in that sense.

 

Concept

Expanding upon the unusual guitar switch I made for last week’s assignment, I decided to introduce a new feature that will detect the distance between the guitar and the edge of the table. I got the idea for this feature from my own carelessness, as I often knock my guitar into my desk on accident. Using input from the ultrasonic sensor in our arduino kit, a red LED light will light up as a warning whenever the body of the guitar gets too close to the edge of the desk.

Setup and Code

Like my previous assignment, I covered my guitar pick with aluminum foil to make it conductive and then taped it to a wire; another wire was attached to a larger sheet of aluminum foil and then the sheet was taped to the bridge of the guitar, allowing the yellow LED to light up when the pick makes contact with any of the guitar strings. On the other half of the breadboard, I installed the red LED bulb and the ultrasonic sensor on the edge facing outward so that it can be positioned on the edge of the desk.

I learned how to make the LED light turn on and off according to ultrasonic sensor input from these posts on the Arduino Forum:

• https://forum.arduino.cc/t/ultrasonic-sensor-with-led-how-can-i-do-this/920861
• https://forum.arduino.cc/t/hc-sr04-always-returns-zero/206641

While trying to get the red LED to work properly, I struggled for a good amount of time as the ultrasonic sensor kept retrieving 0 and only 0 as values on the serial monitor. I eventually realized that I, being my careless self, put the wrong number for the sensor’s echo pin in the definition at the start of the code.

Github code link

Demo

Reflection

The biggest challenge for this assignment was learning how to use the ultrasonic sensor, since we did not get to practice with it in class, but thankfully there were a lot of tutorials on the internet. And while it wasn’t particularly complicated, I’m proud of managing to get it working (despite the minor hiccup with the code as a result of my carelessness).

Physical Computing’s Greatest Hits (and misses)

This article was really fun and reassuring  to read. While listing recurring elements in physical computing projects,  the author stresses that old ideas can always be altered in creative ways and there is no need to actively steer clear of techniques that have been explored before. This was a wake-up call for me, as I have always struggled while trying to come up with ideas that didn’t overlap with those of Intro to IM students in previous semesters. We’re constantly taught and pressured to be different, to make something completely new, but we often forget that innovation isn’t just for the sake of “being different” but also to take existing concepts and technologies and improving upon them to suit new contexts and ideas.

I think the key is to remember that there is a creative side of physical computing that is arguably just as important as important as the technical side; it’s about how you incorporate the same elements into new narratives and fashion them for new purposes. From this point onwards, I will try not to shy away from looking at previous student works for inspiration.

Making Interactive Art: Set the Stage, Then Shut Up and Listen

Sometimes, artists have the tendency to over-explain their work, feeling the need to defend their work and themselves for validation. However, this mindset can be counterproductive when dealing with interactive art. Interactive pieces are meant to be experienced directly by the audience instead of having your audience be passive onlookers. Instead, you are creating a piece that performs and also lets the audience perform on it; you can guide your audience, but the ultimate goal is to let your audience explore what is possible with your design.