Author: Oyungerel

Concept

For my final project, I created an interactive art that aims to celebrate friendship and togetherness. The main idea of the art is that a tree branch appears from the 2 sides of the screen and grows towards each other; once they connect, a flower blooms. Consistent with the theme of togetherness, the project requires 2 people to hold hands together and the strength of their hold influences how fast the branches grow – force sensitive resistor (FSR) is sewed into a glove, which one person puts on and holds the other person’s hand with. With the free hands, each person puts a finger on the temperature sensor and the resulting reading from each sensor is mapped to a color (darkish pink = warm, light yellow = cool) and lerped together to generate a live color gradient for the background.

Video demonstration

IM showcase user interaction

Implementation

I wanted the user interface to be minimal but pretty, so the homepage, the instructions page and the main sketch are all designed to have minimal text and soft color palette.

homepage

instructions page

main sketch (after the branches have connected together)

At the end of the sketch after the branches connect, a quote about friendship and togetherness is randomly chosen from a text file and appears at the bottom of the screen.

initial hand sketches

Interaction design

As the project is about friendship and togetherness, the interaction is designed to involve 2 people’s participation. I was initially using alligator clips for connecting the temperature sensors, but they felt very frail to the hands and got disconnected easily, so I soldered wires into both of the temperature sensors and taped them on top of a cardboard to make sure they are in place. As for the FSR, my very first idea was to place it inside a soft thin object so that the users can put it in between their palms and then hold hands. I could not think of a suitable and meaningful object to place the FSR, not to mention realized it would be strange to ask people to put a random object in between their holding hands. The best option seemed to be using a glove and attaching the FSR inside it. I tested different ways of holding hands and figured that the location where the contact point is highest when holding hands with another person is the purple area in the picture below. This is where I placed the FSR and sewed it on with a thread. During the IM showcase, some people held hands very very faintly, so I had to jump in and demonstrate how they should hold hands to ensure that force was being applied to the FSR. It made me realize that it may have been helpful if there was an image depiction of proper hand holds in the instructions page.

• Arduino code

• int fsrPin = A0;
  int tempSensorPin = A1;
  int tempSensor2Pin = A2;
  int fsrReading;    // variable to store the FSR reading
  float fsrVoltage;     // variable to store the voltage value
  float tempSensorVoltage;
  float tempSensorReading;
  float temperature;
  float tempSensor2Voltage;
  float tempSensor2Reading;
  float temperature2;
  
  void setup() {
    // Start serial communication so we can send data
    // over the USB connection to our p5js sketch
    Serial.begin(9600);
  
    // We'll use the builtin LED as a status output.
    // We can't use the serial monitor since the serial connection is
    // used to communicate to p5js and only one application on the computer
    // can use a serial port at once.
    pinMode(LED_BUILTIN, OUTPUT);
  }
  
  void loop() {
    // read the FSR value (0–1023)
    fsrReading = analogRead(fsrPin); 
    fsrVoltage = fsrReading * (5.0 / 1023.0);
  
    tempSensorReading = analogRead(tempSensorPin);
    // convert the analog reading to a voltage (0–5V)
    tempSensorVoltage = tempSensorReading * (5.0 / 1023.0);
    temperature = (tempSensorVoltage - 0.5) * 100.0;
  
    tempSensor2Reading = analogRead(tempSensor2Pin);
    // convert the analog reading to a voltage (0–5V)
    tempSensor2Voltage = tempSensor2Reading * (5.0 / 1023.0);
    temperature2 = (tempSensor2Voltage - 0.5) * 100.0;
  
    // send the FSR reading and 2 temperatures to p5.js
    Serial.print(fsrReading);
    Serial.print(","); 
    Serial.print(temperature); 
    Serial.print(","); 
    Serial.println(temperature2); 
  
    // blink the built-in LED to show activity
    digitalWrite(LED_BUILTIN, HIGH);
    delay(100);
    digitalWrite(LED_BUILTIN, LOW);
  
    delay(200);
  }

  Circuit

•  p5.js code

Link to fullscreen sketch

// interface variables
let homepageImg, backButtonImg, infoButtonImg, infoButtonWhiteImg, gloveImg, sensorImg;
let ambienceSound, endingSound;
let backButtonX = 20;
let backButtonY = 20;
let backButtonW = 20;
let backButtonH = 20;
let infoButtonX = 920;
let infoButtonY = 20;
let infoButtonW = 20;
let infoButtonH = 20;
let fontNunito, fontNunitoLight;
// transparency variable for the fading start text
let alphaValue = 0;
// variable used for the fade effect of the start text
let fadeDirection = 1;
let c;
let startMsgColor = "#c27e85";
let titleColor = "#745248";
let instructionsBoxColor = "#738059";
let backgroundColor = "#F0EBE5";
let vizOver = true;
let vizStarted = false;
let instructionsOn = false;
let endingSoundPlayed = false;
let ambienceSoundPlaying = false;
let homePage;
let viz;

// illustration variables
let branchGrowth = 0.1;
let maxGrowth;
let leafGrowth;
// for tracking if the branches have connected at the center
let connected = false;
// for storing the coordinates of left branch points
let leftPoints = [];
// for storing the coordinates of right branch points
let rightPoints = [];
let leafImg;
let leadImgFlipped;
let temperature_1;
let temperature_2;
let quotes = [];
let chosenQuote = "";

// arduino variables
let fsrValue = 0;
let temperature = 0;
let temperature2 = 0;
let arduinoConnected = false;

// map the sensor readings to a different range
function mapVals(sensorType, val) {
  // map temperature to colors
  if (sensorType == "temperature") {
    // lowest reading on the temp sensor was 18
    // highest reading on the temp sensor was 29
    // so these are mapped to a number on a wider scale
    // to ensure that the background color changes are more visible/stronger
    return map(val, 18, 30, 1, 38);
  // map fsr reading to branchGrowth
  } else if (sensorType == "fsr") {
    // bound the branch growth to maximum 0.5 to make sure that
    // the visualization doesn't end too quickly
    return map(val, 0, 1023, 0.1, 0.5);
  }
}

function preload() {
  homepageImg = loadImage("/assets/homepage.png");
  backButtonImg = loadImage("/assets/left.png");
  infoButtonImg = loadImage("/assets/information.png");
  infoButtonWhiteImg = loadImage("/assets/information_white.png");
  fontNunito = loadFont("/fonts/Nunito-Medium.ttf");
  fontNunitoLight = loadFont("/fonts/Nunito-Light.ttf");
  leafImg = loadImage("/assets/nature.png");
  leafImgFlipped = loadImage("/assets/nature_flipped.png");
  flowerImg = loadImage("/assets/flower.png");
  gloveImg = loadImage("/assets/glove.png");
  sensorImg = loadImage("/assets/sensor.png");
  ambienceSound = loadSound("/sounds/ambience_long.mp3");
  endingSound = loadSound("/sounds/ending.mp3");
  quotes = loadStrings("/assets/quotes.txt");
  heartImg = loadImage("/assets/heart.png");
}

function setup() {
  createCanvas(960, 540);
  background(0);

  homePage = new HomePage();
  viz = new Visualization();

  // each branch can only grow for a length that's half of the screen
  maxGrowth = width / 2;
  chosenQuote = random(quotes);
}

class HomePage {
  constructor() {
  }

  display() {
    image(homepageImg, 0, 0, width, height);
    image(infoButtonImg, infoButtonX, infoButtonY, infoButtonW, infoButtonH);
    // fade effect on the "Press Enter to start" text
    // by varying the transparency
    alphaValue += fadeDirection * 3;
    if (alphaValue >= 255 || alphaValue <= 0) {
      fadeDirection *= -1;
    }

    push();
    textAlign(CENTER);
    textFont(fontNunito);
    textSize(16);
    c = color(startMsgColor);
    c.setAlpha(alphaValue);
    fill(c);
    // ask user to select serial port first
    if (!arduinoConnected) {
        text("press SPACE to select serial port", width / 2, 500);
    // once serial port selected, show a different text
    } else {
        text("press ENTER to start", width / 2, 500);
    }
    pop();
  }

  showInstructions() {
    this.display();
    let c = color(instructionsBoxColor);
    c.setAlpha(245);
    fill(c);
    noStroke();
    rect(0, 0, width, height);
    image(
      infoButtonWhiteImg,
      infoButtonX,
      infoButtonY,
      infoButtonW,
      infoButtonH
    );
    
    // instructions text
    push();
    textAlign(CENTER);
    textFont(fontNunito);
    textSize(16);
    fill(color(backgroundColor));
    text("h o w    i t    w o r k s", width / 2, 100);
    text(
      "1. have a friend next to you and stand facing each other",
      width / 2,
      200
    );
    text("2. put a finger each on the temperature sensor", width / 2, 250);
    text(
      "3. with the free hands: one person wears the glove and holds the friend's hand",
      width / 2,
      300
    );
    text("4. watch the tree branches bloom!", width / 2, 350);
    pop();
  }
}

class Visualization {
  constructor() {
  }
  
  start() {
    // play the ambience sound if it's not playing
    if (!ambienceSoundPlaying) {
      ambienceSound.play();
      // use a boolean variable to ensure it's only played once
      ambienceSoundPlaying = true;
    }
    
    // stop the sound if visualization is over
    if (connected || vizOver) {
      ambienceSound.stop();
    }
        
    noStroke();
    
    // map the temp sensor readings
    temperature_1 = mapVals("temperature", temperature);
    temperature_2 = mapVals("temperature", temperature2);
    // map each person's temperature to colors
    let color1 = mapTempToColor(temperature_1);  // color for person 1
    let color2 = mapTempToColor(temperature_2);  // color for person 2

    // smooth gradient blending between the two temperatures
    for (let x = 0; x < width; x++) {
      // lerp factor based on x position (left to right transition)
      let lerpFactor = map(x, 0, width, 0, 0.5);

      // blend between color1 and color2
      let col = lerpColor(color1, color2, lerpFactor);

      // apply the color vertically to the canvas
      stroke(col);
      line(x, 0, x, height);  // vertical lines for smooth gradient
    }

    // go back to homepage button
    image(backButtonImg, backButtonX, backButtonY, backButtonW, backButtonH);
    
    // map fsr reading
    let growthSpeed = mapVals("fsr", fsrValue);
    // use the mapped fsr value to grow the branch
    branchGrowth += growthSpeed;
    
    if (branchGrowth < maxGrowth) {
      // introduce noise to left branch
      let noiseOffsetLeftBranch = branchGrowth * 0.01;
      // introduce noise to right branch
      // slightly change the noise offset so that branches don't grow symmetrically and instead looks different
      let noiseOffsetRightBranch = branchGrowth * 0.01 + 1000;
      
      // generate x, y coordinates for points for both branches
      let yLeft = height / 2 + map(noise(noiseOffsetLeftBranch), 0, 1, -40, 40);
      let yRight =
        height / 2 + map(noise(noiseOffsetRightBranch), 0, 1, -40, 40);
      let xLeft = branchGrowth;
      let xRight = width - branchGrowth;
      
      // once the branches are nearing the center, reduce the noise and make them remain near the horizontal middle of the canvas to ensure they connect everytime
      let easingZone = 30;
      if (branchGrowth > maxGrowth - easingZone) {
        let amt = map(branchGrowth, maxGrowth - easingZone, maxGrowth, 0, 1);
        yLeft = lerp(yLeft, height / 2, amt);
        yRight = lerp(yRight, height / 2, amt);
      }
      
      // add the points to the array
      leftPoints.push({
        pos: createVector(xLeft, yLeft),
        // randomly decide if the leaf will be growing on the top or on the bottom
        flip: int(random(2))
});      
      rightPoints.push({
        pos: createVector(xRight, yRight),
        flip: int(random(2))
});     
    } else if (!connected) {
      connected = true;
      
      // play the ending sound if the branches have connected
      if (!endingSoundPlayed) {
        endingSound.play();
        endingSoundPlayed = true;
      }
    }

    // draw branches
    push();
    strokeWeight(3);
    stroke(110, 70, 40); // brown
    noFill();

    beginShape();
    // draw the left branch
    for (let ptObj of leftPoints) {
      let pt = ptObj.pos;
      vertex(pt.x, pt.y);
      
      // for every 75th x coordinate, draw an image of a leaf
      // position of leaf is stored in the flip attribute
      if (int(pt.x) % 75 == 0) {
        if (ptObj.flip === 0) {
          image(leafImgFlipped, pt.x, pt.y - 2, 40, 40);
        } else {
          image(leafImg, pt.x, pt.y - 37, 40, 40);
        }
      }
    }
    endShape();
    
    // draw the right branch
    beginShape();
    for (let ptObj of rightPoints) {
      let pt = ptObj.pos;
      vertex(pt.x, pt.y);

      if (int(pt.x) % 75 == 0) {
        if (ptObj.flip === 0) {
          image(leafImgFlipped, pt.x, pt.y - 2, 40, 40);
        } else {
          image(leafImg, pt.x, pt.y - 37, 40, 40);
        }
      }
    }
    endShape();
    pop();
    
    let leftEnd = leftPoints[leftPoints.length - 1].pos;
    let rightEnd = rightPoints[rightPoints.length - 1].pos;

    let d = dist(leftEnd.x, leftEnd.y, rightEnd.x, rightEnd.y);
    
    // determine if the branches have connected by finding the distance between the 2 branch end points and checking if it's less than 5
    if (d < 5) {
      push();
      rectMode(CENTER);
      image(flowerImg, leftEnd.x - 15, (leftEnd.y - 30), 80, 80);
      
      // console.log(chosenQuote);
      
      // show the quote
      if (chosenQuote !== "") {
        textAlign(CENTER, CENTER);
        textFont(fontNunito);
        textSize(20);
        fill(titleColor);
        text(chosenQuote, width / 2, height - 80);
      }
      
      // heart image at the bottom of the quote
      image(heartImg, width / 2, height - 60, 40, 40);
      pop();
    }
  }
}

function draw() {
  if (instructionsOn) {
    homePage.showInstructions();
  } else if (!vizStarted || vizOver) {
    homePage.display();
  } else {
    viz.start();
  }
  // print(mouseX + "," + mouseY);
}

// map temperature to color which will be used to control the color of the background
function mapTempToColor(temp) {
  let coolColor = color("#F0EBE5");
  let warmColor = color("#dea0a6"); 
  
  // lerp the light yellow color with the dark pink color based on the temperature
  return lerpColor(coolColor, warmColor, map(temp, 1, 38, 0, 1));
}

function readSerial(data) {
  ////////////////////////////////////
  //READ FROM ARDUINO HERE
  ////////////////////////////////////

  if (data != null) {
    let fromArduino = split(trim(data), ",");
    // if the right length, then proceed
    if (fromArduino.length == 3) {
      fsrValue = int(fromArduino[0]);
      temperature = int(fromArduino[1]);
      temperature2 = int(fromArduino[2]);
    }
    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    // let sendToArduino = left + "," + right + "\n";
    // writeSerial(sendToArduino);
  }
}

function keyPressed() {
  // if the enter key is pressed, reset state and start the visualization
  if (keyCode === ENTER) {
    if (vizOver && !instructionsOn) {
      // reset visualization state
      branchGrowth = 20;
      leftPoints = [];
      rightPoints = [];
      connected = false;
      ambienceSoundPlaying = false;
      endingSoundPlayed = false;
      chosenQuote = random(quotes);
      vizStarted = true;
      vizOver = false;
    }
  }
  
  // if space is pressed, setup serial communication
  if (key == " ") {
    setUpSerial();
    arduinoConnected = true;
  }
}

function mousePressed() {
  // if mouse is pressed on the < button, go back to the homepage
  if (
    mouseX >= backButtonX &&
    mouseX <= backButtonX + backButtonW &&
    mouseY >= backButtonY &&
    mouseY <= backButtonY + backButtonH
  ) {
    vizOver = true;
    ambienceSound.stop();
    vizStarted = false; // Reset visualization state when returning to homepage
    console.log("back button pressed");
  // if mouse if pressed on the information button, show the instructions page
  } else if (
    mouseX >= infoButtonX &&
    mouseX <= infoButtonX + backButtonW &&
    mouseY >= infoButtonY &&
    mouseY <= infoButtonY + backButtonH
  ) {
    instructionsOn = !instructionsOn;
    console.log("info button pressed");
  }
}

• Communication between Arduino and p5.js

3 readings from Arduino (2 temperature sensors and 1 FSR) are sent in a X,Y,Z format with commas separating them to p5.js using serial communication. The p5.js code then maps these values to their respective values for color and speed. I could not think of meaningful data to send from p5.js to Arduino that doesn’t just send data for the sake of having a two-way communication. I considered using the LCD to show text (names of the users, reading values etc), buzzer to play music, but as professor mentioned in our chat on the initial evaluation day, these things can be done on the screen as well and does specifically require to be implemented on the hardware. Thus, I decided to continue using Arduino as input only and I think it worked well at the end.

Reflection and Proud Moments

I like how the p5.js interface and the hardware interface ended up looking from a visual standpoint. I put an effort into choosing the visual assets, the fonts and the different colors I used across the sketch to create a coherent visual experience that is simple yet visually pleasing to look at, and I was very happy when people complimented on it! I decorated the cardboard casing covering the Arduino to match the artwork by drawing leaves and flowers, and I think that turned out successfully as well.

For the p5.js artwork, it was tricky to connect the 2 branches every time. The way the drawing of the branches work is that a variable, called branchGrowth, increases every frame, and its speed is determined by the FSR reading. For each new value of branchGrowth, the code calculates a new point for the left and right branches. To ensure that the branches look organic and not like a straight line, Perlin noise (noise(…)) is introduced to create smooth vertical variation in the branches. As each point of the branch is generated randomly (to a certain degree), unless the last segment of each branch was somehow smoothed out and maintained near the horizontal middle, the 2 branches always ended up at different heights. To solve this issue, I eased the y values of both branches toward the center as they approached close each other. This helped to make the branch growth to appear organic but also controlled so that they connected every time.

Overall, I had a lot of fun working on this project and it was a genuinely rewarding experience watching people’s reactions and seeing how they interact with the piece (including finding some loopholes!). I am glad I stuck with the interactive artwork idea and the theme of togetherness, because I got to see some cute interactions of friends who tried out my project <3

Future Improvements

If I were to continue with the project and develop it further, I think I would explore ways of adding more physical components (other than sensors) that people could interact with. One example could be a flower that blooms when the branches meet by using a servo motor. I would also make improvements to the instructions page because it seemed like people didn’t easily see that there was an info button. Even when they did see the button and press on it, there was a tendency for people to take a quick glance at it and head straight into starting the sketch.

Finalized concept

For my final project, I have decided to create an interactive art piece that is based around the idea of friendship and human bonding. The piece requires there to be 2 users at a time, ideally friends. The idea of the art piece is to allow the users to watch their connection to each other visualized on the screen, based on their body temperatures and the way they are holding their hands.

As can be seen in the terrible sketch, the users will be asked to put on a glove on one hand each and hold hands. Inside one of the gloves, there will be a force sensitive resistor (FSR) attached. They will be then be asked to put their other free hand on the temperature sensor on the table.

In p5.js, each user generates their own plant-like/branch-like structure using Perlin noise loops, growing from opposite sides of the screen. The color of each plant reflects the user’s temperature reading, mapped to a color gradient. As the plants grow toward each other, their speed is controlled by how tightly the users are holding hands, measured by the FSR sensor. When their grip becomes strong enough, the plants meet and fuse in the center. Upon connecting, the colors blend into a new gradient based on the users’ combined temperature and the Arduino buzzer will be triggered to play a melody.

Arduino <–> p5.js communication

• Arduino -> p5
  • FSR – send reading to p5
  • temperature sensor – send reading to p5
• p5
  • map FSR and temperature readings to appropriate values to use for controlling speed and color
  • the magnitude of FSR reading changes the speed at which the 2 illustrations “grow” towards each other and connect
  • temperature sensor reading controls the color of the sketch
• p5 -> Arduino
  • when the 2 illustrations touch and merge in the middle, p5 sends to Arduino a digital signal that starts a melody on the buzzer

Progress

I started creating the p5js interface and added basic interactions like showing the instructions, starting the illustration and going back to the homepage to restart.

I tried testing 2 of the temperature & humidity sensor from the IM equipment center but both didn’t work (most likely due to hardware breakages), so for the time being, I plan to stick to using the temperature sensor provided in our Sparkfun kit.

let rVal = 0;
let alpha = 255;
let left = 0; // True (1) if mouse is being clicked on left side of screen
let right = 0; // True (1) if mouse is being clicked on right side of screen
let velocity;
let gravity;
let position;
let acceleration;
let wind;
let drag = 0.99;
let mass = 50;

function setup() {
  createCanvas(640, 480);
  textSize(18);
  noFill();
  position = createVector(width/2, 0   );
  velocity = createVector(0,0);
  acceleration = createVector(0,0);
  gravity = createVector(0, 0.5*mass);
  wind = createVector(0,0);
}

function draw() {
  // one value from Arduino controls the background's red color
  background(255);
  applyForce(wind);
  applyForce(gravity);
  velocity.add(acceleration);
  velocity.mult(drag);
  position.add(velocity);
  acceleration.mult(0);
 
  // make everything black
  fill(0,0,0);
 
  // draw bounce ball
  ellipse(position.x,position.y,mass,mass);
  if (position.y > height-mass/2) {
      velocity.y *= -0.9;  // A little dampening when hitting the bottom
      position.y = height-mass/2;
  }

  // horizontal ellipse -- goes across screen based on potentiometer
  circle(map(alpha, 0, 1023, 0, width), height/2, 100);
 
  if (!serialActive) {
    text("Press Space Bar to select Serial Port", 20, 30);
  } else {
    text("Connected", 20, 30);
   

    // Print the current values
    text('light sensor = ' + str(rVal), 20, 50);
    text('potentiometer = ' + str(alpha), 20, 70);
  }

  // bar graph for analog LED
  rect(width-50,0,50,right);
 
  // wind -- change wind direction based on light sensor
  if (rVal < 600){
    wind.x=-1;
  }
  if (rVal > 600){
    wind.x=1;
  }
}

function keyPressed(){
  if (keyCode==ENTER){
    position.y=-mass;
    velocity.mult(0);
  }
  if (key == " ") {
    // important to have in order to start the serial connection!!
    setUpSerial();
  }
}

// This function will be called by the web-serial library
// with each new *line* of data. The serial library reads
// the data until the newline and then gives it to us through
// this callback function
function readSerial(data) {
  ////////////////////////////////////
  //READ FROM ARDUINO HERE
  ////////////////////////////////////

  if (data != null) {
    // make sure there is actually a message
    // split the message
    let fromArduino = split(trim(data), ",");
    // if the right length, then proceed
    if (fromArduino.length == 2) {
      // only store values here
      // do everything with those values in the main draw loop
     
      // We take the string we get from Arduino and explicitly
      // convert it to a number by using int()
      // e.g. "103" becomes 103
      rVal = int(fromArduino[0]);
      alpha = int(fromArduino[1]);
    }

    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    let sendToArduino = left + "," + right + "\n";
    writeSerial(sendToArduino);
  }
}

let rVal = 0;
let alpha = 255;
let left = 0; // True (1) if mouse is being clicked on left side of screen
let right = 0; // True (1) if mouse is being clicked on right side of screen
let velocity;
let gravity;
let position;
let acceleration;
let wind;
let drag = 0.99;
let mass = 50;

function setup() {
  createCanvas(640, 480);
  textSize(18);
  noFill();
  position = createVector(width/2, 0   );
  velocity = createVector(0,0);
  acceleration = createVector(0,0);
  gravity = createVector(0, 0.5*mass);
  wind = createVector(0,0);
}

function draw() {
  // one value from Arduino controls the background's red color
  background(255);
  applyForce(wind);
  applyForce(gravity);
  velocity.add(acceleration);
  velocity.mult(drag);
  position.add(velocity);
  acceleration.mult(0);
 
  // make everything black
  fill(0,0,0);
 
  // draw bounce ball
  ellipse(position.x,position.y,mass,mass);
  if (position.y > height-mass/2) {
      velocity.y *= -0.9;  // A little dampening when hitting the bottom
      position.y = height-mass/2;
  }
 
  // horizontal ellipse -- goes across screen based on potentiometer
  circle(map(alpha, 0, 1023, 0, width), height/2, 100);
 
  if (!serialActive) {
    text("Press Space Bar to select Serial Port", 20, 30);
  } else {
    text("Connected", 20, 30);
   

    // Print the current values
    text('light sensor = ' + str(rVal), 20, 50);
    text('potentiometer = ' + str(alpha), 20, 70);
  }

  // click on one side of the screen, LED will get brighter
  // click on the other side, LED will dim
  if (mouseIsPressed) {
    if (mouseX <= width / 2) {
      if (right < 255) {
        right++;
      }
    } else {
      if (right > 0) {
        right--;
      }
    }
  }
  // bar graph for analog LED
  rect(width-50,0,50,right);
}

function keyPressed(){

  if (keyCode==ENTER){
    position.y=-mass;
    velocity.mult(0);
  }
  if (key == " ") {
    // important to have in order to start the serial connection!!
    setUpSerial();
  }
}

// This function will be called by the web-serial library
// with each new *line* of data. The serial library reads
// the data until the newline and then gives it to us through
// this callback function
function readSerial(data) {
  ////////////////////////////////////
  //READ FROM ARDUINO HERE
  ////////////////////////////////////

  if (data != null) {
    // make sure there is actually a message
    // split the message
    let fromArduino = split(trim(data), ",");
    // if the right length, then proceed
    if (fromArduino.length == 2) {
      // only store values here
      // do everything with those values in the main draw loop
     
      // We take the string we get from Arduino and explicitly
      // convert it to a number by using int()
      // e.g. "103" becomes 103
      rVal = int(fromArduino[0]);
      alpha = int(fromArduino[1]);
    }

    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    let sendToArduino = left + "," + right + "\n";
    writeSerial(sendToArduino);
  }
}

let rVal = 0;
let alpha = 255;
let left = 0; // True (1) if mouse is being clicked on left side of screen
let right = 0; // True (1) if mouse is being clicked on right side of screen
let velocity;
let gravity;
let position;
let acceleration;
let wind;
let drag = 0.99;
let mass = 50;

function setup() {
  createCanvas(640, 480);
  textSize(18);
  noFill();
  position = createVector(width/2, 0   );
  velocity = createVector(0,0);
  acceleration = createVector(0,0);
  gravity = createVector(0, 0.5*mass);
  wind = createVector(0,0);
}

function draw() {
  // one value from Arduino controls the background's red color
  background(255);
  applyForce(wind);
  applyForce(gravity);
  velocity.add(acceleration);
  velocity.mult(drag);
  position.add(velocity);
  acceleration.mult(0);
 
  // make everything black
  fill(0,0,0);
 
  // draw bounce ball
  ellipse(position.x,position.y,mass,mass);
  if (position.y > height-mass/2) {
      velocity.y *= -0.9;  // A little dampening when hitting the bottom
      position.y = height-mass/2;
  }
 
  // if ball touches ground, make LED light
  if (position.y >= height-mass/2-10) {
    // light up LED if ball touch ground
    left = 1;  
  }
  else {
    left = 0;
  }

  // horizontal ellipse -- goes across screen based on potentiometer
  circle(map(alpha, 0, 1023, 0, width), height/2, 100);
 
  if (!serialActive) {
    text("Press Space Bar to select Serial Port", 20, 30);
  } else {
    text("Connected", 20, 30);
   

    // Print the current values
    text('light sensor = ' + str(rVal), 20, 50);
    text('potentiometer = ' + str(alpha), 20, 70);
  }
}

function keyPressed(){

  if (keyCode==ENTER){
    position.y=-mass;
    velocity.mult(0);
  }
  if (key == " ") {
    // important to have in order to start the serial connection!!
    setUpSerial();
  }
}

// This function will be called by the web-serial library
// with each new *line* of data. The serial library reads
// the data until the newline and then gives it to us through
// this callback function
function readSerial(data) {
  ////////////////////////////////////
  //READ FROM ARDUINO HERE
  ////////////////////////////////////

  if (data != null) {
    // make sure there is actually a message
    // split the message
    let fromArduino = split(trim(data), ",");
    // if the right length, then proceed
    if (fromArduino.length == 2) {
      // only store values here
      // do everything with those values in the main draw loop
     
      // We take the string we get from Arduino and explicitly
      // convert it to a number by using int()
      // e.g. "103" becomes 103
      rVal = int(fromArduino[0]);
      alpha = int(fromArduino[1]);
    }

    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    let sendToArduino = left + "," + right + "\n";
    writeSerial(sendToArduino);
  }
}

arduino

// Week 11.2 Example of bidirectional serial communication

// Inputs:
// - A0 - sensor connected as voltage divider (e.g. potentiometer or light sensor)
// - A1 - sensor connected as voltage divider 
//
// Outputs:
// - 2 - LED
// - 5 - LED

int leftLedPin = 2;
int rightLedPin = 5; // analog

void setup() {
  // Start serial communication so we can send data
  // over the USB connection to our p5js sketch
  Serial.begin(9600);

  // We'll use the builtin LED as a status output.
  // We can't use the serial monitor since the serial connection is
  // used to communicate to p5js and only one application on the computer
  // can use a serial port at once.
  pinMode(LED_BUILTIN, OUTPUT);

  // Outputs on these pins
  pinMode(leftLedPin, OUTPUT);
  pinMode(rightLedPin, OUTPUT);

  // Blink them so we can check the wiring
  digitalWrite(leftLedPin, HIGH);
  digitalWrite(rightLedPin, HIGH);
  delay(200);
  digitalWrite(leftLedPin, LOW);
  digitalWrite(rightLedPin, LOW);



  // start the handshake
  while (Serial.available() <= 0) {
    digitalWrite(LED_BUILTIN, HIGH); // on/blink while waiting for serial data
    Serial.println("0,0"); // send a starting message
    delay(300);            // wait 1/3 second
    digitalWrite(LED_BUILTIN, LOW);
    delay(50);
  }
}

void loop() {
  // wait for data from p5 before doing something
  while (Serial.available()) {
    digitalWrite(LED_BUILTIN, HIGH); // led on while receiving data

    int left = Serial.parseInt();
    int right = Serial.parseInt();
    if (Serial.read() == '\n') {
      digitalWrite(leftLedPin, left);
      analogWrite(rightLedPin, right);
      int sensor = analogRead(A0);
      delay(5);
      int sensor2 = analogRead(A1);
      delay(5);
      Serial.print(sensor);
      Serial.print(',');
      Serial.println(sensor2);
    }
  }
  digitalWrite(LED_BUILTIN, LOW);
}

For the final project, I am leaning more towards creating an interactive experience/artwork rather than a game. The user would interact with the physical components on Arduino which would trigger changes / feedback on the p5js sketch. I want my project to have a coherent theme that ties the physical and visual components together, and though I haven’t settled on this idea completely, this is what I’m currently thinking:

• • theme: human connection / friendship / bonding
    • 2 users each put one hand on a surface that measures temperature (temperature sensor) and uses the other hand to hold each other’s hand, with an object containing a force dependent resistor (FDR) in between
    • the warmth of the hands are used to change the color of the visualization and the 2 temperatures make the colors lerp
    • how strongly the 2 are holding hands together, measured by the FDR, changes the stroke width of the illustration (tbd) or how big/small it is etc
    • the resulting sketch would represent the 2 users’ connection; their body warmths dictate the color changes and the illustration dynamically changes depending on how strongly the 2 are holding hands together
  • resources: temperature sensor, force dependent resistor bundled up inside a flat/thin object that 2 people can hold in between their palms while holding hands

I want the p5js visuals to be minimalistic, calming and have a mild/muted color palette.

I am currently looking for inspiration for the actual illustration/art on the p5js and here are some I really liked after searching on the web and revisiting some of our past class readings:

Design Meets Disability

This reading was probably my favorite out of all the pieces we’ve read so far. It shifted how I thought about design and its relationship to disability; before this, I had mostly seen assistive devices as purely functional tools but Pullin’s writing made me realize that it’s really not the case. He reframes assistive devices – i.e. hearing aid, prosthetic hand/leg, wheelchair – as opportunities for self-expression and avenues for celebrating individuality, the same way how eyeglasses have evolved into fashion statements. So much of design for disability has been shaped by a desire to hide and minimize differences and prioritized functionality over aesthetics. It has also often been suboptimally fulfilled through universal design where a product is overburdened with mediocre at best features that attempts to accommodate everyone’s needs, a concept which is perfectly summarized by the expression “flying submarine” in the piece. To bridge this disconnect between design and disability, the author argues that disabled people should participate in the design process as collaborators alongside engineers, designers and medical practitioners to actualize a true inclusive design for assistive devices that balance function with aesthetics, and embrace individuality over invisibility. The idea that inclusion is not just about access and compliance, but also deeply involves choice and self-expression, is one that I will remember and apply in the things I do now moving forward.

Description:

Vivian and I decided to create a musical instrument using force sensitive resistors (FSR) that reminded us of drumpads. There are 3 resistors in total, and each represents a different octave. Moreover, if you touch multiple at the same time you again get different octaves. There are different combinations you can try, totaling to 7 altogether. These FSR are used as digital inputs in this case, but we have a potentiometer that moves you along the scale within each octave. By turning the potentiometer, you get the full range of an octave, and by touching the FSR, you get different octaves. We also use an LCD screen to visually represent what you’re playing. Each octave has its own symbol/emoji on the screen and depending on the note you’re playing, the emoji slides across the screen. The higher the note, the farther across the screen it slides.

Video Demonstration:

video demonstration

Image of circuit:

Difficulties:

At first, we wanted to use the FSR as analog inputs and attempted many ideas of using it, but ended up on this method of using them as digital inputs because we liked that you can make different combinations. The FSR reminded us of drumpads, which is why we chose them. We also tried soldering the FSR to wires to make them easier to connect to the breadboard, but that ended up taking way too long, so we just stuck them directly to the breadboard.

We ended up using many wires and at some point, our breadboard became a bit overcrowded, and there was one time when we didn’t realize that one of the resistors was connected to 5V instead of GND.

To detect whether or not someone touched the FSR at first, we checked whether or not the reading was above 0, but realized quickly that even without touching the FSR, it was rarely ever 0. It teetered on values just above 0, so we instead used a threshold of 100.

Another huge problem we encountered in one of our past ideas was using multiple speakers at the same time. We were going to have each FSR correspond to their own speaker, but found out that the tone function doesn’t allow for multiple outputs.

Code Snippets:

Printing the custom emojis:

// make some custom characters:
byte Heart[8] = {
0b00000,
0b01010,
0b11111,
0b11111,
0b01110,
0b00100,
0b00000,
0b00000
};

// LCD screen
if (lastPotInd != potInd) {
  lcd.clear();
}
lcd.setCursor(potInd, 0);
lcd.write(byte(melodyInd));

2D array of notes:

int melody[7][12] = {
  {NOTE_C1, NOTE_CS1, NOTE_D1, NOTE_DS1, NOTE_E1, NOTE_F1, NOTE_FS1, NOTE_G1, NOTE_GS1, NOTE_A1, NOTE_AS1, NOTE_B1},
  {NOTE_C2, NOTE_CS2, NOTE_D2, NOTE_DS2, NOTE_E2, NOTE_F2, NOTE_FS2, NOTE_G2, NOTE_GS2, NOTE_A2, NOTE_AS2, NOTE_B2},
  {NOTE_C3, NOTE_CS3, NOTE_D3, NOTE_DS3, NOTE_E3, NOTE_F3, NOTE_FS3, NOTE_G3, NOTE_GS3, NOTE_A3, NOTE_AS3, NOTE_B3},
  {NOTE_C4, NOTE_CS4, NOTE_D4, NOTE_DS4, NOTE_E4, NOTE_F4, NOTE_FS4, NOTE_G4, NOTE_GS4, NOTE_A4, NOTE_AS4, NOTE_B4},
  {NOTE_C5, NOTE_CS5, NOTE_D5, NOTE_DS5, NOTE_E5, NOTE_F5, NOTE_FS5, NOTE_G5, NOTE_GS5, NOTE_A5, NOTE_AS5, NOTE_B5},
  {NOTE_C6, NOTE_CS6, NOTE_D6, NOTE_DS6, NOTE_E6, NOTE_F6, NOTE_FS6, NOTE_G6, NOTE_GS6, NOTE_A6, NOTE_AS6, NOTE_B6},
  {NOTE_C7, NOTE_CS7, NOTE_D7, NOTE_DS7, NOTE_E7, NOTE_F7, NOTE_FS7, NOTE_G7, NOTE_GS7, NOTE_A7, NOTE_AS7, NOTE_B7}
};

Mapping potentiometer value to an array index:

potInd = map(potReading, 0, 1000, 0, 12);

Logic for changing the octaves using FSR combos and selecting correct array index:

// all 3
if (fsrReading1 > 100 && fsrReading2 > 100 && fsrReading3 > 100) {
  melodyInd = 5;
}

Future Improvements:

Next time, we want to find a way to make the FSR lie flat on a surface so touching them feels more intuitive and more like a drumpad. We can also try controlling different factors about the sounds like the volume. We can also make the note progressions more predictable so that users can go through different notes more easily and play an actual song.

Through the painting class I am taking this semester, I am constantly learning to appreciate how rich the drawing experience is using a variety of different brushes and mediums and canvases, which makes it worlds apart from painting digitally using just a stylus. Different brushes have different handle lengths, width and feels, and depending on each of these factors combined with the type of texture I am aiming to create, I have to change my grip on the brush to create the stroke I want. Though painting is something we are all taught from a very young age, I realized I have never fully paid attention to the rich combination of grips and positions that my fingers take on unconsciously while holding the brush.

This is exactly what Bret brings to our attention through his written piece — the human hand is an extraordinary tool that can do extraordinary things. It is pitiful that the interfaces which are inevitably becoming essential parts of our daily lives strip down the magic of our bodies and reduces interaction to mere swiping and tapping. Though haptics and sensory feedback mechanisms are being incorporated into touch-screen devices, we are still far from having a truly dynamic interface that fully utilizes the human sensory capabilities and the richness of hand movements, even after 14 years of the publication of Bret’s writing. As Bret writes, the first step to solving the problem is to become aware of it in the first place. We all need to start recognizing the limitations of our current interfaces and start thinking out of the box — humans created airplanes and space rockets so I am sure that, if we start ideating, infusing our diverse lived experiences, coming together, we can create dynamic interfaces that employs more than just a single finger.

Physical Computing’s Greatest hits and misses

The blog post compiles and reviews popular themes in physical computing. As we head into the last few weeks of the semester and as I start thinking about the final project, I found myself having a hard time coming up with ideas from scratch. This reading reminded me that it is okay to build upon existing themes and not be pressured to create something entirely novel. There is a lot of room for “personal touch” and creativity in each of the themes discussed and by the end of the reading, I found myself feeling a little less anxious about the final project as it provided a solid foundation to start creating on. Out of the many examples mentioned, I am most intrigued by the musical instruments theme, especially because I am taking a musical instruction course this semester and appreciating the intricacies of musical instruments more deeply, so I hope to explore this idea and think about ways of elevating it with creativity and surprises.

Making Interactive Art: Set the Stage, Then Shut Up and Listen

This reading emphasizes the perspective shift required to create interactive art that makes it uniquely distinct from traditional art forms. To create a good interactive art, the artist needs to be undertake less of the role of an expressive painter who prescribes the interpretation of the piece, and more of a performance director who offers only the basic context and subtle cues, and observes the rest from a distance. I do think that this is a skill that will take practice and conscious thinking/planning, especially because there needs to be a delicate balance between providing guidance and offering ample space for creative experimentation for the audience. But this is precisely what appeals to me about interactive art — the fact that there usually exists room for spontaneity, and the opportunity to figure out the piece in my own pace.

Concept

It was my birthday recently and while thinking of this week’s assignment, I came up with the idea to create a birthday-themed project. I wanted to specifically capture the moment when the lights are turned off and your loved ones bring a cake to you with candles on it while singing happy birthday.

Implementation

Link to demonstration video

The analog sensor is a photoresistor that controls the on/off state of the 3 LEDs (red, blue and green). When the brightness is below a certain threshold, i.e. the room is dark, the 3 LEDs light up, as if party lights are turned on. The digital sensor (switch) is a slide switch that we perform a digitalRead from. When the switch closes the circuit, the yellow LED in the middle lights up to simulate a birthday candle being lit up and the piezo speaker starts playing the birthday song. The song and the yellow LED can be stopped by sliding the switch to open the circuit.

We didn’t use the slide switch in class and I was a bit confused by the 3 pins, so I referred to this video to complete the analog sensor portion. Another cool resource I found is this GitHub repo that contains Arduino code for a bunch of different songs and melodies. I simply copy pasted the code for the birthday melody from this repo and it worked perfectly!

// ------------ source: https://github.com/robsoncouto/arduino-songs/blob/master/happybirthday/happybirthday.ino
// happy birthday song portion
// change this to make the song slower or faster
int tempo = 140;

// change this to whichever pin you want to use
int buzzer = 8;

// notes of the melody followed by the duration.
// a 4 means a quarter note, 8 an eighteenth , 16 sixteenth, so on
// !!negative numbers are used to represent dotted notes,
// so -4 means a dotted quarter note, that is, a quarter plus an eighteenth!!
int melody[] = {

  // Happy Birthday
  // Score available at https://musescore.com/user/8221/scores/26906

  NOTE_C4,4, NOTE_C4,8, 
  NOTE_D4,-4, NOTE_C4,-4, NOTE_F4,-4,
  NOTE_E4,-2, NOTE_C4,4, NOTE_C4,8, 
  NOTE_D4,-4, NOTE_C4,-4, NOTE_G4,-4,
  NOTE_F4,-2, NOTE_C4,4, NOTE_C4,8,

  NOTE_C5,-4, NOTE_A4,-4, NOTE_F4,-4, 
  NOTE_E4,-4, NOTE_D4,-4, NOTE_AS4,4, NOTE_AS4,8,
  NOTE_A4,-4, NOTE_F4,-4, NOTE_G4,-4,
  NOTE_F4,-2,
 
};

// sizeof gives the number of bytes, each int value is composed of two bytes (16 bits)
// there are two values per note (pitch and duration), so for each note there are four bytes
int notes = sizeof(melody) / sizeof(melody[0]) / 2;

// this calculates the duration of a whole note in ms
int wholenote = (60000 * 4) / tempo;

int divider = 0, noteDuration = 0;
// ------------

int analogPin = A2;
int switchPin = 3;

void setup() {
  Serial.begin(9600);

  pinMode(12, OUTPUT);
  pinMode(11, OUTPUT);
  pinMode(10, OUTPUT);
  pinMode(9, OUTPUT);
  pinMode(switchPin, INPUT);
}

void loop() {

  int lightValue = analogRead(analogPin);
  int switchState = digitalRead(switchPin);
  
  // turn on the red, blue, green LEDs
  if (lightValue < 400) {
    digitalWrite(12, HIGH);
    digitalWrite(11, HIGH);
    digitalWrite(10, HIGH);
  } else {
    // turn off the 3 LEDs
    digitalWrite(12, LOW);
    digitalWrite(11, LOW);
    digitalWrite(10, LOW);
  }

  if (switchState == HIGH) {
    // turn on the yellow LED
    digitalWrite(9, HIGH);
    // play happy birthday
    for (int thisNote = 0; thisNote < notes * 2; thisNote = thisNote + 2) {
      // Check switch state during playback
      if (digitalRead(switchPin) == LOW) {
        digitalWrite(9, LOW);   // Turn off the LED
        noTone(buzzer);         // Stop any sound
        break;                  // Exit the loop
      }

      divider = melody[thisNote + 1];
      if (divider > 0) {
        noteDuration = (wholenote) / divider;
      } else if (divider < 0) {
        noteDuration = (wholenote) / abs(divider);
        noteDuration *= 1.5;
      }

      tone(buzzer, melody[thisNote], noteDuration * 0.9);
      delay(noteDuration);
      noTone(buzzer);
    }
  } else {
    // turn off the yellow LED
    digitalWrite(9, LOW);
    // stop playing happy birthday
    noTone(buzzer);
  }

  Serial.println(lightValue);
}

Challenges and Future Improvements

I really liked working on this week’s assignment and I am quite happy with the concept as well as the final result. I did try to make the candle (aka the yellow LED) blow-able, by taking the wind-blowing idea from Mustafa, so that when you make a blow, the LED turns off, but I got stuck on seamlessly closing/opening the circuit with wind without using a very lightweight/airy material like aluminium foil (as I didn’t have any with me ), so I let go of that idea in the end. All in all, I am happy I got to further review and practice working with Arduino, and now I am feeling a lot more comfortable working with it!

Concept

I have a little jewelry plate that I try my best to put all my jewelry on at the end of the day so that it is easier to find them next time. However, sometimes I forget to put my jewelry on it, especially my ring, and then I would have to look around to figure out where I put it last time.

This became my inspiration for creating my unusual switch – to remind myself to put my ring where it is supposed to be. When the ring is put on the plate, it closes the switch circuit and turns on the LED bulb!

My hands are involved in operating this switch, but not in the traditional sense, as it is the ring that is acting as the switch and is the main object, so I decided to continue with the idea.

Video demonstration

Implementation

I mainly referred to the switch circuit we saw in class to build my circuit. There are 2 exposed wires on the plate that are not touching. One of the wires is connected to the digital pin 2 and pin 2 is connected to GND with a 10k resistor in between. The other wire is connected to 5V. When the copper ring is put on the 2 wires, it acts as a conductor and closes the circuit.

digitalRead is used on pin 2 to detect when the switch circuit is open or closed. When the circuit is closed (ring is on the plate covering both wires), the LED lights up, and vice versa.

const int switchPin = 2;
const int ledPin = 13; 

void setup() {
  pinMode(switchPin, INPUT);
  pinMode(ledPin, OUTPUT);
  Serial.begin(9600);
}

void loop() {
  // read switch state
  int switchState = digitalRead(switchPin);
  if (switchState == HIGH) {
    // turn the LED on when switch is closed
    digitalWrite(LED_BUILTIN, HIGH);
    Serial.println("Ring detected! Circuit closed.");
  } else {
    // turn the LED off when switch is open
    digitalWrite(LED_BUILTIN, LOW);
    Serial.println("No ring detected. Circuit open.");
  }
  delay(100);
}

Challenges and Future Improvements

This week’s project helped me to get a lot more comfortable with building circuits and working with Arduino. I still had to refer back to the schemes we saw in class but I am sure I will get confident to build circuits on my own as we progress.

One of the mistakes I did this week was getting straight into building my circuit before reading again the project description. I glossed over the fact that we had to use digitalRead to get the state of the switch and do something with it. I initially built a circuit with a 330 resistor and an LED that simply closes when the ring is placed and causes the LED to light up, so initially there was no switch at all . After reading through the requirement, I modified the circuit such that there are 2 segments to the circuit: 1) LED circuit that is closed, and 2) switch circuit connected to 5V and a digital pin with a 10k resistor in between that is open and gets closed with the ring. Whether the LED lights up or not depends on the open/closed state of the switch circuit.

As I now write this documentation, I wonder if it would have been more intuitive if the LED turns off when the switch is on (when the ring is placed on the plate), so as to have the lit-up LED as a reminder to put my ring on the plate. Though this change does not alter the concept of the project, I think it makes it more meaningful in terms of usability and intentions. I hope to be more intentional with my future projects; to make things that are well thought out and build things that do actions for a reason, not just for the sake of it.