Final Project: Pet-A-Butterfly

Concept and Inspiration

It was easy to come up with an overall abstract direction for the final project as I had set my mind early on to revisit the butterfly motif in different ways throughout the course. As for the actual concept behind the final, I wanted to experiment with something more centered around interaction and visuals in an installation-like manner and explore a new mode of design that I have not tapped into yet. After iterations and iterations of deliberations and conversations with Professor Aaron, I settled on creating a small piece centered around a mechanical butterfly that flutters when touched. The butterfly would be mounted atop a physical canvas, onto which p5-generated animations would be mapped and projected. The idea was to create a cohesive piece, with the hardware and the software working hand-in-hand to bring some life to a butterfly.

The mechanical butterfly is constructed out of two servo motors, with one moving at an angle supplementary to that of the other. The butterfly wings are printed on paper, laminated, cut, and attached to the servo motor blades. The butterfly “senses” touch through its antennas. My mechanical butterfly’s antennas are made of wires stripped, twisted to shape, and connected to a touch capacitive sensor. I used a box, which I wrapped with multiple layers of white paper and decorated with flowers (to look like the butterfly is in a flower field), with an opening for the Arduino and the circuit.

Interaction Design

For this piece, I wanted to emphasize designing a relatively simple interaction optimally well. The name I chose for the piece, “Pet-A-Butterfly” would be displayed to the user and would act as a signifier to touch the butterfly. The placement of the butterfly antennas opposite the user is intentional to maximize the probability that a user strokes the wires in the chance that they do not realize the antennas are to be touched. The user can interact with the piece by touching the butterfly antennas. Once touched, the butterfly wings flap, and a kaleidoscope of small p5-generated/projected butterflies emerge from beneath the butterfly and move outward in a synergistic, spiral motion.

Implementation

Arduino

The Arduino program gets the input from the sensor through the

Below is the full Arduino sketch: 

/*************************************************** 
  This is a library for the CAP1188 I2C/SPI 8-chan Capacitive Sensor

  Designed specifically to work with the CAP1188 sensor from Adafruit
  ----> https://www.adafruit.com/products/1602

  These sensors use I2C/SPI to communicate, 2+ pins are required to  
  interface
  Adafruit invests time and resources providing this open source code, 
  please support Adafruit and open-source hardware by purchasing 
  products from Adafruit!

  Written by Limor Fried/Ladyada for Adafruit Industries.  
  BSD license, all text above must be included in any redistribution
 ****************************************************/
 
#include <Wire.h>
#include <SPI.h>
#include <Adafruit_CAP1188.h>
#include <Servo.h>

// Reset Pin is used for I2C or SPI
#define CAP1188_RESET  9

// CS pin is used for software or hardware SPI
#define CAP1188_CS  10

// These are defined for software SPI, for hardware SPI, check your 
// board's SPI pins in the Arduino documentation
#define CAP1188_MOSI  11
#define CAP1188_MISO  12
#define CAP1188_CLK  13

#define CAP1188_SENSITIVITY 0x1F
// For I2C, connect SDA to your Arduino's SDA pin, SCL to SCL pin
// On UNO/Duemilanove/etc, SDA == Analog 4, SCL == Analog 5
// On Leonardo/Micro, SDA == Digital 2, SCL == Digital 3
// On Mega/ADK/Due, SDA == Digital 20, SCL == Digital 21

// Use I2C, no reset pin!
Adafruit_CAP1188 cap = Adafruit_CAP1188();

// Or...Use I2C, with reset pin
//Adafruit_CAP1188 cap = Adafruit_CAP1188(CAP1188_RESET);

// Or... Hardware SPI, CS pin & reset pin 
// Adafruit_CAP1188 cap = Adafruit_CAP1188(CAP1188_CS, CAP1188_RESET);

// Or.. Software SPI: clock, miso, mosi, cs, reset
//Adafruit_CAP1188 cap = Adafruit_CAP1188(CAP1188_CLK, CAP1188_MISO, CAP1188_MOSI, CAP1188_CS, CAP1188_RESET);

// make a servo object
Servo servoRight;
Servo servoLeft;

// servo pposition 
int position=50; 
// direction of wing movement
boolean direction = true;  

unsigned long previousMillis = 0;
const long interval = 100;  // interval between each wing flap in milliseconds

void setup() {
  Serial.begin(9600);
  Serial.println("CAP1188 test!");

  // Initialize the sensor, if using i2c you can pass in the i2c address
  if (!cap.begin(0x28)) {
  if (!cap.begin()) {
    while (1);
  }
  cap.writeRegister(CAP1188_SENSITIVITY, 0x5F);
  // attach the servo to pin 9 
  servoRight.attach(11); 
  servoLeft.attach(5); 
  // write the position 
  servoRight.write(180- position);
  servoLeft.write(position);
  // // start the handshake
  while (Serial.available() <= 0) {
    digitalWrite(LED_BUILTIN, HIGH); // on/blink while waiting for serial data
    Serial.println("0"); // send a starting message
    delay(300);            // wait 1/3 second
    digitalWrite(LED_BUILTIN, LOW);
    delay(50);
  }
}
}

void loop() {
  // wait for data from p5 before doing something

  while (Serial.available()) {
    uint8_t touched = cap.touched();
    int isMoving = Serial.parseInt(); // check if butterfly is still moving 
    Serial.print(touched); 
    Serial.print(',');
    if (isMoving == 1) {
        unsigned long currentMillis = millis();
        // check if it's time to update the wing position
        if (currentMillis - previousMillis >= interval) {
            // move servos to simulate wing flapping motion
            if (direction) {
                position += 10;
                if (position >= 50) { // flip direction twhen max angle is reached
                    direction = false;
                }
            } else {
                position -= 10;
                if (position <= 25) {
                    direction = true;
                }
            }
            // move servos in opposite directions 
            servoRight.write(180-position);
            servoLeft.write(position);

            previousMillis = currentMillis;
          }
      };
    Serial.println(position); // send servc position to p5 sketch 
  }
  digitalWrite(LED_BUILTIN, LOW);
}

P5

The p5 sketch is mainly responsible for triggering the animation of the smaller butterflies and for performing projection mapping which is essential for ensuring that the canvas of the sketch can always be calibrated to fit the surface of the physical box. For the latter, I made use of the

function setup() {
  createCanvas(windowWidth, windowHeight, WEBGL);
  
  // create mapper object
  pMapper = createProjectionMapper(this);
  quadMap = pMapper.createQuadMap(mapWidth, mapHeight);
  
  // loads calibration in the "maps" directory
  pMapper.load("maps/map.json");

  // initialize objects
  bigButterfly = new Butterfly(
    centerX,
    centerY,
    null,
    null,
    null,
    null,
    null,
    false,
    false,
    null,
    null,
    false
  ); // dummy butterfly object simulating the state of the physical butterfly 
  interaction = new Interaction(); // an interaction object that handles all interaction-related animations 
  
  // play background music in loop
  backgroundMusic.loop(); 
}

To implement the animation, I created an

// class that controls the animation trigger by the interaction 
class Interaction {
  constructor() {
    this.bigButterfly = bigButterfly; // the butterfly object containing information about the physical butterfly in the center
    this.smallButterflies = []; // array that stores the smaller butterflies whose animation is triggered and displayed when signal is received from arduion
    this.numButterflies = 100; // number of small butterflies 
    this.inTheCenter = this.numButterflies; // number of butterflies in the center 
    // initialize randomly colored butterfly objects and append to the smallButterflies array 
    let randomNum;
    for (let i = 0; i < this.numButterflies; i++) {
      randomNum = random([1, 2, 3]);
      if (randomNum == 1) {
        this.smallButterflies.push(
          new SmallButterfly(
            centerX,
            centerY,
            smallButterflySpritesheet2,
            4,
            10,
            0,
            3,
            true,
            false,
            null,
            null,
            false
          )
        );
      }
      else if (randomNum == 2){
        this.smallButterflies.push(
        new SmallButterfly(
            centerX,
            centerY,
            smallButterflySpritesheet1,
            4,
            10,
            0,
            5,
            true,
            false,
            null,
            null,
            false
          )
        ); 
      }
      else if (randomNum == 3){
        this.smallButterflies.push(
          new SmallButterfly(
              centerX,
              centerY,
              smallButterflySpritesheet3,
              4,
              10,
              0,
              13,
              true,
              false,
              null,
              null,
              false
            )
          ); 
      }
    }
  }

  play(pg) {
    /* function that controls that controls the sketch 
    display -> passed to mappper object's displaySketch function 
    */
    pg.clear();
    pg.push();
    pg.background(color("#B2D2A2"));
    // display instructions text only before connecting to serial 
    if (textShow){
        pg.push()
        pg.fill(color("#2c4c3b"))
        pg.textFont(font); 
        pg.textAlign(CENTER);
        pg.textSize(16)
        pg.text(textString, centerX+20, centerY+150);
        pg.pop()
    }

    // display butterflies
    for (let i = 0; i < interaction.numButterflies; i++) {
      pg.push();
      let angle = radians(180); 
      pg.translate(
        interaction.smallButterflies[i].x,
        interaction.smallButterflies[i].y
      );
      pg.rotate(angle); // rotate butterflies 180 degrees --> better visibility for the user 
      if (interaction.smallButterflies[i].moving) { // display the small butterfly if it's moving 
        pg.image(interaction.smallButterflies[i].show(), 0, 0, 40, 40);
        interaction.smallButterflies[i].move(); // update movement of butterflies 
      }
      pg.pop();
    }

    pg.push();
    
    // ellipse enclosing projected surface area of the physical butterfly
    pg.fill(color("#B2D2A4"));
    // pg.fill(color("black"))
    pg.noStroke();
    // pg.ellipse(215, 180, butterflyWidth, butterflyHeight)
    pg.pop();

    // stop butterfly from moving after a set time has elapsed and only if the 
    // position of the servo is in the right direction 
    if (millis() - movementTime >= interval && servoPos == 50) {
      bigButterfly.moving = false;
    }
  }
}

The movement of the butterflies follows a spiral-like path, originating outward and around the physical butterfly. It is implemented in a method of the

move() {
  // update the step of the animation 
  if (frameCount % this.animationSpeed == 0) {
    this.step = (this.step + this.animationDir * 1) % this.numSpritesCol;
  }

  // control the direction of the sprite movement as spritesheet must be traversed back and forth to display correct movement 
  if (this.step == 0) {
    this.animationDir = 1;
  } else if (this.step == this.numSpritesCol - 1) {
    this.animationDir = -1;
  }
    // update the x and y positions based on the current angle and radius 
    this.x = centerX + cos(this.angle)* this.radius + random(-0.5,0.5); 
    this.y = centerY + sin(this.angle)* this.radius + random(-0.5,0.5);
    this.angle += this.angleSpeed; // increment angle to move the butterfly along a circular path 
    this.radius += this.radiusSpeed; // increment the radius to move the butterfly outward 

  
  // move back to center if butterfly exceeds the bounds 
  if (this.x < minX || this.y < minY || this.x > maxX || this.y > maxY) {
    this.x = centerX;
    this.y = centerY;
    interaction.inTheCenter += 1; // butterfly is now counted as being in the center
    this.moving = false; // stop butterfly from moving 

  // update angle and radius speed parameters to random values 
    this.angleSpeed = random(-0.02, 0.02);
    this.radiusSpeed = random(0.5,1.2);
    this.angle = 0; 
    this.radius = 0; 
  }
  // flip butterfly direction depending on location in the sketch 
  if (this.x < centerX && this.sprites.length > 1) {
    this.dir = 1;
  } else {
    this.dir = 0;
  }
}

When the p5 sketch receives the touch state and servo position from Arduino, it sets the

function readSerial(data) {
  ////////////////////////////////////
  //READ FROM ARDUINO HERE
  ////////////////////////////////////

  if (data != null) {
    // make sure there is actually a message
    let fromArduino = data;
    // if the right length, then proceed
    if (fromArduino.length > 0) {
      // get value only when data sent from arduino is greater than 0
      fromArduino = split(trim(fromArduino), ",");
      touchSensorVal = int(fromArduino[0]); // get touch sensor val
      servoPos = int(fromArduino[1]); // get servo pos
      if (touchSensorVal >= 1) { // if sensor is touched, set the bigButterfly moving attribut to true 
        interaction.bigButterfly.moving = true;

        movementTime = millis(); // record starting movement time
        interaction.inTheCenter = 0;
        // move smaller butterflies 
        for (let i = 0; i < interaction.numButterflies; i++) {
          interaction.smallButterflies[i].moving = true;
        }
      }
    }

    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    let sendToArduino;
    if (interaction.bigButterfly.moving == true) {
      sendToArduino = 1 + "\n"; // send 1 to Arduino if the butterfly is moving 
    } else {
      sendToArduino = 0 + "\n"; // send 0 to Arduino if the butterfly is done with its animation 
    }
    writeSerial(sendToArduino);
  }
}

Here is an embedding of the full sketch (you can press the ‘d’ key to play the animation without the signal from Arduino):   

 

Reflections and Parts I am Proud of

My biggest concern going into this, especially as I was going to employ projection mapping, was that I would be unable to align the p5 sketch and the physical butterfly together in a cohesive manner that still looks visually pleasing. I am, thus, proud that the final product resembles what I had envisioned. I also spent a lot of time thinking of the proper mechanism to automate the wing flapping motion and where/how to place the wings. I experimented with a lot of methods, such as attaching a vertical straw/wooden stick from the middle of the wings to the servo blades, and tugging on the wings when moving down to move the wings up and down. When that proved to be unhelpful, I switched to simply attaching each wing to a blade, which, in hindsight, should have been what I experimented with first. I also love the detail of having the connection between the butterfly and the sensor be through antenna-looking sensors, resembling the sense mechanisms of an actual butterfly (thanks to Professor Aaron for pointing this out). Finally, I am proud that I managed to properly calibrate the sensitivity of the touch sensor, as it initially was too sensitive, sometimes even detecting signals even when untouched. Keeping the sensitivity in check was a major challenge that I thankfully was able to overcome to keep the interaction consistent.

Areas for Future Improvements

I think the project could definitely be enhanced in a lot of ways. Because I spent a lot of time putting the interface together, an area of future improvement could be the p5-generated animation itself. I could have different path movements triggered with every touch, for example. I had initially wanted to map an actual animated butterfly from p5 onto a blank silhouette cutout of a butterfly, controlled by the servos in the same way. Because of difficulties in mapping the software animations to the movement of the hardware, I decided to pivot toward having the central butterfly be completely in hardware form.  One improvement to explore is going in that direction, where I effectively add physical objects, like flowers, on the surface of the box and map simpler, more limited animations onto them.