Author: Melika Shahin

import jp.nyatla.nyar4psg.*;
 
Capture cam;
MultiMarker nya;
Movie movie;
 
void setup() {
  size(640,480,P3D);
  colorMode(RGB, 100);
  println(MultiMarker.VERSION);
  cam=new Capture(this,640,480);
  nya=new MultiMarker(this,width,height,"camera_para.dat",NyAR4PsgConfig.CONFIG_PSG);
  nya.addARMarker(loadImage("ar marker 5.png"),16,10,80);
  movie = new Movie(this, "IMG_2953.mp4");
  movie.loop();
  cam.start();
}
 
void movieEvent(Movie movie) {  
  movie.read();
}
 
 
void draw()
{
  if (cam.available() !=true) {
   
      return;
  }
  cam.read();
  nya.detect(cam);
  background(0);
  nya.drawBackground(cam);
  if((!nya.isExist(0))){
    return;
  }
  nya.beginTransform(0);
  fill(0,0,255);
  translate(25,35,0);
  //box(40);
  rotate(PI);
  image(movie, 0, 0, 50, 70);
  nya.endTransform();
}

While setting up, I placed the tripods as far away from the book as possible, so as to create the illusion that they are not there, and allow the audience to be immersed into the ‘magic’. I accidentally printed the drawings in black and white, and then decided that it looked better this way, because it would make the style of the different drawings match more. I also printed a specific quote for each page, that created sort of story behind each drawing, to relate it to magic.

I user tested my product and found that everything was working fine, except that the projection wasn’t going white when the marker was not detected, and instead showed what the camera was picking up continuously.

This was something that I was able to fix easily, with adding just one more line of code, but it felt great because a few weeks ago, I would have not at all been able to navigate my way through code, and now I could.

After fixing the issue, I did another user test;

This time everything worked perfectly fine. However I felt that I want it to be a bit longer, and have some sort of finish, since people kept going to the next page, so I decided to add one more animation, that would make it a bit longer, create a sense of a finish, and bring it all back to the theme of the book; magic.

I displayed this one in color, and without a background drawing, so as to make it stand out as the ending (animation of a pickup truck leaving a note that says “It’s magic” before driving away, as though to say how the book operates).

import processing.video.*;
import jp.nyatla.nyar4psg.*;

Capture cam;
MultiMarker nya;
Movie[] movies;
//int [] objects = {1,2,3,4};
boolean [] playMovie = {false, false, false, false};
int whichMovie = 0;
//int i = objects.length;
float mWidth, mHeight;
boolean showCameraFeed=true;
void setup() {
  fullScreen(P3D, 2);
  //size(640,480,P3D);

  println(MultiMarker.VERSION);
  cam=new Capture(this, 640, 480);
  nya=new MultiMarker(this, 640, 480, "camera_para.dat", NyAR4PsgConfig.CONFIG_PSG);
  nya.addARMarker(loadImage("IMG_2982.PNG"), 16, 10, 80); //id = 0
  nya.addARMarker(loadImage("final marker 2.PNG"), 16, 10, 80);
  nya.addARMarker(loadImage("final marker 3.PNG"), 16, 10, 80);
  nya.addARMarker(loadImage("Untitled_Artwork 5.PNG"), 16, 10, 80);
  movies = new Movie[4];
  movies[0] = new Movie(this, "Untitled_Artwork 4 copy 4.mp4");
  movies[1] = new Movie(this, "IMG_2984.mp4");
  movies[2] = new Movie(this, "final video 3.mp4");
  movies[3] = new Movie(this, "Untitled_Artwork 4.mp4");

   
  
  for (int k = 0; k<movies.length; k++) {
    movies[k].loop();
  }
  cam.start();
}

void movieEvent(Movie movie) {  
  movie.read();
}


void draw()
{
  if (cam.available() !=true) {

    return;
  }
  cam.read();
  nya.detect(cam);
  background(255);
  if (showCameraFeed)
    nya.drawBackground(cam);
  for (int i = 0; i<movies.length; i++) {
    if ((!nya.isExist(i))) {
      playMovie[i]=false;
        background(255);

      continue;
    }
    //nya.beginTransform(i);
    //fill(0, 0, 255);
    ////translate(30,37,0);
    ////box(40);
    //rotate(PI);

    //nya.endTransform();
    whichMovie=i;
    playMovie[i]=true;
    println("marker "+whichMovie+" found!");
  }

  if (playMovie[whichMovie]==true) {
    pushMatrix();
    translate(width, 0);
    rotate(PI/2);

    image(movies[whichMovie], 0, 0, height, width);

    popMatrix();
  }
}

void keyPressed() {
  showCameraFeed=!showCameraFeed;
}

The showcase was a lot of fun because I got to see people interact with my product. I loved that it made them smile. Several of the people actually jumped back when they saw the drawings were moving, which was so entertaining to watch. People kept looking around to find out how it’s working, and the funny thing was that a few of them looked directly at the projector and camera but did not notice the drawing was being projected through them. When they would ask me how it works, I would tell them it will be revealed at the end (the note that said “it’s magic!” – which Aaron said was cheesy, but I find hilarious – before telling them how it really worked).

I was a bit worried that when turning the page, people might hold the place where the marker (page number) is, and the animation wouldn’t appear, but thankfully out of everyone maybe only two people did this, who I had to explain it to. It was really interesting to see how people interacted with it. Some people were not sure if they should open it, others kept flipping the pages back and forth to see if the animations would still appear on each drawing, which they did, some people really spent time taking in the quotes, while others were too distracted by the drawing to read the quotes, some people kept touching the number because they thought it’s some sort of button or pressure sensor, and some people kept waving their hand over the drawing to see what would happen. It also made me happy how some people took videos of it, or went to get someone else to come see it as well. I was also happy because a friend of mine who basically does nothing but criticize me was very amused by it, and was finally impressed!

*ps the person in this photo isn’t the friend I’m talking about

I’m really glad that my product had the effect on people that I wanted. I can’t believe how far I’ve come considering I could not understand any coding before this semester. Coding was such a foreign language to me and I’ve learnt a lot. In the beginning of this semester I could not even understand what the example code was saying, but now I can go through this entire code and know what’s going on and what each line is doing. I’m really thankful for your patience and support Aaron. And to the entire class and instructors, because everyone was always really supportive. Thank you.

I user tested my product and found that everything was working fine, except that the projection wasn’t going white when the marker was not detected, and instead showed what the camera was picking up continuously.

This was something that I was able to fix easily, with adding just one more line of code, but it felt great because a few weeks ago, I would have not at all been able to navigate my way through code, and now I could.

After fixing the issue, I did another user test;

This time everything worked perfectly fine. However I felt that I want it to be a bit longer, and have some sort of finish, since people kept going to the next page, so I decided to add one more animation, that would make it a bit longer, create a sense of a finish, and bring it all back to the theme of the book; magic.

I’m glad I added the last animation because I think it really helped create an ending. A lot of little challenges arised during this project and I’m so happy that it turned out how I envisioned it.

import processing.video.*;
import jp.nyatla.nyar4psg.*;

Capture cam;
MultiMarker nya;
Movie movie;

void setup() {
  size(640,480,P3D);
  colorMode(RGB, 100);
  println(MultiMarker.VERSION);
  cam=new Capture(this,640,480);
  nya=new MultiMarker(this,width,height,"camera_para.dat",NyAR4PsgConfig.CONFIG_PSG);
  nya.addARMarker(loadImage("ar marker 5.png"),16,10,80);
  movie = new Movie(this, "IMG_2953.mp4");
  movie.loop();
  cam.start();
}

void movieEvent(Movie movie) {  
  movie.read();
}


void draw()
{
  if (cam.available() !=true) {
   
      return;
  }
  cam.read();
  nya.detect(cam);
  background(0);
  nya.drawBackground(cam);
  if((!nya.isExist(0))){
    return;
  }
  nya.beginTransform(0);
  fill(0,0,255);
  translate(25,35,0);
  //box(40);
  rotate(PI);
  image(movie, 0, 0, 50, 70);
  nya.endTransform();
}

I started with the idea of using AR markers by placing them on the side of the page and having the drawing projected onto each page but then I decided that I really want to focus on creating the feeling of each of the drawings coming to life, so I was determined to find a way to use my drawing itself as the marker. Through research and trial and error, I finally got it working properly. I tried out different things and learnt that AR markers actually don’t need to be in square form, only the edge needs to be a square. So I inserted the final drawing marker into the data folder and changed the code, edge percentage, and positioning, and size. I used example AR code that displayed a cube on top of the marker, and changed the code so as to have a video file play on top of the marker.

However I need to now test this out with a webcam and projector to see if it works to have a full drawing animation projected on top of the drawing or if I need to create a different video file with only the moving elements to be projected. I plan to include around 5 drawings and have them on different pages of a hard paper book to avoid bending of the paper, which would affect the marker detection and projection.

Storytelling through interactive art

My final project is going to help me display my art in a creative way. I want to make each drawing come to life with an animation. I floated around different ideas and versions but my finalized idea is to have a physical book, with my drawings printed on the different pages, and using an AR marker, I’m going to have it so that every page that the viewer turns, an animation plays on the drawing that brings it to life in some way, and be a way of storytelling. I’m also considering having a character that walks on the page and sort of adds to the interaction, by having the character react to each page being turned. This might be a bit too ambitious though but we’ll see. I’m quite excited about this project but also anxious about the amount of work it will require and if it turns out how I want it to.

The possibilities of creating interactive art continues to amaze me. It was interesting to see interactive media being made four decades ago. everytime I read about the incorporation of interactive media into performance, I find it very interesting and though provoking. For some reason it reminds me of the movie Step up; revolution. I think the movie was sort of my first encounter with interactive art dance performances that were mind blowing to me. I think performance is something that interactive art and computer vision can really enhance. However, as cool as I find the different examples of computer vision being used, it all sounds very complicated and challenging.

For this week, I thought it would be really cool to make my snake game more advanced by using a joystick to control the movement using both Processing and Arduino. For this I used 4 male and 4 female cables and a joystick. I faced different challenges such as the arduino not transferring to processing or only two sides of the joystick working and it was more challenging than I initially thought, but it I got it working in the end!

Arduino:

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  Serial.println("0,0");
}

void loop() {
  if(Serial.available()>0){
    char inByte=Serial.read();
    int sensor = analogRead(A0);
    delay(0);
    int sensor2 = analogRead(A1);
    delay(0);
    Serial.print(sensor);
    Serial.print(',');
    Serial.println(sensor2);
  }
}

Processing:

int grid = 20; //How big each grid square will be
import processing.serial.*;
PVector food;
int speed = 10;
boolean dead = true;
int highscore = 0;
Snake snake;
Serial myPort;
int xPos=0;
int yPos=0;

void setup() {
  size(500, 500);
  snake = new Snake();
  food = new PVector();
  newFood();
    String portname=Serial.list()[2];
  println(portname);
  myPort = new Serial(this,portname,9600);
  //myPort.clear();
  //myPort.bufferUntil('\n');
}

void draw() {
  background(0,0,50);
  fill(200,200,200);
  if (!dead) {
    
    if (frameCount % speed == 0) {
      snake.update(xPos,yPos);
    }
    snake.show();
    snake.eat();
    fill(200,50,100);
    rect(food.x, food.y, grid, grid);
    textAlign(LEFT);
    textSize(15);
    fill(255);
    text("Score: " + snake.len, 10, 20);
  } else {
    textSize(25);
    textAlign(CENTER, CENTER);
    text("Alien snake Game\nAre you up for the challenge?\nClick to start" + "\nHighscore: " + highscore, width/2, height/2);
  }
}

void newFood() {
  food.x = floor(random(width));
  food.y = floor(random(height));
  fill( random(255), random(255), random(255), random(255));
  food.x = floor(food.x/grid) * grid;
  food.y = floor(food.y/grid) * grid;
}

void mousePressed() {
  if (dead) {
    snake = new Snake();
    newFood();
    speed = 10;
    dead = false;
  }
}

void serialEvent(Serial myPort){
  String s=myPort.readStringUntil('\n');
  s=trim(s);
  if (s!=null){
    int values[]=int(split(s,','));
    if (values.length==2){
      xPos=(int)map(values[0],0,1023,0, width);
      yPos=(int)map(values[1],0,1023,0, height);
    }
  }
  println(xPos+ " "+yPos);
  myPort.write('0');
}

class Snake {
  PVector pos;
  PVector vel;
  ArrayList<PVector> hist;
  int len;
  int moveX = 0;
  int moveY = 0;

  Snake() {
    pos = new PVector(0, 0);
    vel = new PVector();
    hist = new ArrayList<PVector>();
    len = 0;
  }

  void update(int x, int y) {

    if (x>300 && y<300 && y>200 && snake.moveX != 2) {
      snake.vel.x = -1;
      snake.vel.y = 0;
    } else if (x<200 && y<300 && y>200 && snake.moveX != -1) {
      snake.vel.x = 1;
      snake.vel.y = 0;
    } else if (x<300 && x>200 && y<200 && snake.moveY != 2) {
      snake.vel.y = -1;
      snake.vel.x = 0;
    } else if (x<300 && x>200 && y>300 && snake.moveY != -1) {
      snake.vel.y = 1;
      snake.vel.x = 0;
    }

    hist.add(pos.copy());
    pos.x += vel.x*grid;
    pos.y += vel.y*grid;
    moveX = int(vel.x);
    moveY = int(vel.y);

    pos.x = (pos.x + width) % width;
    pos.y = (pos.y + height) % height;

    if (hist.size() > len) {
      hist.remove(0);
    }

    for (PVector p : hist) {
      if (p.x == pos.x && p.y == pos.y) {
        dead = true;
        if (len > highscore) highscore = len;
      }
    }
  }

  void eat() {
    if (pos.x == food.x && pos.y == food.y) {
      len++;
      if (speed > 5) speed--;
      newFood();
    }
  }

  void show() {
    noStroke();
    fill( random(255), random(255), random(255), random(255));
    ;
    rect(pos.x, pos.y, grid, grid);
    for (PVector p : hist) {
      rect(p.x, p.y, grid, grid);
    }
  }
}
void keyPressed() {
  if (keyCode == LEFT && snake.moveX != 1) {
    snake.vel.x = -1;
    snake.vel.y = 0;
  } else if (keyCode == RIGHT && snake.moveX != -1) {
    snake.vel.x = 1;
    snake.vel.y = 0;
  } else if (keyCode == UP && snake.moveY != 1) {
    snake.vel.y = -1;
    snake.vel.x = 0;
  } else if (keyCode == DOWN && snake.moveY != -1) {
    snake.vel.y = 1;
    snake.vel.x = 0;
  }
}

I initially took IM for 2 reasons; to get a bit familiar with coding incase I need to adjust parts of my website, and because all my IM major friends told me I’d enjoy it, which I did. But having looked at and understood how such interesting things can and are made with coding has been really interesting and thought provoking for me. From the games that I used to think only professionals could make, to mind blowing interactive art.

In high school I took a course that taught us about the impact of computing on the world, from RFID to the Internet and security. But in this course I learnt to actually make things and work with computing, and actually do something. Coding is still very challenging for me to wrap my head around, but I no longer see it as a foreign skill that only certain people can do.

Throughout my childhood, whenever I was asked what I wanted to become, my only answer was “An inventor”. Not a doctor or lawyer or even an artist. Inventor. I was told that is not a job and so I sort of slowly let it go. I still remember when we got our IM kits, and I saw it said “Inventor’s kit”. It felt like destiny.

I still feel a bit intimidated by coding and computing but I definitely feel  it has given me much more room for exploration.

I tried a bunch of different text generation ways. I watched a video on the coding train and now I feel I have a much deeper understanding of text generation. I decided to go with a simple text fade.

float imeInterval;
float timePast;

int textAlpha = 100;
int textFade = 2;
float timeInterval = 2000.0f;
PFont f;
void setup() {
  size(400,400);
   timePast = millis();
   f = createFont("Zapfino", 64);
  
}

void textFade() {
  if (millis() > timeInterval + timePast){
  timePast = millis();
  textFade*= -1;
}
textAlpha += textFade;
}
void draw() {
background(0);
textFade();
textSize(90);
textFont(f,50);
fill(255,255,255, textAlpha);
text("Dream", width/4, height/2);
}

While reading the text talking about waze application, the ideas were interesting to me to read but an application that looks at the traffic and tells you the best route didn’t seem new to me because from what I remember theres an application like this in Iran. This idea of the app becoming more useful the more people were using it was interesting to me but made me wonder if there were privacy issues with it. (The house price increases it talked about would be a perfect example of a privacy issue. Using people’s data against them.) It might be an obvious thing to others but what caught my attention was that at one point referred to this increase in effectiveness with the increase of users as the “network effect”. And it made me think back on the fact that social media networks are very much dependent on this. The best social media platform would be nothing without a network of people. You go on Facebook to connect with other people but if those people weren’t there it would serve no benefit.

I love the statement that design depends on constraints. It was also very interesting to read about origins of plywood, because we are currently making skateboards in another class, which are made of plywood. In the case of fashion vs. disability, I feel that even though it is unfortunately true that disability is seen by some as something to be hidden, and fashion could definitely be a tool to embrace each person’s unique attributes, fashion can also sometimes be too over the top. I’ve had 3 different glasses designs over the years and had never thought of glasses as design meeting disability, but this is definitely true, and each design can make a person be perceived completely differently.