My final project idea was to create a magical book with my drawings on some of the pages, and have the drawings come to life through animation. Since a few people have asked me in the showcase, I came up with this idea mainly because of the first drawing that I used. It’s a pen sketch of one of the rooms in the Shakespeare and company bookstore, which I visited in Paris. I did my class research on the bookstore, and so found that artists, writers, and travelers were free to stay at the bookstore and sleep in the beds as long as they wished for, as along as they read one book a day, wrote a one page autobiography for the bookstore’s archives, and helped out a little around the shop. Another thing about the bookstore was that people would leave notes in a box, that the book-keepers would place into random books, to be found by readers. So looking at this drawing always made me imagine the people who stayed there, and the books they read. It all seemed very magical to me, so I decided to bring the drawing to life, with stop motion animation. I’ve been intrigued by storytelling through art lately, so this was the perfect way to experiment with it.
Disclaimer: as you’ll see in this project, I’m a big Harry Potter fan.
import jp.nyatla.nyar4psg.*;
Capture cam;
MultiMarker nya;
Movie movie;
void setup() {
size(640,480,P3D);
colorMode(RGB, 100);
println(MultiMarker.VERSION);
cam=new Capture(this,640,480);
nya=new MultiMarker(this,width,height,"camera_para.dat",NyAR4PsgConfig.CONFIG_PSG);
nya.addARMarker(loadImage("ar marker 5.png"),16,10,80);
movie = new Movie(this, "IMG_2953.mp4");
movie.loop();
cam.start();
}
void movieEvent(Movie movie) {
movie.read();
}
void draw()
{
if (cam.available() !=true) {
return;
}
cam.read();
nya.detect(cam);
background(0);
nya.drawBackground(cam);
if((!nya.isExist(0))){
return;
}
nya.beginTransform(0);
fill(0,0,255);
translate(25,35,0);
//box(40);
rotate(PI);
image(movie, 0, 0, 50, 70);
nya.endTransform();
}
I started with the idea of using AR markers by placing them on the side of the page and having the drawing projected onto each page but then I decided that I really want to focus on creating the feeling of each of the drawings coming to life, so I was determined to find a way to use my drawing itself as the marker. Through research and trial and error, I finally got it working properly.
I tried out different things and learnt that AR markers actually don’t need to be in square form, only the edge needs to be a square. So I inserted the final drawing marker into the data folder and changed the code, edge percentage, and positioning, and size. I used example AR code that displayed a cube on top of the marker, and changed the code so as to have a video file play on top of the marker. While this worked perfectly fine on the screen, when projected, I found out there was a big issue I hadn’t considered. That if the video was projected on the same place that the camera was facing, the camera would pick up the projection and so the projector would project that, and it created an endless loop of a mirror effect. In order to fix this, I decided to have the projector project on the drawing on one page, and the camera pick up the marker on the other page, which would be either a drawing or a page number. I eventually went with the page number so as to not distract from the animated drawings.
Having the several drawings was much more complicated than having just one drawing, so I tried different methods, such as arraylist, array, and for loop. It feels amazing to be able to go through the code and actually understand what everything is doing, because a few months ago coding was so foreign to me.
While setting up, I placed the tripods as far away from the book as possible, so as to create the illusion that they are not there, and allow the audience to be immersed into the ‘magic’. I accidentally printed the drawings in black and white, and then decided that it looked better this way, because it would make the style of the different drawings match more. I also printed a specific quote for each page, that created sort of story behind each drawing, to relate it to magic.
I user tested my product and found that everything was working fine, except that the projection wasn’t going white when the marker was not detected, and instead showed what the camera was picking up continuously.
This was something that I was able to fix easily, with adding just one more line of code, but it felt great because a few weeks ago, I would have not at all been able to navigate my way through code, and now I could.
After fixing the issue, I did another user test;
This time everything worked perfectly fine. However I felt that I want it to be a bit longer, and have some sort of finish, since people kept going to the next page, so I decided to add one more animation, that would make it a bit longer, create a sense of a finish, and bring it all back to the theme of the book; magic.
I displayed this one in color, and without a background drawing, so as to make it stand out as the ending (animation of a pickup truck leaving a note that says “It’s magic” before driving away, as though to say how the book operates).
import processing.video.*;
import jp.nyatla.nyar4psg.*;
Capture cam;
MultiMarker nya;
Movie[] movies;
//int [] objects = {1,2,3,4};
boolean [] playMovie = {false, false, false, false};
int whichMovie = 0;
//int i = objects.length;
float mWidth, mHeight;
boolean showCameraFeed=true;
void setup() {
fullScreen(P3D, 2);
//size(640,480,P3D);
println(MultiMarker.VERSION);
cam=new Capture(this, 640, 480);
nya=new MultiMarker(this, 640, 480, "camera_para.dat", NyAR4PsgConfig.CONFIG_PSG);
nya.addARMarker(loadImage("IMG_2982.PNG"), 16, 10, 80); //id = 0
nya.addARMarker(loadImage("final marker 2.PNG"), 16, 10, 80);
nya.addARMarker(loadImage("final marker 3.PNG"), 16, 10, 80);
nya.addARMarker(loadImage("Untitled_Artwork 5.PNG"), 16, 10, 80);
movies = new Movie[4];
movies[0] = new Movie(this, "Untitled_Artwork 4 copy 4.mp4");
movies[1] = new Movie(this, "IMG_2984.mp4");
movies[2] = new Movie(this, "final video 3.mp4");
movies[3] = new Movie(this, "Untitled_Artwork 4.mp4");
for (int k = 0; k<movies.length; k++) {
movies[k].loop();
}
cam.start();
}
void movieEvent(Movie movie) {
movie.read();
}
void draw()
{
if (cam.available() !=true) {
return;
}
cam.read();
nya.detect(cam);
background(255);
if (showCameraFeed)
nya.drawBackground(cam);
for (int i = 0; i<movies.length; i++) {
if ((!nya.isExist(i))) {
playMovie[i]=false;
background(255);
continue;
}
//nya.beginTransform(i);
//fill(0, 0, 255);
////translate(30,37,0);
////box(40);
//rotate(PI);
//nya.endTransform();
whichMovie=i;
playMovie[i]=true;
println("marker "+whichMovie+" found!");
}
if (playMovie[whichMovie]==true) {
pushMatrix();
translate(width, 0);
rotate(PI/2);
image(movies[whichMovie], 0, 0, height, width);
popMatrix();
}
}
void keyPressed() {
showCameraFeed=!showCameraFeed;
}
The showcase was a lot of fun because I got to see people interact with my product. I loved that it made them smile. Several of the people actually jumped back when they saw the drawings were moving, which was so entertaining to watch. People kept looking around to find out how it’s working, and the funny thing was that a few of them looked directly at the projector and camera but did not notice the drawing was being projected through them. When they would ask me how it works, I would tell them it will be revealed at the end (the note that said “it’s magic!” – which Aaron said was cheesy, but I find hilarious – before telling them how it really worked).
I was a bit worried that when turning the page, people might hold the place where the marker (page number) is, and the animation wouldn’t appear, but thankfully out of everyone maybe only two people did this, who I had to explain it to. It was really interesting to see how people interacted with it. Some people were not sure if they should open it, others kept flipping the pages back and forth to see if the animations would still appear on each drawing, which they did, some people really spent time taking in the quotes, while others were too distracted by the drawing to read the quotes, some people kept touching the number because they thought it’s some sort of button or pressure sensor, and some people kept waving their hand over the drawing to see what would happen. It also made me happy how some people took videos of it, or went to get someone else to come see it as well. I was also happy because a friend of mine who basically does nothing but criticize me was very amused by it, and was finally impressed!
*ps the person in this photo isn’t the friend I’m talking about
I’m really glad that my product had the effect on people that I wanted. I can’t believe how far I’ve come considering I could not understand any coding before this semester. Coding was such a foreign language to me and I’ve learnt a lot. In the beginning of this semester I could not even understand what the example code was saying, but now I can go through this entire code and know what’s going on and what each line is doing. I’m really thankful for your patience and support Aaron. And to the entire class and instructors, because everyone was always really supportive. Thank you.