My final project is going to help me display my art in a creative way. I want to make each drawing come to life with an animation. I floated around different ideas and versions but my finalized idea is to have a physical book, with my drawings printed on the different pages, and using an AR marker, I’m going to have it so that every page that the viewer turns, an animation plays on the drawing that brings it to life in some way, and be a way of storytelling. I’m also considering having a character that walks on the page and sort of adds to the interaction, by having the character react to each page being turned. This might be a bit too ambitious though but we’ll see. I’m quite excited about this project but also anxious about the amount of work it will require and if it turns out how I want it to.
The possibilities of creating interactive art continues to amaze me. It was interesting to see interactive media being made four decades ago. everytime I read about the incorporation of interactive media into performance, I find it very interesting and though provoking. For some reason it reminds me of the movie Step up; revolution. I think the movie was sort of my first encounter with interactive art dance performances that were mind blowing to me. I think performance is something that interactive art and computer vision can really enhance. However, as cool as I find the different examples of computer vision being used, it all sounds very complicated and challenging.
My main takeaway of this reading is that since the beginning of computer interactive art, the computer has been one of its most powerful tools. The reading goes through a bit of the history of computer vision and interactive art and it mentions that one of the first computer interactive art performance was made using computer vision. Lately, in the reading, it is also mentioned that experts are still working to improve the algorithms to make computer vision more efficient. Hence, this means that computer vision was a cornerstone technique in interactive art and that there are still things to explore in computer vision.
I also found this reading helpful for our project. I learned that Kinect uses the Infrared technique mention in the reading and that we are going to use detect motion. This reading help me to visualize all the potential techniques that I may use to create a better user experience for my final project
What I tried to do was figure out a way of setting up the Kinect in a way where it would be able to- using brightness/depth thresholding- sense the movement of people (whether they’re in front of the display or simply walking in the background) in order to play the soundtrack or pause it/replay it. I used a sample soundtrack, but I’ll start working on the actual soundtrack today and tomorrow.
One of the challenges is figuring out whether I’ll have two separate soundtracks for the two parts of the exhibit, or simply have one that plays as long as someone is standing in front of the exhibit. I’ll have to do some tests in order to figure this out.
Here is my code so far:
// Credits: Daniel Shiffman, Depth thresholding example
import org.openkinect.freenect.*;
import org.openkinect.processing.*;
import processing.sound.*;
Kinect kinect;
SoundFile soundfile;
// Depth image
PImage depthImg;
//soundtrack playing
boolean playing = false;
// pixels to be shown
int minDepth = 60;
int maxDepth = 860;
// kinect's angle
float angle;
void setup() {
size(1280, 480);
kinect = new Kinect(this);
kinect.initDepth();
angle = kinect.getTilt();
// blank image showing everything
depthImg = new PImage(kinect.width, kinect.height);
//load soundtrack
soundfile = new SoundFile(this, "soundtrack.aiff");
println("SFSampleRate= " + soundfile.sampleRate() + " Hz");
println("SFSamples= " + soundfile.frames() + " samples");
println("SFDuration= " + soundfile.duration() + " seconds");
}
void draw() {
// draw the raw image
image(kinect.getDepthImage(), 0, 0);
// threshold the depth image
int[] rawDepth = kinect.getRawDepth();
int counter =0;
for (int i=0; i < rawDepth.length; i++) {
if (rawDepth[i] >= minDepth && rawDepth[i] <= maxDepth) {
depthImg.pixels[i] = color(255);
counter++;
} else {
depthImg.pixels[i] = color(0);
}
}
if (counter > 10000){
if (!playing)
{
soundfile.play();
playing = true;
}
}
else
{
if (playing)
{
soundfile.stop();
playing = false;
}
}
// Draw the thresholded image
depthImg.updatePixels();
image(depthImg, kinect.width, 0);
fill(0);
text("TILT: " + angle, 10, 20);
text("THRESHOLD: [" + minDepth + ", " + maxDepth + "]", 10, 36);
}
//// Adjust the angle and the depth threshold min and max
//void keyPressed() {
// if (key == CODED) {
// if (keyCode == UP) {
// angle++;
// } else if (keyCode == DOWN) {
// angle--;
// }
// angle = constrain(angle, 0, 30);
// kinect.setTilt(angle);
// } else if (key == 'a') {
// minDepth = constrain(minDepth+10, 0, maxDepth);
// } else if (key == 's') {
// minDepth = constrain(minDepth-10, 0, maxDepth);
// } else if (key == 'z') {
// maxDepth = constrain(maxDepth+10, minDepth, 2047);
// } else if (key =='x') {
// maxDepth = constrain(maxDepth-10, minDepth, 2047);
// }
//}
This thorough article gave me a very good understanding of how a computer is taught to see. Yet, primarily, shifted my attention to how we take our vision and the way we see and react to the world for granted since it’s a result of countless subconscious processes happing in our brain.
I found it very thought-provoking: humans can distinguish dark and light or objects and humans in a blink of an eye, without having to forcefully learn it. The processes are automatic and we cannot influence them, alter them nor stop them. Similarly to the way our brains are wired in to recognize faces – this mechanism is so efficient that we start seeing faces even where there are none. (As I recently learned, this phenomenon is called pareidolia, see a screenshot from a google search below for quick reference).
That’s why it was very interesting to read about how to teach computers all of these things through codes. Not only people programmed computers to “see” but they also alter what, how and when they see it based on the frame differences and pixel comparisons. They set very rigid boundaries of what the computer can and cannot see, which we can’t achieve with a human brain. It’s not even an imitation of how our brains work – it’s a completely different mechanism.
It made me realize that the way we make computers see (as described in the article) can actually serve as a crucial difference between computer “minds” and human minds (just think of the “I’m not the robot” kind of tests that show you a picture that a computer cannot transfer into text- and that I failed a way too many times that I’m slowly starting to doubt my own humanness).
Computer vision seems much harder than the way the article explains. Despite the author breaking down the idea into three elementary techniques used by computer vision, it’s still hard to get a grasp of what is meant by frame differencing, background subtraction, and brightness thresholding. Reading about the idea of how computer vision works is tricky for me (looks like there’s many mathematical equations that aren’t explained to isolate objects), but with a demonstration, it probably won’t seem as hard. Computer vision seems to be very frustrating to work with because there’s many room for error. Regardless, I’m excited to use some computer vision concepts in any interactive projects I might work with since it’s great at tracking people’s activities. Now that I think about it, many of the art gallery places and videos shown in class has a lot to do with computer vision.
Final Project:
For the final project, I worked on the ProcessingJS part of it. It’s basically a class of ripples. For the time being, the ripples only appear when you click on the screen. Other than ripples, I also created one with points coming from where the mouse is pressed. Currently, the way it’s programmed, only one or the other pattern will show up since I’m commenting out code depending on which pattern I want. I am planning on using Arduino to pick which pattern to run when the user plays with the program. It’d probably be mapped with the analog value of a sensor that’s attached to an El wire.
Here are the two patterns so far.
Additionally, I tested out the tilt sensors to see if it would trigger the patterns on Processing. Currently, any tilt will cause a pattern, but it still doesn’t choose decide on which pattern to run. I will test it out more, but I really want to get the El wires working first so I have to wait for it to arrive.
I worked intensively on my final project this weekend and I think I am about 80% done with the project, the rest of what I need to complete being fine tuning animations and creating a few physical components to be used along with my joystick. I initially used a button at the beginning of my creation process, but found that it was not engaging enough. I am now using two force sensors: one for harvesting resources and the other for interacting with the animals.
I plan to put one sensor into a stuffed animal’s head requiring the user to “pet” the animals. In order to harvest materials, users must hit the force sensor, which I plan to hide under a piece of wood, with a mallet I plan to construct.
I have completed the most difficult aspects of the project by coding nearly all the stages and the mechanics and animations. Although some of the timings and animations are not what I intended, I think I can complete this within a few days. I have also completed the Arduino circuit to receive information from the joystick and the two force sensors which are completely usable in the game.
Although I am not incorporating computer vision into my final project, this is still a field of interactive media that I am very interested in. What I found most interesting was that this technology has been used for art since the 1960s/1970s. I thought that computer vision’s use for art and media was relatively new, and that computer vision in general was a field that has only become useful in the past two decades. The reading mentions that its use has been limited to law-enforcement and military purposes, so this really makes me wonder how much computer vision for art would change if this technology was declassified.
I am extremely interested in how computer vision can be further used for media and art, but I cannot help but feel intimidated by the sophistication of it. At least to me, computer vision seems like technology out of a Star Wars movie, but this article allowed me to gain insight into how this technology is not as new as I thought and how it is very accessible to interactive media artists.
For the initial stage of the final project – WaterBox, the main focus for this week was getting the Kinect working for the specific purpose designated towards this project. And, for this project, I have decided to use Kinect v2. After some time trying to figure out how Kinect v2 works, I have encountered several issues that took me few hours to solve.
One of the main problem I encountered was that Kinect v2’s video image had different dimensions compared to the other lenses (the IR, depth, and registered). Since I was looking to use the depth functionality as well as the RGB value of the pixel on different points within the image, my take on using the video image and the depth image was not a viable option – since the index of the depth image did not point towards the same position in the video image. Thus, I had decided to use the RegisteredImage, which combines the RGB and depth view, to analyze the RGB value on a certain pixel.
For both versions, I have set the minimum threshold as 480 and maximum threshold as 830 – but these values are subject to change as I test it on water. Following videos are some of the try-outs and testing with Kinect v2:
The first version is a test to see whether the Kinect accurately captures a certain color value (in this case, when B-value is higher than 200). For the test medium, I have used an empty blue screen on my mobile phone. As you can see in the video above, the Kinect successfully captures the designated values.
import org.openkinect.processing.*;
Kinect2 kinect2;
float minThresh = 360;
float maxThresh = 830;
PImage img, video;
int[][] screen;
void setup() {
size(512, 424);
kinect2 = new Kinect2(this);
kinect2.initRegistered();
kinect2.initDepth();
kinect2.initDevice();
screen = new int[kinect2.depthWidth][kinect2.depthHeight];
for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
screen[x][y] = 0;
}
}
img = createImage(kinect2.depthWidth, kinect2.depthHeight, RGB);
}
void draw() {
background(255);
img.loadPixels();
video = kinect2.getRegisteredImage();
int[] depth = kinect2.getRawDepth();
for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
int index = x + y * kinect2.depthWidth;
int d = depth[index];
if (d > minThresh && d < maxThresh && blue(video.pixels[index]) > 230) {
img.pixels[index] = color(0, 0, blue(video.pixels[index]));
screen[x][y] = 1;
} else {
img.pixels[index] = color(255);
screen[x][y] = 0;
}
}
}
img.updatePixels();
noStroke();
for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
int index = x + y * kinect2.depthWidth;
if (screen[x][y] == 1) {
int mapVal = (int)map(depth[index], minThresh, maxThresh, 255, 0);
fill(mapVal, 0, 0);
ellipse(x, y, 2, 2);
}
}
}
}
Since I have to track where these recognized points are within the canvas, I have created a separate 2-dimensional array that tracks at which x and y coordinates the Kinect recognizes. So, the above video is a test with drawing ellipses where the pixels with blue value higher than 200, and each ellipse has a color gradient based on the depth in between the thresholds. The depth, in the later part of the project, will be used as the value that changes the pitch or speed of the sound created.
import org.openkinect.processing.*;
Kinect2 kinect2;
float minThresh = 360;
float maxThresh = 830;
PImage img, video;
int[][] screen;
void setup() {
size(512, 424);
kinect2 = new Kinect2(this);
kinect2.initRegistered();
kinect2.initDepth();
kinect2.initDevice();
screen = new int[kinect2.depthWidth][kinect2.depthHeight];
for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
screen[x][y] = 0;
}
}
img = createImage(kinect2.depthWidth, kinect2.depthHeight, RGB);
}
void draw() {
background(255);
img.loadPixels();
video = kinect2.getRegisteredImage();
int[] depth = kinect2.getRawDepth();
int countTL = 0;
int countTR = 0;
int countBL = 0;
int countBR = 0;
for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
int index = x + y * kinect2.depthWidth;
int d = depth[index];
if (d > minThresh && d < maxThresh && blue(video.pixels[index]) > 230) {
img.pixels[index] = color(0, 0, blue(video.pixels[index]));
screen[x][y] = 1;
if (x < kinect2.depthWidth/2 && y <kinect2.depthHeight/2) {
countTL++;
fill(0, 255, 0);
rect(0, 0, kinect2.depthWidth/2, kinect2.depthHeight/2);
text("hello", 0, 0);
}
if (x > kinect2.depthWidth/2 && y < kinect2.depthHeight/2) {
countTR++;
fill(0, 0, 255);
rect(kinect2.depthWidth/2, 0, kinect2.depthWidth, kinect2.depthHeight/2);
}
if (x < kinect2.depthWidth/2 && y > kinect2.depthHeight/2) {
countBL++;
fill(255, 0, 0);
rect(0, kinect2.depthHeight/2, kinect2.depthWidth/2, kinect2.depthHeight/2);
}
if (x > kinect2.depthWidth/2 && y > kinect2.depthHeight/2) {
countBR++;
fill(100, 100, 100);
rect(kinect2.depthWidth/2, kinect2.depthHeight/2, kinect2.depthWidth, kinect2.depthHeight);
}
} else {
img.pixels[index] = color(255);
screen[x][y] = 0;
}
}
}
img.updatePixels();
noStroke();
for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
int index = x + y * kinect2.depthWidth;
if (screen[x][y] == 1) {
int mapVal = (int)map(depth[index], minThresh, maxThresh, 255, 0);
fill(mapVal, 0, 0);
ellipse(x, y, 2, 2);
}
}
}
}
For the third version, I wanted to check whether I can track how many pixels are at a certain part within the screen, specifically for this case the sections include Top-Left, Top-Right, Bottom-Left, and Bottom-Right. Here, I am tracking not only where the recognized pixels are located in terms of the screen, but also storing the count of how many recognized pixels are at the position of a certain area.
The next step of the project would be integrating sound and testing on a water surface – which I will need to start building the container.
I feel like this is the breaking point for me. Computer vision is so cool and I want to use it in absolutely everything I do and that’s a horrible mindset. It’s one of those things that should only be used if it’s absolutely essential to the project. If it can be done in another way, do it in another way. But if it’s what makes the project unique or adds a key element, then do it. As I continue working with IM, it’ll be hard not to suppress the urge to incorporate computer vision into everything.
This reading was very technical, but I think it was a good basis for understanding how it works, what the right conditions are for it to work, and other things that we will need to know for going into the very beginnings of computer vision. For example, setting up the physical environment, especially in this sort of festival/installation vibe we have going for the final show. Making sure that the space is high contrast/low noise and is able to communicate properly with the computer will be an essential part of the process.
However, I really want to talk about an experience I had a little less than a year ago at the Museum of Contemporary Art in Montreal. They had recently installed Zoom Pavillion, an interactive art piece by Rafael Lozano-Hemmer and Krzysztof Krzysztof Wodiczko. In this, all of the walls as well as the ceiling were covered in video footage…of yourself. You would walk into this room and cameras would pick up on your face, gestures, gait, a group that you walked in on. Then, the cameras would start pairing you with people to mark your relationship with them. It was absolutely incredible. I walked in with a group of 18 year olds, all of us extremely good friends, so the cameras drew a large square around the entire group. However, as we started exploring the space, it kept us in communication with each other. It noticed that I kept looking at my good friend Adam, so it would draw a line between us and write “friendly”. I would move far away from my other friend and it would write “hostile”.
This was all fun and games, however, until another group of teenagers walked in. At that point, my friend group had gathered in the corner of the room and were making funny faces into the camera. The other teenagers walked in and almost immediately, we had two boxes, each around the groups and a line between us that read, “Potential Friendly”. How had this machine told us that a bunch of people we could easily connect with had entered the room just by observing a few seconds of their movement and interaction with the space? I will never forget this. It was absolutely incredible.
