As automation became one of the primary sector of human technology, computer vision has become an important field in the areas of computer science and interactive media studies. We have continuously seen a rapid development in algorithms to track movements and distinguish a certain specific figure among a crowd of people. As people gained depth knowledge in how computer vision works, more attempts to combine machine learning into the algorithm to automate the process has also been a high-interest for many aspiring computer scientists.
There are great uses with computer vision that can assist our lives, but relying it to replace our decision completely would be a problem – such as using computer vision system to find possible suspects. I was fascinated by the different applications and methods of computer vision, but I could not get rid of the thought of a recent issue on using facial recognition to identify suspected criminals. As Golan Levin says, “the reliability of computer vision algorithms is limited according to the quality of the incoming video scene, and the definition of a scene’s ‘quality’ is determined by the specific algorithms which are used to analyze it” (Levin). Yes, it is a technology that can be used in such cases, but I believe the technology is not just ready for such use to odd out human decision.
The issue of the above article may not be a problem – and could be forgotten in few decades – as our software and hardwares develop to overcome the quality and accuracy issues. However, the issue of “fully-trusting” the result of complex computer vision is a topic that we should continue to question.
I am ashamed to admit that I only barely completed one and a half of the 3 challenges I had set out for myself last week – who knew learning was so difficult? Most of my accomplishments over the weekend consisted of referencing geometric code we created in class and finding a cool replacement for the blur effect, which is the wonderful library – Postfx, which is great for creating graphics. As for the second challenge concerning the Kinect, I watched most of the Daniel Shiffman’s videos on the thing and I intend to start hooking it up today. My last challenge involves connecting the Kinect to the pattern generated by the code, and making it possible for the user’s movements to control the movements of the pattern. I also intend to create a wayyyy cooler color palette.
creds to this github account for having great examples for working with postFX: https://github.com/cansik/processing-postfx/blob/master/examples/SimpleEffect/SimpleEffect.pde
So far for my final project, I worked on the software for the game. I made 3 distinct classes: one for the raindrops, one for a timer and one for the ‘catcher’. I also added a timer onto the screen; however, it is currently counting up and needs to do the opposite. I also worked out how to print out the values for each time a raindrop intersects with the ‘catcher’ which I can further use to build the levels of my game. I also found the images I would like to use in an array to portray the growing flower and need to code this next as opposed to the ‘catcher’.
I found most aspects of coding thus far very difficult as I didn’t understand the logic to use, I solved this by asking a lot of questions from the Unix Lab. I’m still very unsure on how I am going to resolve all the upcoming issues regarding the animation aspect of the game and also connecting the IR camera.
Catcher catcher; // One catcher object
Timer timer; // One timer object
Drop[] drops; // An array of drop objects
int totalDrops = 0;
int numCaught = 0; // Starting a count for the number of caught raindrops
PImage background;
void setup() {
size(480, 270);
background = loadImage("background.jpg"); // Putting in a background
catcher = new Catcher(32); // Create the catcher with a radius of 32
drops = new Drop[1000]; // Create 1000 spots in the array
timer = new Timer(300); // Create a timer that goes off every 300 milliseconds
timer.start(); // Starting the timer
}
void draw() {
background(255);
// Set catcher location
catcher.setLocation(mouseX, height);
// Display the catcher
catcher.display();
// Check the timer
if (timer.isFinished()) {
// Deal with raindrops
// Initialize one drop
drops[totalDrops] = new Drop();
// Increment totalDrops
totalDrops ++ ;
// If we hit the end of the array
if (totalDrops >= drops.length) {
totalDrops = 0; // Start over
}
timer.start();
}
{
text(timer.getTime(), 420, 50);
fill(30);
textSize(30);
// Move and display all drops
for (int i = 0; i < totalDrops; i++ ) {
drops[i].move();
drops[i].display();
if (catcher.intersect(drops[i])) {
drops[i].caught();
numCaught++;
print(numCaught);
}
}
}
}
class Catcher {
float r; // radius
color col; // color
float x, y; // location
Catcher(float tempR) {
r = tempR;
col = color(50, 10, 10, 150);
x = 0;
y = 0;
}
void setLocation(float tempX, float tempY) {
x = tempX;
y = tempY;
}
void display() {
stroke(0);
fill(col);
ellipse(x, y, r*2, r*2);
}
// A function that returns true or false based on
// if the catcher intersects a raindrop
boolean intersect(Drop d) {
// Calculate distance
float distance = dist(x, y, d.x, d.y);
// Compare distance to sum of radii
if (distance < r + d.r) {
return true;
} else {
return false;
}
}
}
class Drop {
float x, y; // Variables for location of raindrop
float speed; // Speed of raindrop
color c;
float r; // Radius of raindrop
Drop() {
r = 8; // All raindrops are the same size
x = random(width); // Start with a random x location
y = -r*4; // Start a little above the window
speed = random(1, 5); // Pick a random speed
c = color(50, 100, 150); // Color
}
// Move the raindrop down
void move() {
// Increment by speed
y += speed;
}
// Check if it hits the bottom
boolean reachedBottom() {
// If we go a little beyond the bottom
if (y > height + r*4) {
return true;
} else {
return false;
}
}
// Display the raindrop
void display() {
// Display the drop
fill(c);
noStroke();
for (int i = 2; i < r; i++ ) {
ellipse(x, y + i*4, i*2, i*2);
}
}
// If the drop is caught
void caught() {
// Stop it from moving by setting speed equal to zero
speed = 0;
// Set the location to somewhere way off-screen
y = -1000;
}
}
class Drop {
float x, y; // Variables for location of raindrop
float speed; // Speed of raindrop
color c;
float r; // Radius of raindrop
Drop() {
r = 8; // All raindrops are the same size
x = random(width); // Start with a random x location
y = -r*4; // Start a little above the window
speed = random(1, 5); // Pick a random speed
c = color(50, 100, 150); // Color
}
// Move the raindrop down
void move() {
// Increment by speed
y += speed;
}
// Check if it hits the bottom
boolean reachedBottom() {
// If we go a little beyond the bottom
if (y > height + r*4) {
return true;
} else {
return false;
}
}
// Display the raindrop
void display() {
// Display the drop
fill(c);
noStroke();
for (int i = 2; i < r; i++ ) {
ellipse(x, y + i*4, i*2, i*2);
}
}
// If the drop is caught
void caught() {
// Stop it from moving by setting speed equal to zero
speed = 0;
// Set the location to somewhere way off-screen
y = -1000;
}
}
class Timer {
int savedTime; // When Timer started
int totalTime; // How long Timer should last
Timer(int tempTotalTime) {
totalTime = tempTotalTime;
}
// Starting the timer
void start() {
// When the timer starts it stores the current time in milliseconds.
savedTime = millis();
}
// The function isFinished() returns true if 5,000 ms have passed.
// The work of the timer is farmed out to this method.
boolean isFinished() {
// Check how much time has passed
int passedTime = millis()- savedTime;
if (passedTime > totalTime) {
return true;
} else {
return false;
}
}
int getTime() {
return millis()/600;
}
}
Next, I’m going to be working on using the IR camera and learning how to program it with the interaction I would like.
The first challenge that we mentioned was to create a user experience that allows people to feel like their iron man. In order to beat this challenge the first step that we took was using a Kinect to detect the hand of a user and to detect whether this hand moves in the X, Y or Z axis. This weekend we learned how the infrared sensor of the Kinect works and how it can detect the distance of objects. So we created a processing program that detects the hand of the user when it’s in a specific distance and it puts an ellipse in the middle of the hand and then if the user moves his hand closer than the code change the color of the ellipse simulating that the user shot at a target. The code is attached.
Credits to Daniel Shiffman from the Coding Train to teach us how to read kinect data
Our second obstacle was to create a glove that would represent one that would be identical to Iron Man’s. When we searched on Google, we found out that a replica of Iron Man’s glove was around 300 dirhams, but we realized we can still make something similar without paying that much. If we find a red glove and put metal ornaments and be careful with our design, we can achieve the similar glove and it’s also fine since we will be using kinetic glove movement. This means that the kinetic camera will only detect the hand rather than any light or design-related issues from the glove. Therefore, we can put more effort into making the glove as similar as possible.
Finally, our last challenge was to make the game more video-gameish in the sense that the game will be more interactive than just hitting the play button and playing the game. What we have come up, for more interactive play, is the function that allows users to write their names once they play the game and see who is on the top of the scoreboard. Not only that, we are also thinking of putting in an option button at the start of the game that will have controls to determine how fast the targets will appear, etc. Hopefully these updates will allow the player to have more freedom and thus have a better experience with our upcoming game.
This detailed article embodies most of what I enjoy and love about interactive media. I enjoyed forming a deeper understanding of how computers are programmed to “see”, and the article provided a history lesson of some sorts as it detailed the origins of computer vision, and all the fascinating projects which nurtured this concept into the wonderful world of interactive media that we know and continue to develop today.
The first project that showcased the use of computer vision in interactive art was Myron Krueger’s Videoplace, and despite being first conceived in the early 1970s it seems to hold a lot in common with several modern digital art installations that I enjoy and am inspired by. I kept making mental references to the works of American immersive artist, Chris Milk, who creates large interactive installations that require the audience’s interactions. I found it fascinating how I could identify elements of elementary computer vision techniques in Milk’s and other artist’s work. This made me realize that despite the conception of certain computer vision techniques decades ago, much of them pertain and apply to interactive art in modern times. I also considered how there could be endless applications for this kind of creative technology, which could generate revolutionary changes in the fields of education, architecture, media, art, etc.
Furthermore, I noticed how prior to reading this article I was starting to consider how to operate elements of motion and presence detection for my project, since I intended to use a Kinect to detect body movements and mimic them through colors and patterns.
My final project is going to help me display my art in a creative way. I want to make each drawing come to life with an animation. I floated around different ideas and versions but my finalized idea is to have a physical book, with my drawings printed on the different pages, and using an AR marker, I’m going to have it so that every page that the viewer turns, an animation plays on the drawing that brings it to life in some way, and be a way of storytelling. I’m also considering having a character that walks on the page and sort of adds to the interaction, by having the character react to each page being turned. This might be a bit too ambitious though but we’ll see. I’m quite excited about this project but also anxious about the amount of work it will require and if it turns out how I want it to.
The possibilities of creating interactive art continues to amaze me. It was interesting to see interactive media being made four decades ago. everytime I read about the incorporation of interactive media into performance, I find it very interesting and though provoking. For some reason it reminds me of the movie Step up; revolution. I think the movie was sort of my first encounter with interactive art dance performances that were mind blowing to me. I think performance is something that interactive art and computer vision can really enhance. However, as cool as I find the different examples of computer vision being used, it all sounds very complicated and challenging.
My main takeaway of this reading is that since the beginning of computer interactive art, the computer has been one of its most powerful tools. The reading goes through a bit of the history of computer vision and interactive art and it mentions that one of the first computer interactive art performance was made using computer vision. Lately, in the reading, it is also mentioned that experts are still working to improve the algorithms to make computer vision more efficient. Hence, this means that computer vision was a cornerstone technique in interactive art and that there are still things to explore in computer vision.
I also found this reading helpful for our project. I learned that Kinect uses the Infrared technique mention in the reading and that we are going to use detect motion. This reading help me to visualize all the potential techniques that I may use to create a better user experience for my final project
What I tried to do was figure out a way of setting up the Kinect in a way where it would be able to- using brightness/depth thresholding- sense the movement of people (whether they’re in front of the display or simply walking in the background) in order to play the soundtrack or pause it/replay it. I used a sample soundtrack, but I’ll start working on the actual soundtrack today and tomorrow.
One of the challenges is figuring out whether I’ll have two separate soundtracks for the two parts of the exhibit, or simply have one that plays as long as someone is standing in front of the exhibit. I’ll have to do some tests in order to figure this out.
Here is my code so far:
// Credits: Daniel Shiffman, Depth thresholding example
import org.openkinect.freenect.*;
import org.openkinect.processing.*;
import processing.sound.*;
Kinect kinect;
SoundFile soundfile;
// Depth image
PImage depthImg;
//soundtrack playing
boolean playing = false;
// pixels to be shown
int minDepth = 60;
int maxDepth = 860;
// kinect's angle
float angle;
void setup() {
size(1280, 480);
kinect = new Kinect(this);
kinect.initDepth();
angle = kinect.getTilt();
// blank image showing everything
depthImg = new PImage(kinect.width, kinect.height);
//load soundtrack
soundfile = new SoundFile(this, "soundtrack.aiff");
println("SFSampleRate= " + soundfile.sampleRate() + " Hz");
println("SFSamples= " + soundfile.frames() + " samples");
println("SFDuration= " + soundfile.duration() + " seconds");
}
void draw() {
// draw the raw image
image(kinect.getDepthImage(), 0, 0);
// threshold the depth image
int[] rawDepth = kinect.getRawDepth();
int counter =0;
for (int i=0; i < rawDepth.length; i++) {
if (rawDepth[i] >= minDepth && rawDepth[i] <= maxDepth) {
depthImg.pixels[i] = color(255);
counter++;
} else {
depthImg.pixels[i] = color(0);
}
}
if (counter > 10000){
if (!playing)
{
soundfile.play();
playing = true;
}
}
else
{
if (playing)
{
soundfile.stop();
playing = false;
}
}
// Draw the thresholded image
depthImg.updatePixels();
image(depthImg, kinect.width, 0);
fill(0);
text("TILT: " + angle, 10, 20);
text("THRESHOLD: [" + minDepth + ", " + maxDepth + "]", 10, 36);
}
//// Adjust the angle and the depth threshold min and max
//void keyPressed() {
// if (key == CODED) {
// if (keyCode == UP) {
// angle++;
// } else if (keyCode == DOWN) {
// angle--;
// }
// angle = constrain(angle, 0, 30);
// kinect.setTilt(angle);
// } else if (key == 'a') {
// minDepth = constrain(minDepth+10, 0, maxDepth);
// } else if (key == 's') {
// minDepth = constrain(minDepth-10, 0, maxDepth);
// } else if (key == 'z') {
// maxDepth = constrain(maxDepth+10, minDepth, 2047);
// } else if (key =='x') {
// maxDepth = constrain(maxDepth-10, minDepth, 2047);
// }
//}
This thorough article gave me a very good understanding of how a computer is taught to see. Yet, primarily, shifted my attention to how we take our vision and the way we see and react to the world for granted since it’s a result of countless subconscious processes happing in our brain.
I found it very thought-provoking: humans can distinguish dark and light or objects and humans in a blink of an eye, without having to forcefully learn it. The processes are automatic and we cannot influence them, alter them nor stop them. Similarly to the way our brains are wired in to recognize faces – this mechanism is so efficient that we start seeing faces even where there are none. (As I recently learned, this phenomenon is called pareidolia, see a screenshot from a google search below for quick reference).
That’s why it was very interesting to read about how to teach computers all of these things through codes. Not only people programmed computers to “see” but they also alter what, how and when they see it based on the frame differences and pixel comparisons. They set very rigid boundaries of what the computer can and cannot see, which we can’t achieve with a human brain. It’s not even an imitation of how our brains work – it’s a completely different mechanism.
It made me realize that the way we make computers see (as described in the article) can actually serve as a crucial difference between computer “minds” and human minds (just think of the “I’m not the robot” kind of tests that show you a picture that a computer cannot transfer into text- and that I failed a way too many times that I’m slowly starting to doubt my own humanness).
