For the initial stage of the final project – WaterBox, the main focus for this week was getting the Kinect working for the specific purpose designated towards this project. And, for this project, I have decided to use Kinect v2. After some time trying to figure out how Kinect v2 works, I have encountered several issues that took me few hours to solve.
One of the main problem I encountered was that Kinect v2’s video image had different dimensions compared to the other lenses (the IR, depth, and registered). Since I was looking to use the depth functionality as well as the RGB value of the pixel on different points within the image, my take on using the video image and the depth image was not a viable option – since the index of the depth image did not point towards the same position in the video image. Thus, I had decided to use the RegisteredImage, which combines the RGB and depth view, to analyze the RGB value on a certain pixel.
For both versions, I have set the minimum threshold as 480 and maximum threshold as 830 – but these values are subject to change as I test it on water. Following videos are some of the try-outs and testing with Kinect v2:
The first version is a test to see whether the Kinect accurately captures a certain color value (in this case, when B-value is higher than 200). For the test medium, I have used an empty blue screen on my mobile phone. As you can see in the video above, the Kinect successfully captures the designated values.
import org.openkinect.processing.*;
Kinect2 kinect2;
float minThresh = 360;
float maxThresh = 830;
PImage img, video;
int[][] screen;
void setup() {
size(512, 424);
kinect2 = new Kinect2(this);
kinect2.initRegistered();
kinect2.initDepth();
kinect2.initDevice();
screen = new int[kinect2.depthWidth][kinect2.depthHeight];
for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
screen[x][y] = 0;
}
}
img = createImage(kinect2.depthWidth, kinect2.depthHeight, RGB);
}
void draw() {
background(255);
img.loadPixels();
video = kinect2.getRegisteredImage();
int[] depth = kinect2.getRawDepth();
for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
int index = x + y * kinect2.depthWidth;
int d = depth[index];
if (d > minThresh && d < maxThresh && blue(video.pixels[index]) > 230) {
img.pixels[index] = color(0, 0, blue(video.pixels[index]));
screen[x][y] = 1;
} else {
img.pixels[index] = color(255);
screen[x][y] = 0;
}
}
}
img.updatePixels();
noStroke();
for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
int index = x + y * kinect2.depthWidth;
if (screen[x][y] == 1) {
int mapVal = (int)map(depth[index], minThresh, maxThresh, 255, 0);
fill(mapVal, 0, 0);
ellipse(x, y, 2, 2);
}
}
}
}
Since I have to track where these recognized points are within the canvas, I have created a separate 2-dimensional array that tracks at which x and y coordinates the Kinect recognizes. So, the above video is a test with drawing ellipses where the pixels with blue value higher than 200, and each ellipse has a color gradient based on the depth in between the thresholds. The depth, in the later part of the project, will be used as the value that changes the pitch or speed of the sound created.
import org.openkinect.processing.*;
Kinect2 kinect2;
float minThresh = 360;
float maxThresh = 830;
PImage img, video;
int[][] screen;
void setup() {
size(512, 424);
kinect2 = new Kinect2(this);
kinect2.initRegistered();
kinect2.initDepth();
kinect2.initDevice();
screen = new int[kinect2.depthWidth][kinect2.depthHeight];
for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
screen[x][y] = 0;
}
}
img = createImage(kinect2.depthWidth, kinect2.depthHeight, RGB);
}
void draw() {
background(255);
img.loadPixels();
video = kinect2.getRegisteredImage();
int[] depth = kinect2.getRawDepth();
int countTL = 0;
int countTR = 0;
int countBL = 0;
int countBR = 0;
for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
int index = x + y * kinect2.depthWidth;
int d = depth[index];
if (d > minThresh && d < maxThresh && blue(video.pixels[index]) > 230) {
img.pixels[index] = color(0, 0, blue(video.pixels[index]));
screen[x][y] = 1;
if (x < kinect2.depthWidth/2 && y <kinect2.depthHeight/2) {
countTL++;
fill(0, 255, 0);
rect(0, 0, kinect2.depthWidth/2, kinect2.depthHeight/2);
text("hello", 0, 0);
}
if (x > kinect2.depthWidth/2 && y < kinect2.depthHeight/2) {
countTR++;
fill(0, 0, 255);
rect(kinect2.depthWidth/2, 0, kinect2.depthWidth, kinect2.depthHeight/2);
}
if (x < kinect2.depthWidth/2 && y > kinect2.depthHeight/2) {
countBL++;
fill(255, 0, 0);
rect(0, kinect2.depthHeight/2, kinect2.depthWidth/2, kinect2.depthHeight/2);
}
if (x > kinect2.depthWidth/2 && y > kinect2.depthHeight/2) {
countBR++;
fill(100, 100, 100);
rect(kinect2.depthWidth/2, kinect2.depthHeight/2, kinect2.depthWidth, kinect2.depthHeight);
}
} else {
img.pixels[index] = color(255);
screen[x][y] = 0;
}
}
}
img.updatePixels();
noStroke();
for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
int index = x + y * kinect2.depthWidth;
if (screen[x][y] == 1) {
int mapVal = (int)map(depth[index], minThresh, maxThresh, 255, 0);
fill(mapVal, 0, 0);
ellipse(x, y, 2, 2);
}
}
}
}
For the third version, I wanted to check whether I can track how many pixels are at a certain part within the screen, specifically for this case the sections include Top-Left, Top-Right, Bottom-Left, and Bottom-Right. Here, I am tracking not only where the recognized pixels are located in terms of the screen, but also storing the count of how many recognized pixels are at the position of a certain area.
The next step of the project would be integrating sound and testing on a water surface – which I will need to start building the container.