Intro
Sometimes, unexpected pop-ups may not be serendipities but discoveries of limits that propel us to set off on a new adventure.
My expenditure this time didn’t turn out to be very smooth in terms of my realizing my initial vision and how that vision deviates from my expectations. On the other hand, I believe these sorts of experiences would be quite beneficial as a lesson learned to guide future planning.
As my initial vision (or product concept) changed many times in the course, it’s rather difficult to start this documentary with a definitive concept aforesaid. Therefore, I will briefly touch on the concept of each of the prototypes alongside my process description.
process
A Definitive Heart in 3D Space
In the first stage of my development, my goal was to construct a framework of basic interactions that could satisfy my needs later on – which was attempted later on but eventually did not stand for my final product. Hence, I stuck to the idea of generating a visible coordinate system in 3D space at the beginning and wrote a basic ‘arrow generating-shooting’ mechanism.
The loops in this prototype are mostly for displaying every ‘arrow’ I stored in an array and the axis plane in each frame.
let arrowArray = [];
function draw() {
...
// Display all arrows in arrowArray for 1 frame
if (arrowArray[0]) {
for (let i = 1; j = arrowArray[i] ;i += 1) {
if (j._3Dz < -50) {
arrowArray.splice(i, 1); // Remove the arrow that is behind the plane
} else {
j.arrowDisplay(); // A method in Arrow class
}
}
}
}
// Draw axis plane on a Graphics object
function drawAxis() {
// translate to 2D coordinate system
axis.translate(-axis.width / 2, -axis.height / 2);
axis.background(10);
// draw the axises
axis.stroke('white');
axis.line(0, axis.height / 2, axis.width, axis.height / 2);
axis.line(axis.width / 2, 0, axis.width / 2, axis.height);
// draw the marks on the y axis
for (let i = 0; i < width; i += axis.height / 20) {
axis.line(axis.width / 2,
0 + i,
axis.width / 2 + 5,
0 + i);
}
// draw the marks on the x axis
for (let i = 0; i < width; i += axis.width / 20) {
axis.line(0 + i,
axis.height / 2,
0 + i,
axis.height / 2 - 5);
}
}
Here, I adopted a math function that has the graphic appearance of a heart to test out the prototype. As the heart function could have multiple y values for one x value, I used the unit circle to generate the coordinates of the points on the function.
// Heart shape function
function functionHeartX(t) {
return 160 * pow(sin(t), 3);
}
function functionHeartY(t) {
return 150 * cos(t) - 55 * cos(2 * t) - 30 * cos(3 * t) - 10 * cos(4 * t);
}
Besides, I also tried some basic lighting to hue the ‘arrows’ with redish-pinky color. It was quite interesting to see how the gamma factor affects the actual light color.
lights();
ambientLight(50); // as a gamma factor
pointLight(
255, 0, 0, // color
0, 0, 0 // position
);
A Probability Heart in 2D Space
Then, I started a new prototype to specifically realize generating random positions based on the probability distribution in a 2D space (in order to later adopt into the 3D space). Multiple probability functions were tested in this stage, including:
// Gaussian probability function
function gaussianProbability(distance) {
if (distance > maxDistance) {
return 0; // No points beyond maxDistance
}
return exp(-pow(distance, 2) / (2 * pow(sigma, 2))); // Gaussian decay
}
// Quadratic probability function
function quadraticProbability(distance) {
if (distance > maxDistance) {
return 0; // No points beyond maxDistance
}
return max(0, 1 - pow(distance / maxDistance, 2)); // Quadratic decay
}
It is also noticeable that the default random() function could be used as such to mimic the probability realization:
// Add point based on probability
if (random() < probability) {
points.push(createVector(x + width / 2, -y + height / 2)); // Store the point
}
A Probability Heart in 3D Space
After that, it was much easier to adopt the probability parts into the 3D version (and to be presented as a cliche gift in a long distance relationship):
# Please access the p5js page directly from the instance to allow webcam/mic usage for the following demonstration.
On top of that, I started to experiment with the webcam as an input:
// Create a video capture from the webcam
video = createCapture(VIDEO, { flipped:true });
video.hide(); // Hide the default video element that appears under the canvas
My intentions in this stage is: 1. convert the video into grayscale values instead of RGBs (as the grayscale values can be later on mapped into probabilities); 2. downgrade the video resolution into a mosaic (as, for probability generation purposes, it would save a lot of time and be even more beneficial for the depiction of the overall shape that I planned to deliver with a collective of ‘arrows’).
To achieve the first goal, we have to operate directly on the pixel array that holds the RGB values for each pixel of each frame in the video with a time complexity of O(n) (instead of calling a filter() method for the displayed effect on the video stream when showing the video).
function videoToGray() {
// Load the pixels from the video
video.loadPixels();
// Check if the video has pixels loaded
if (video.pixels.length > 0) {
// Convert to grayscale
for (let i = 0; i < video.pixels.length; i += 4) {
let r = video.pixels[i]; // Red
let g = video.pixels[i + 1]; // Green
let b = video.pixels[i + 2]; // Blue
// Calculate grayscale value
let gray = (r + g + b) / 3;
// Set the pixel color to the grayscale value
video.pixels[i] = gray; // Red
video.pixels[i + 1] = gray; // Green
video.pixels[i + 2] = gray; // Blue
// pixels[i + 3] stays the same (Alpha)
}
// Update the video pixels
video.updatePixels();
}
}
As for the second goal, however, it was also at this stage that I did a lot of technical idle work. I started to write my method for downgrading the video resolution right away without checking the features of the video.size() method, and ended up with a function that loops through the pixel arrays with a time complexity of O(n^2) (which became very time-consuming when the original video resolution is relatively high):
function videoToMosaic(mosaicSize) {
// Load the pixels from the video
video.loadPixels();
// Check if the video has pixels loaded
if (video.pixels.length > 0) {
// Clear the mosaicPixels array
mosaicPixels = new Uint8ClampedArray(video.pixels.length);
// Loop through the canvas in blocks
for (let y = 0; y < height; y += mosaicSize) {
for (let x = 0; x < width; x += mosaicSize) {
// Calculate the average color for the block
let r = 0, g = 0, b = 0;
let count = 0;
// Loop through the pixels in the block
for (let j = 0; j < mosaicSize; j++) {
for (let i = 0; i < mosaicSize; i++) {
let pixelX = x + i;
let pixelY = y + j;
// Check if within bounds
if (pixelX < width && pixelY < height) {
let index = (pixelX + pixelY * video.width) * 4;
r += video.pixels[index]; // Red
g += video.pixels[index + 1]; // Green
b += video.pixels[index + 2]; // Blue
count++;
}
}
}
// Calculate average color
if (count > 0) {
r = r / count;
g = g / count;
b = b / count;
}
// Set the color for the entire block in the mosaicPixels array
for (let j = 0; j < mosaicSize; j++) {
for (let i = 0; i < mosaicSize; i++) {
let pixelX = x + i;
let pixelY = y + j;
// Check if within bounds
if (pixelX < width && pixelY < height) {
let index = (pixelX + pixelY * video.width) * 4;
mosaicPixels[index] = r; // Set Red
mosaicPixels[index + 1] = g; // Set Green
mosaicPixels[index + 2] = b; // Set Blue
mosaicPixels[index + 3] = 255; // Set Alpha to fully opaque
}
}
}
}
}
At the end of the day, it turned out that I could simply lower the hardware resolution with video.size() method:
video.size(50, 50); // Set the size of the video
Random Points Generated Based on Probability
Next, I wrote a prototype to see how the mosaic of probabilities could guide the random generation. The core code at this stage is to map the x-y coordinates on the canvas to the mosaic pixel array and map the grayscale value to probability (the greater the grayscale, the lower the probability, as the brighter the mosaic, the fewer the points):
function mapToPixelIndex(x, y) {
// Map the y-coordinate to the pixel array
let pixelX = Math.floor((x + windowWidth / 2) * j / windowWidth);
let pixelY = Math.floor((y + windowHeight / 2) * k / windowHeight);
// Ensure the pixel indices are within bounds
pixelX = constrain(pixelX, 0, j - 1);
pixelY = constrain(pixelY, 0, k - 1);
// Convert 2D indices to 1D index
return pixelX + pixelY * j;
}
function shouldDrawPoint(x, y) {
let index = mapToPixelIndex(x, y);
let grayscaleValue = pixelArray[index];
// Convert grayscale value to probability
let probability = 1 - (grayscaleValue / 255);
// Decide to draw the point based on probability
return random() < probability; // Return true or false based on random chance
}
‘Eventually’, I incorporated all the components together to try out my initial vision of creating a fluid, responsive, and vague but solid representation of images from the webcam with arrows generated based on probabilities flying towards an axis plane.
Unfortunately, although the product with many tuning of variables like the amount of arrows, the spread and speed of the arrows, and the lapse of arrows on the axis, etc., there could be a vague representation captured (aided by re-mapping & stretching out the probabilities with more contrasting grayscale values), the huge amount of 3D objects required to shape a figure significantly undermines the experience of running the product.
function applyHighContrast(array) {
// Stretch the grayscale values to increase contrast
let minVal = Math.min(...array);
let maxVal = Math.max(...array);
// Prevent division by zero if all values are the same
if (minVal === maxVal) {
return array.map(() => 255);
}
// Apply contrast stretching with a scaling factor
const contrastFactor = 6; // Increase this value for more contrast
return array.map(value => {
// Apply contrast stretching
let stretchedValue = ((value - minVal) * (255 / (maxVal - minVal))) * contrastFactor;
// Clip the value to ensure it stays within bounds
return constrain(Math.round(stretchedValue), 0, 255);
});
}
Besides, the 3D space did not benefit the demonstration of this idea but hindered it as the perspectives of the arrow farther away from the focus would make them occupy more visual space and disturb the probability distribution.
At the end of the day, after trying the ortho() method in the 3D version (which makes the objects appear without the affecting perspectives), I realized that reconstructing a 2D version was the right choice to better achieve my goals.
In this latest 2D version, I gave up the idea of drawing the axis plane and introduced the concept of probability distribution affected by ‘mic level’.
function setup() {
...
// Create an audio from mic
audio = new p5.AudioIn();
audio.start(); // start mic
}
function draw() {
...
// Map the audio level to contrast factor
let level = map(audio.getLevel(), 0, 1, 1.05, 30);
// Apply high contrast transformation
pixelArray = applyHighContrast(pixelArray, level);
...
}
function applyHighContrast(array, contrastFactor) {
// Stretch the grayscale values to increase contrast
let minVal = Math.min(...array);
let maxVal = Math.max(...array);
// Prevent division by zero if all values are the same
if (minVal === maxVal) {
return array.map(() => 255);
}
// Apply contrast stretching with a scaling factor
return array.map(value => {
// Apply contrast stretching
let stretchedValue = ((value - minVal) * (255 / (maxVal - minVal))) * contrastFactor;
// Clip the value to ensure it stays within bounds
return constrain(Math.round(stretchedValue), 0, 255);
});
}
reflection
TBH, the technical explorations did consume much of my time this week, and it came to me later to realize that I could have thought of more about the theme ‘loop’ before getting started – as it appears now to be a very promising topic to delve deeper into. Nevertheless, I believe our workflow of production could be like this from time to time, and it is crucial to maintain this balance thoughout.
