One fundamental difference between human and computer is that computer cannot easily distinguish between objects. They only recognize sources as pixels and algorithms have to be implemented to distinguish a specific object. What impressed me most in the reading is Myron Krueger’s Videoplay. The system he built has a diverse enough reaction bank that users can play with. Based on the movement of the participants, the computer generates visual cues accordingly to that input. Based on the participants’ actions, the computer generates visual cues accordingly. I think the ability to respond flexibly gives the system an impression of intellectual thinking, rather than just following rigid algorithms. I also noticed how Krueger interpreted physical interaction in human-computer interaction. For example, squeezing a virtual character on the screen causes it to pop and disappear (4:12), making the interaction between human and computer feel more realistic and reducing the sense of untouchability of objects on the screen.
Other techniques that I find interesting to be used in interactive design are detecting presence and simple interactions described by Jonah Warren. Because the way computer is used in daily life is mostly rigid and only tailored to specific needs, tweaking computer’s reaction based on the aforementioned input can create an interesting art media. For example, the Cheese Installation by Christian Möller is a unique way of interpreting computer data. Normally, we would not perceive the intensity of a smile in specific percentage. However, the way the machine interprets how we smile and return it into visual cue create a creative material for artwork. As such, a simple footage of smiling can be turned into a scrutiny of someone’s emotions.
Computer vision is an extremely interesting concept, and emerging technologies significantly boost its development. Although the author mentioned that machines were not capable of doing ‘generalized’ visual input analysis and were heavily dependent on the specific set of programming techniques that would allow them to recognize only certain real-world scenes, I believe that soon (if not already), thanks to the rapid growth of AI, computers will be able to receive and process the information in a ‘general’ way without the need for the special setup in the software. Still, such a technology will probably not be widely applied at the beginning. In one of the Core classes that I recently took, we were talking a lot about Machine Learning, and one of the techniques that are used in that field with the purpose of computer vision enhancement is the Convolutional Neural Network, or CNN which learns by filtering the inputs of text, images and even sounds. Other techniques that were mentioned in the reading include brightness adjustment threshold, background subtraction, and frame differencing. All of them are described to be used to help a computer recognize the visual patterns and track changes in them, but I am sure that now there certainly exist some highly advanced technologies that, although probably based on the techniques described by the author, but still work much more efficiently and maybe incorporate a number of other methods simultaneously.
The development of computer vision is obviously a large step in our technological and scientific progress. It can be used for a lot of good things aimed at improving our lives. In healthcare, for example, it can be used to help doctors diagnose and treat patients or even assist them during surgeries. It is also widely used in self-driving vehicles, which can potentially improve road traveling conditions by making them much more regulated and safer. Computer vision can also be used for leisure activities and entertainment like games – I remember playing Xbox 360 with Kinect, a device that could track your motions and allow you to play some very cool games where you actually needed to jump and do some physical activities in real life, and they would affect the actions of your character in the game. As for Interactive Art, a great example of computer vision implementation is the installation that we recently saw in class, where you can catch the letters with your arms.
However, by implementing all these advanced technologies like motion and face recognition, we should always keep in mind the potential ethical issues that can arise as the line between interaction and privacy becomes more and more blurred. Not to mention, surveillance devices can also raise concerns among people as they become more and more ubiquitous and use high-level precision that is trained by analyzing hundreds of thousands of patterns of human facial expression, emotional behavior, etc. There are many questions that put in doubt the legitimacy of the development of such technologies as human rights can be at stake, but I guess such a discussion can last for a very long time. In my opinion, we just need to keep in mind that each of us can have different perspectives and attitudes toward computer vision, and this is completely normal.
I was influenced by the optical illusions that I encountered during my childhood. Consequently, I resolved to develop something similar. The code uses the way motion, shape, and color interact with each other to make a dynamic visual experience that looks like an optical illusion.
Optical illusions use the way our brains handle patterns, color changes, and movement to make us feel things like depth, movement, or distortion that aren’t really there.
The squares’s lineup here make it look like the image is spinning and pulsing in the opposite direction when the user “pauses” the rotating effect by pressing the “Space” button because the speed of the movement changes abruptly. The slowly changing colors and the many concentric squares’ borders also mess with how we think about depth, drawing our attention inward while the colors in the background change easily, throwing us off.
Sketch
Code I am proud of
“For()” loops are very important to this sketch because they let you make grids of circular squares over and over again. The ‘for()’ loops make placing each square easier by going through different places over and over again. Without them, it would be hard to do by hand. The design of these loops is both organized and complicated, which improves the illusion itself.
// Loop for centers of every possible square patterns
for (let x = -500 + 50; x < 500; x += 100) {
for (let y = -500 + 50; y < 500; y += 100) {
// Variable stroke color based on a sine wave
changeCol = sin(t * 100);
let newCol = map(changeCol, -1, 1, 0, 255);
// Draw multiple concentric squares with decreasing sizes and random RGB values
stroke(newCol);
strokeWeight(3);
fill(r, g, b, 60); // Random RGB with fixed alpha
rect(x, y, 80, 80);
stroke(newCol / 2);
fill(r, g, b, 80); // RGB for smaller square
rect(x, y, 60, 60);
stroke(newCol / 4);
fill(r, g, b, 120); // RGB for even smaller square
rect(x, y, 40, 40);
stroke(newCol / 6);
fill(r, g, b, 140); // RGB with different alpha
rect(x, y, 20, 20);
stroke(newCol / 8);
fill(r, g, b, 160); // RGB with a different alpha
rect(x, y, 10, 10);
}
}
These loops also make the sketch scalable and easy to edit in the future.
Future Improvements
There should be a live control panel with settings for things like spinning speed, background color transition speed, and square size. This would give people more control over the illusion and let them make the experience their own.
Adding a z-axis or moving based on depth would make things even more complicated. The sense of 3D space could be better created by making the squares bigger in the center and smaller in the background to show depth.
Random colors add variety, but a color pattern that works well together can make something look better.
This week’s reading was very interesting as it dives into the technicalities of computer vision, making it seem a lot more doable, than what one might think. By breaking down computer vision into its algorithm’s basic fundamental nature, it becomes something as simple as is this pixel’s brightness greater than or less than my threshold value. Although this type of algorithmic analysis is completely different from how we as humans use vision to understand the world, it does seem like an interesting idea to conceptualize. The farthest I can reach in comparison between computer vision and human vision (as someone with no visual impairments) might be in looking at a black-and-white photo or viewing something with a stark contrast filter.
These algorithms are super interesting though because we can literally just work from light comparison in order to send different messages to the computer about what it’s seeing. In order to optimize this it is important that we give the computer clear cut distinctions from what we want it to focus on and what we don’t. I also find it interesting how beyond light, we can use things such as heat to act as the eyes of the computer.
I think computer vision’s capacity for tracking and surveillance affects its use in interactive art because opens new avenues for creativity and expression through computers. I think with the term surveillance specifically, it can also significantly reduce an individual’s right to privacy for the sake of “art.” For example, the Suicide Box project, in my opinion is completely unethical and an abuse of ‘public privacy.’ However, that then stems the issue beyond interactive art because it becomes an issue of privacy and security. Nonetheless, because computer vision has a multitude of applications I don’t believe it is at all limited to what it can be used for in interactive art, which is why we need to stay on top of its usage to ensure people’s security.
Computer Vision has consistently been a prominent subject among programmers. It is a principal catalyst in the AI sector today, emerging as a viable option in the healthcare business and enhancing security. However, this reading not only provides a historical context to Computer Vision being the cold, technical niche used by military and mainstream industry; it has now become an accesible tool for artists, paving the way for an increased integration between computers and artists. This seems especially applicable in the modern day, when the lines separating engineering and art are becoming increasingly hazy and anyone can create interactive, visually stunning experiences.
The recurring theme in the reading is how artists have used Computer Vision to to build interactive exhibits that engage spectators in real time. Myron Kruger’s “VideoPlace” is a fitting example of this, turning a cold, impersonal technology to something deeply human-centered: using full-body interactions as a dynamic input to create immersive environments. In Computer Engineering, this is closely tied to the evolving user interfaces, which today powers technologies like motion sensors and gesture-based controls in gaming (Like VR).
Regarding the difference between Computer and Human Vision, one important difference between computers and humans is that computers use algorithms to understand images, but human vision is intuitive and contextual. The computer can “see” what “we” want it to by using techniques like frame differencing, background removal, and brightness thresholding, but these approaches are frequently inflexible, periodic and task-specific in contrast to the flexible nature of human vision. Another key difference is the number of input channels in human and computer vision. Humans take multiple inputs like colors, contrast, visual acuity and so on to make a cohesive perception, while computers only take a limited input based on the task they are intended to perform.
In interactive art, this rigidity in Computer Vision can actually be useful as it helps in simplifying the interaction to specific movements or gestures, allowing for a clearer, more focused experience for the viewers. However, as seen in the case of “Sorting Daemon” by David Robesky, such automated systems can profile people, leading to intrusive monitoring and raise other ethical concerns. As Computer Vision technology develops further, it is imperative to guarantee accountability, equity, and transparency in its application.
The topic of computer vision has been an increasingly popular idea, and I believe in the next couple years there will continue to be big leaps in its development and applications. To think that less than a decade ago computer vision was solely for the military and for those in higher education has now turn into a readily available technology for artists and the general public is amazing. As such, I really enjoy how the author presented the paper for beginners in computer vision and provided advice on how to approach projects with computer vision. The projects on tracking and surveillance were actually some of the projects that stuck with me the most throughout the reading. The Suicide Box project I felt on one hand sort of telling the story of the individual whom society, especially when their death wasn’t accounted for, but on the other hand super demoralizing because the project relies on the death of these individual. As such, I feel the capacity for tracking and surveillance for computer vision is a sensitive and difficult issue to uncover fully. There is of course a huge capacity and room for growth in the field of computer vision, however the ethnicity needs to be checked and balance with the morality and freedoms of individuals.
Through the reading, the author mentions how computer visions programs will need to be chosen mindful in order to optimally tackle the problem at hand. With complicated terms such as background subtract and frame differencing, I believe the complexity of differentiating between objects, backgrounds, and lighting is the biggest different from computer vision to human vision. As humans, we process millions of tiny information at once without noticing it and as programmers, the little things like recognizing an object are magnified as humans attempt to describe in code what that object is in computer language. Working with interactive media and computer vision for the rest of the semester, I believe the techniques regarding background subtraction and brightness tracking will play the biggest role in our projects. I feel many of our projects will rely on clear differentiating of human movements and interactions with the project. Without these techniques, I fear our projects may fall apart or not work as the screen would act as a camera and not a program which can be interacted with.
In the realm of digital art, Casey Reas’s perspectives on ‘Order’, ‘Chaos’, and ‘Controlled Randomness’ are quite intriguing. Art was perceived as a conservatively rigorous field that necessitated meticulous, well-planned strategies in contemporary forms. Nevertheless, Casey proposes to broaden the definition of “Digital Art” through his views. His perspective on the use of algorithms and randomization as creative mechanisms enables the emergence of unanticipated patterns and themes that never would have been anticipated. He draws inspiration from historical examples from the 20th century and presents examples of how randomness can be incorporated into a creative process.
Casey’s “Process 18” is an appropriate example for this. Using parameters to control the extent of randomness is a subtle masterstroke as it is able to encompass the artist’s intentions and also show the creative factor. His work raises the question of whether the concept of “randomness” is genuinely random or if it is a more complex form of order that has yet to be fully comprehended. This phenomenon is also observed in other domains, such as the “Lava Lamps” of Cloudfare for secure data encryption, the “Quantum random number generator” of Google for the generation of random numbers based on quantum mechanics, and the “Ivy Bridge” of Intel for the generation of random numbers based on thermal noise.
I intend to integrate Casey’s perspectives into my work by employing parameters to produce unpredictable variations in my projects. These variations would be beneficial in assisting me in the exploration of a variety of aesthetics, patterns, themes, and interactions that are prompted by user inputs. These would ensure that my projects offer a distinctive experience to all users.
The Art of Interactive Design” by Chris Crawford is a fascinating book as it explores the distinction between a really interactive system and basic “reactive” gimmicks. Crawford characterizes the interaction as a dynamic process with three fundamental elements: hearing, thinking, and speaking. He also emphasizes on how these three elements are crucial for a meaningful interaction between the technology and the user and extends on them separately. This is why, in my opinion, every single interaction of the system with the user needs to be meticulously prepared and ensure that all three ways in which we interact with the system—listening, speaking, and thinking—are in line with the intended usage.
Regarding implementing the reading’s insights into my work, I think I might increase the degree of interaction by paying more attention on how the system “listens” and “speaks” to the user. My p5.js sketches could react to basic inputs like mouse movements or clicks right now, but they don’t necessarily provide deliberate or meaningful response. For examples, in the pixelated text sketch (Week 4: Generative text), I could have incorporated nuances into the ways user could react with my sketch. Maybe a random song plays when the user presses the “Enter” key, or an interesting home screen with brief instructions, and so on. This would mean that I would have to place myself into the user’s shoes and make design choices according to that.
After reading Don Norman’s “The Design of Everyday Things”, I was reminded to take a look around me, and find out other examples of bad user experience. One of the first examples that comes to mind, is from my short time at NYUAD’s dorms. As you can see, there are 2 blinds in the same window, one partially translucent, and one fully opaque.
NYUAD’s blinds
While the mechanism to control a particular blind is simple enough, and a common enough pattern for us to intuitively use it, what is extremely confusing is which string controls which blind, as they are both identical! Also, it seemed liked different windows had a different arrangement/side of strings controlling each blind! My solution to fix this, would be to have the opaque one keep it’s metal beads, but to switch the translucent one’s beads to like a white plastic. This way, you can both see and feel the difference between the 2 strings, allowing you to easily choose which blind to control. Additionally, they can be in any order/side, but it must be consistent for all the windows, allowing you to develop muscle memory.
Another example, though not affecting everyone, is USB. You might think I’m referring to the USB A style connectors, that somehow always take 3 tries to get in.
I didn’t know they exhibited quantum properties!
No, that has mostly been solved with USB C (ah, a godsend). Unfortunately, USB C has introduced it’s own complications, by becoming the universal standard!
Wait, a minute, that doesn’t make sense. Isn’t that a good thing?
Well mostly yes, but this also means that USB C’s capabilities have far expanded outside the original scope of a USB connector, allowing the transfer of not just power and files (each at different levels), but also HDMI, DisplayPort, Thunderbolt, Ethernet, and so much more. The result is an incredibly convenient ecosystem and user experience, where everything fits together seamlessly… as long it works. The issue isn’t USB C’s reliability, but rather that since it supports so many protocols and extensions, almost all of which are optional, it can be quite hard to know which cable or device (or even port on the same device!) supports which features (this could supporting a protocol, like DisplayPort, or a certain power and speed output). As a result, it often leads to frustration, realising only after a long time wondering why it isn’t working, that the cable I have in hand is inadequate for the task.
The solution I would recommend, would be to support everything that makes sense, and the highest speed & power level. Although, of course I would say that, I’m the user! It’s true that going this route is going to considerably increase costs for manufacturers, though as Don Norman said, poor design is sometimes due to decisions aimed at reducing cost, and this is certainly an example of that, so maybe it would be worth it for manufacturers to implement it, at least for our sanity.
Note:
Obviously, the capabilities supported should be within reason. FOr example, it makes no sense for a router which lacks any direct graphical interface, to support HDMI or DisplayPort in their USB C ports. Also, there does exist a standard that supports almost everything, and the best power & speed levels, which makes it much easier to know what works, Thunderbolt . However, it is usually pretty expensive and isn’t offered/supported on many devices. Oh also, USB’s naming needs to be fixed. They have changed it so many times, leading to a massive amount of confusion. Search it up.
Coming to second part of our prompt, I believe the main way we can apply his principles, is by integrating discoverability, affordances, & signifiers, with deliberate thought. Namely, we should make sure that we strive towards adapting our work around the person, having intuitive design, where the user doesn’t need to read an instruction manual (or ideally, read any instructions at all, possibly by using familiar visual elements and design practices people have already gotten accustomed to) to interact with our work. It should instead nudge or guide the user subtly (if required), and perhaps most importantly, provide immediate (and satisfying) feedback, and provide the space for people to make mistakes, but gracefully get back on track.
They should enjoy interacting with our work, not be afraid or overwhelmed by it that, so that it doesn’t end up like the unfortunate (but probably technically great) Italian washer-dryer combo 😉
A sad, abandoned boi, forever waiting for someone to enjoy using it🥺
This assignment is mainly based on the text patterns of early games like “Space Invaders” or “Tetris,” where the text and images were made out of square pixels. The way the blocks move and respond to user input is like the jumpy, random effects we see in old games where graphics were powered by simple hardware.
The key interactions involve entering any word which you want to display, and pressing “Generate” forms the word in the grid. The grid can be toggled on/off with a separate “Toggle Grid” button. When the mouse hovers over the text, the blocks near the cursor change color and become more diverse. When we hold down the “Enter” key, it causes the blocks representing the text to scatter randomly, simulating random jumpy movements. As soon as the key is let go, the boxes come back to their original positions.
Sketch
Code I am proud of
// Setup Function
function setup() {
// Create the canvas
createCanvas(600, 400);
cols = floor(width / gridSize);
rows = floor(height / gridSize);
// Taking the input and generating the other buttons
input = createInput();
input.position(20, height + 20);
button = createButton('Generate');
button.position(input.x + input.width + 10, height + 20);
button.mousePressed(startAnimation);
toggleButton = createButton('Toggle Grid');
toggleButton.position(button.x + button.width + 10, height + 20);
toggleButton.mousePressed(toggleGrid);
textSize(fontSize);
textAlign(CENTER, CENTER);
pg = createGraphics(width, height);
pg.pixelDensity(1);
}
// Function to show/hide the grid
function toggleGrid() {
showGrid = !showGrid;
}
// Draw function
function draw() {
// Smooth background blending
bgColor.r = lerp(bgColor.r, targetBgColor.r, 0.01);
bgColor.g = lerp(bgColor.g, targetBgColor.g, 0.01);
bgColor.b = lerp(bgColor.b, targetBgColor.b, 0.01);
background(bgColor.r, bgColor.g, bgColor.b);
if (showGrid) {
drawGrid();
}
if (animating && word.length > 0) {
fillLetterPixels();
}
for (let i = 0; i < pixelIndex; i++) {
let px = letterPixels[i].x * gridSize;
let py = letterPixels[i].y * gridSize;
// Jumpy explosion effect
if (exploding) {
// Randomize position for the explosion effect
px += random(-10, 10);
py += random(-10, 10);
}
else {
// Smooth return to original positions
px = lerp(px, originalPositions[i].x * gridSize, 0.1);
py = lerp(py, originalPositions[i].y * gridSize, 0.1);
}
let d = dist(mouseX, mouseY, px + gridSize / 2, py + gridSize / 2);
if (d < gridSize * 4) {
let blockColor = blockColors[i];
fill(blockColor.r, blockColor.g, blockColor.b, map(d, 0, gridSize * 4, 255, 50));
}
else {
fill(0);
}
noStroke();
rect(px, py, gridSize, gridSize);
if (random(1) < sparkleChance) {
fill(random(255), random(255), random(255));
rect(px, py, gridSize, gridSize);
}
}
}
Setup() function sets up the canvas, makes the Graphics buffer and initializes the canvas color to white.
Draw() function is the function which is the main loop that runs continuously. It controls the background color blending, sparkle effect, jumpy explosion effect, stores which keys are pressed, checks grid is toggled on/off, and so on.
// Startup animation after the generate button is clicked
function startAnimation() {
word = input.value();
pixelIndex = 0;
animating = true;
getLetterPixels();
blockColors = letterPixels.map(() => {
return { r: random(255), g: random(255), b: random(255) };
});
originalPositions = letterPixels.map((p) => ({ ...p })); // Store original positions
targetBgColor = {
r: random(255),
g: random(255),
b: random(255)
};
}
– This function is responsible for the short simulation of making the letters of the word when “Generate button” is pressed.
function getLetterPixels() {
letterPixels = [];
// Use the off-screen buffer (pg) to render the text
pg.background(255);
pg.textSize(fontSize);
pg.textAlign(CENTER, CENTER);
pg.fill(0);
pg.text(word, pg.width / 2, pg.height / 2);
pg.loadPixels();
for (let y = 0; y < pg.height; y += gridSize) {
for (let x = 0; x < pg.width; x += gridSize) {
let index = (x + y * pg.width) * 4;
let r = pg.pixels[index];
if (r < 128) {
let gridX = floor(x / gridSize);
let gridY = floor(y / gridSize);
letterPixels.push({ x: gridX, y: gridY });
}
}
}
}
This function uses the off-screen buffer (pg) to render the input text and identifies the grid positions where the pixels are black (indicating a letter).
function fillLetterPixels() {
if (pixelIndex < letterPixels.length) {
let px = letterPixels[pixelIndex].x * gridSize;
let py = letterPixels[pixelIndex].y * gridSize;
fill(0);
noStroke();
rect(px, py, gridSize, gridSize);
pixelIndex++;
}
else {
animating = false;
}
}
This function simulates the pixels that form the letters. It gradually reveals the blocks one by one.
// Run the function if ENTER key is pressed
function keyPressed() {
if (keyCode === ENTER) {
exploding = true; // Start the explosion effect
}
}
// Run the function if ENTER key is released
function keyReleased() {
if (keyCode === ENTER) {
exploding = false; // Stop the explosion and return to original positions
}
}
When the Enter key is pressed or released, it starts or stops the jumpy explosion effect. When the Enter key is pressed, the exploding flag is set to true, causing the blocks to scatter randomly. When the Enter key is released, the exploding flag is set to false, and the blocks return to their original positions.
Challenges/ Future Improvements
Some of the major challenges I had faced were:
Creating the random “jumpy” explosion effect and making sure that the boxes making up the letters stay in place after key is released.
The “sparkle” effect was another challenge as it required creating intermittent flashes of random color across specific blocks.
The color changing due to the “hovering mouse” was another one. However, by measuring the distance (dist()) between the mouse and the block’s center, the program smoothly blends colors based on proximity to the cursor, giving the effect of interaction with the text.
However, the code can be further improved upon by introducing some collission, introducing different layers to create a 3D pixelated text feel, and so on.
Note:
. However, it is usually pretty expensive and isn’t offered/supported on many devices. Oh also, USB’s naming needs to be fixed. They have changed it so many times, leading to a massive amount of confusion. Search it up.