Category: Spring 2024 – Aya (Section 002)

This article provided a fascinating exploration of the evolution of computer vision technology and its integration into the realm of artistic expression. Among the showcased artworks, “Standards and Double Standards” by Rafael Lozano-Hemmer particularly intrigued me due to its metaphorical approach to incorporating computer vision. What captivates me about this piece is its ability to evoke complex narratives and reflections on societal dynamics through a seemingly simple and tangible interaction. The choice of belts and their rotation adds a layer of symbolism, raising questions about control, authority, and the implications of automated surveillance. 

This artwork also reminded me of an immersive installation I experienced on Lulu Island in January, “Pulse Island” by the same artist. It featured stations with PPG pulse sensors that detected the participant’s heart rate, surrounded by an array of over 4,000 Edison lightbulbs. Consequently, the lightbulbs around that area glimmer to the rhythm of the heartbeat as the speakers echo the heartbeat. As people add a new recording of their heartbeat, the oldest recording in the group is replaced, creating a Memento Mori. By capturing and recording this physiological data, it felt like the artwork was engaging in a subtle form of surveillance of the participants’ vital signs.

It is also important to look at the ethical considerations surrounding the use of computer vision. I was recently invited to an event with industry experts where we discussed ethical considerations of AI and it prompted me to reflect on the following. One primary ethical concern highlighted is the potential misuse or unintended consequences of technology originally designed for creative and benign purposes. The shift from artistic experimentation to potential mass surveillance raises questions about the responsibility of creators and the broader ethical framework within which these technologies operate. Another controversial issue in computer vision is bias and fairness. Surveillance Algorithms can exhibit biases based on the data they are trained on. This bias can lead to discriminatory outcomes, influencing areas such as hiring, law enforcement, and financial services. Moreover, there are several privacy concerns associated with surveillance technologies. The integration of computer vision in art installations blurs the boundaries between observation and intrusion. For example, I wasn’t aware that our heart rates were being stored in a database for a while before I participated in the installation.

The core idea of this project is to develop an interactive sound visualizer that not only generates music based on keyboard inputs but also visualizes this music with unique, noise-driven shapes on the screen. Each key press corresponds to a different note, and the shapes evolve in real-time to reflect the pitch, volume, and timbre of the sounds being produced. The project aims to blur the lines between music creation and visual art, offering users a multi-sensory experience of composing and visualizing music in real time.

Users interact with the application primarily through the keyboard, where each key triggers a distinct musical note or sound. The visual component consists of evolving shapes whose characteristics (e.g., size, color, noise level) change in response to the music. The proposed enhancement involves integrating hand detection technology, potentially through a webcam and machine learning models, to allow users to manipulate sound characteristics (like volume and pitch) and visual aspects (like shape complexity) with hand movements and gestures.

The main component is the visual class:

class ShapeVisualizer {
  constructor(x, y) {
    this.x = x;
    this.y = y;
  }

  draw(size, alpha, weight) {
    strokeWeight(weight);
    push();
    translate(this.x, this.y);
    beginShape();
    for (let a = 0; a < TWO_PI; a += 0.02) {
      let xoff = map(cos(a), -1, 1, 0, 5) + angle;
      let yoff = map(sin(a), -1, 1, 0, 5) + angle;
      let r = map(noise(xoff, yoff), 0, 1, size * 0.5, size);
      let x = r * cos(a);
      let y = r * sin(a);
      let hue = (angle * 50) % 360;
      stroke(hue, 100, 100, alpha);
      vertex(x, y);
    }
    endShape(CLOSE);
    pop();
  }
}

 

Hopefully, for future I want to make this more of an interactive synthesizer, hopefully, as I said, using hand motion

I found the reading really interesting, especially because it made me think and do research about real world applications, as well as examples in which we can use computer vision technology. Moreover, the democratization of computer vision technology has indeed paved the way for innovative applications across different sectors, not only arts. For instance, in the field of healthcare, computer vision algorithms are being used to assist radiologists in analyzing medical images such as X-rays and MRIs. By automating the process of identifying anomalies or patterns indicative of diseases, these algorithms not only enhance diagnostic accuracy but also expedite patient care, especially in regions with limited access to healthcare professionals.

Furthermore, the integration of computer vision techniques into home automation systems presents another compelling example of its democratization. Home security systems equipped with computer vision capabilities can analyze live video feeds to detect suspicious activities or unauthorized entries, alerting homeowners and authorities in real-time. Moreover, these systems can be programmed to differentiate between pets, family members, and intruders, thereby minimizing false alarms and optimizing security measures. By harnessing computer vision in this context, even individuals without extensive technical expertise can create sophisticated home security solutions, underscoring the democratizing impact of accessible tools and resources in transforming everyday environments.

Concept:

For the midterm project, I plan to create the classic Angry Birds game. In addition, I got inspiration from the movie Rio, so I plan to join both universes and incorporate elements from the movie such as characters, settings, and storyline. In this game, players will use a slingshot mechanism to launch birds towards targets (e.g., enemy characters or objects) to progress through levels.

Code design:

When it comes to the code design, I will define functions for launching birds, detecting collisions, scoring points, and advancing through levels. I will also use classes for birds, targets, obstacles, and any other interactive elements I plan to implement in the game. Then, the user input mechanisms will be added for controlling the slingshot, aiming, and launching birds against the enemies.

Example of a class structure from my code:

class Bird {
  constructor(x, y) {
    this.position = createVector(x, y);
    this.velocity = createVector();
    this.acceleration = createVector();
    this.r = 20;
    this.released = false;
  }

  display() {
    fill(255, 255, 0);
    ellipse(this.position.x, this.position.y, this.r * 2);
  }
  
  update() {
    if (this.released) {
      this.velocity.add(this.acceleration);
      this.position.add(this.velocity);
      this.acceleration.mult(0); // Reset acceleration
      this.velocity.mult(0.99); // Air resistance
    }
  }

Challenges:

One of the most challenging aspects of my project would be implementing realistic physics for bird movement and collision detection. To minimize this risk, I will start by writing code to simulate basic projectile motion and collision detection. I also plan to use existing libraries or frameworks for physics simulations, which can help for the structure of the game.

Conclusion:

In conclusion, there are several additional elements that I plan to incorporate to improve the overall gameplay experience. Beyond the existing mechanics of slingshot aiming and bird launching, future iterations of the game will feature dynamic sound effects to accompany actions such as bird launches and target destruction. Furthermore, I aim to enhance the visual elements of the game by incorporating more detailed character sprites, vibrant backgrounds, and visually appealing animations. The current sketch provides a foundational framework for the game, and that’s how it looks like so far:

 

Concept 

I am creating an interactive game that revolves around collecting rotten apples in a trash can. What inspired me to do this is that most games always make you collect the “good” items and leave the “bad” items out of the box. So, I flipped this norm that we always see in kids games and I am now collecting rotten apples (bad item) and leaving the good apples (normal items). There is no specific game that inspired the apple catching game but I just thought about using an apple since apples fall from trees so it makes sense that they are falling from the sky and can be collected.

Design

There are normal apples falling from the sky and rotten apples. The purpose of this game is to try to collect as much rotten apples into the trash can as possible. The trash can is in the bottom of the canvas and can move horizontally to catch the falling rotten apples. There is a score that is being incremented by 10 every time a rotten apple is collected in the trash can. For the design of the trash can and the apples, I uploaded pictures of a trash can and pictures of apples to make the aesthetics of this game better. I still did not design the “start” page yet but I will also be uploading pictures to make it look aesthetically pleasing. In terms of sound, I am aiming to add background music just like kids games and also a sound for every time an apple is caught in the trash can. Since the game is still in progress I did not add it to this post.

Improvements

I think what will make this project interesting is adding timer to the game so that the player has limited time to collect the apples. Also I think being able to ask for a players name in the beginning and then add it to an array that holds their name and score can make this a more competitive game so when the game ends the player can see what position he has compared to his friends. I also need to add a function that allows the player to start over the whole game and go back to the home page or only start over the game without having to go to the home page. I also thought power ups can make this game more interesting but I do not know if I can achieve that yet. Lastly, I need to make the screen be as big as possible (take as much of the screen as  possible).

Limitations 

The images that are uploading have a white background, so, if the background is any other color than white, the frame of the image will show and it is not aesthetically pleasing. Also the apple randomizer might make the game unfair because you never know how many rotten apples you will get in the span of your turn. Another player might get more rotten apples in the screen, thus, a better chance to get a higher score. Another limitation is that the apple images might be confusing because they are small and some players might get confused between the rotten and good apple. I obviously will be working to reduce these limitations and make this game as good as possible.

For my Midterm Project, I wanted to create a simple yet fun and interactive game that you could play on the go. I decided to create “Catch It!”, a catch game where elements fall from the top and the player needs to move the basket to avoid bombs and collect enough elements before the time ends to move onto the next level.

(Reference)

My game would consist of elements falling from the top of the screen randomly at different positions. There would be a basket present at the bottom which is controlled by left and right keys. Every element that is caught by the basket gives points to the user. However, if a bomb is falling, the user should move to another position and try to avoid it. There will be a time limit for every level. If the user gains enough points within the time limit, they move on to the next level. The difficulty increases with each level. The speed of falling elements will increase, the frequency of bombs will increase, and the time given will decrease. Furthermore, once in a while, a special power-up element will fall, which if you are able to catch will make you immune to 1 bomb. For each level, the player gets to choose a theme.

I implemented a prototype just to test the falling and collision detection, the most crucial parts of the game. The screen is divided into rows and columns forming tiles. The element would fall tile by tile down the screen from a random column and will either be caught by the basket or miss the basket and go out of the screen. Once the element goes off-screen, it respawns at the top.

Here is what my sketch looks like so far:

It is quite thought-provoking to examine how technology is dual in nature in both art and surveillance. The books explore the philosophical and ethical implications of technology’s development, emphasizing how inventions that were first created for beneficial and creative uses might later be transformed into instruments for widespread control and surveillance. This contradiction is best illustrated by the way that facial recognition has evolved from a scientific marvel to a possible tool for privacy invasion. This should cause readers to consider the wider effects of technology progress. Furthermore, a rich field for discussion on the nature of observation and the hazy boundaries between watcher and watched is provided by the irony in works of art that criticize surveillance by using surveillance techniques themselves. This dichotomy challenges readers to consider the social values that influence technology advancement as well as the motivations behind it.

These thoughts resonate with my own experiences and opinions on technology on a personal level. Just as I was astounded when I was a child and could talk to distant relatives via video chats, I have always been in awe of the advances made possible by digital innovation. However, this nostalgia is now flavored with a cautionary tale about the invisible eyes that could be hiding behind any gadget. I’ve been forced by the reading to reevaluate how I use technology and to acknowledge the fine line that separates convenience from invasion. It speaks to my growing knowledge of privacy issues and the covert, frequently undetectable methods that I could be monitored. The aforementioned artworks function as a reminder of the ubiquitous nature of monitoring and the necessity of maintaining awareness of the trade-offs between the advantages of technology and individual privacy. My grasp of the societal effects of technology has grown as a result of this thinking, which has also motivated me to handle my digital imprint more carefully.

Concept and Inspiration

For my midterm, I want to create some sort of audio-reactive (?) artwork. I am unsure about the audio-reactive part because I don’t think I want it to actually live react to audio; rather, I want the artwork to depend on the preloaded sounds.

Essentially, I created a playlist of 12 songs. I want to upload these songs and their matching cover images as well as the title and the artist. The idea is that the user can switch between songs with the left and right arrows keys, and the visuals in the background will change depending on which song is playing.

Here are the songs I picked:

1. Heart-Shaped Box – Nirvana
2. Cool Colorado – La Femme
3. Weak For Your Love – Thee Sacred Souls
4. Spooky – Dusty Springfield
5. Karma Police – Radiohead
6. Buddy’s Rendezvous – Lana Del Rey
7. Althea – Grateful Dead
8. Naive – The Kooks
9. Drink Before The War – Sinead O’Connor
10. Right Down The Line – Sam Evian
11. She – The Blaze
12. Belong In The Sun – ¿Téo?

My inspiration comes from the feelings I get when I listen to music. Sometimes, I just lay in bed and close my eyes and I start visualizing the energy of the music. So I simply wanted to create an artwork that matches visuals to music.

Back to my first uncertainty, I think I want the visuals to depend on the characteristics of the song itself. In other words, I would like to extract the amplitudes and frequencies of each songs and load them in the functions of the visuals in order for them to control what the visuals will look like.

For the visuals, I want to have some sort of melting effect. That is what will be controlled by the characteristics of the music. I am also thinking, for the color, whether to also have it controlled by one of the music’s characteristics or whether it should be based on the colors of the cover image (by extracting the pixels).

Risks and Problems

1. I tried loading the songs as mp3 files and most of them didn’t work. The only two that worked I had compressed through a website as they were over 5mb. The other ones wouldn’t load for some reason, so I need to figure that out.
2. I tried playing with the visuals a bit, but I am not satisfied. As I am still exploring how to make them, I think it will take some time for me to achieve something close to what I want to get. Basically, I tried to create lines that have a melting effect by using Perlin noise, in order to make it dynamic
3. I think the hardest part will be having the amplitude/frequency of the music control the visuals. I will be working on that next!

Sketch Progress

Anyway, here is what my sketch looks like so far (I hate!!!! the way the background looks like currently):

You can use the left arrow and right arrow keys to switch between songs, though no music will play :/

What’s Left to Add and Change

• • At least one sound (12, technically)
  • A screen giving instructions and wait for user input (button / key / mouse / etc.) before starting
  • A way to start a new session (without restarting the sketch), though I’m not sure what exactly this would entail
  • The shape, perhaps, and to have it controlled by the music
  • The colors, and to have them controlled either by the music or by the cover image – then the text will be more readable hehe

The exploration of computer vision in interactive artworks is truly fascinating, especially considering its roots in Myron Krueger’s Videoplace from the late 1960s. The idea that the entire human body should be involved in interactions with computers was revolutionary at the time. Videoplace’s ability to digitize a participant’s silhouette and synthesize graphics based on their posture and gestures was ahead of its time. I believe that the notion of involving human body with computer interaction through computer vision to be an essential yet dangerous idea. While it does produce valuable possibilities like the Videoplace or Sorting Daemon, but there could be dangerous implications for our society. For instance, computer vision algorithms may exhibit bias, leading to discriminatory outcomes, especially if the training data is not representative of diverse populations. Biases in such algorithms can result in unfair and discriminatory decisions, affecting individuals in areas like hiring, law enforcement, and financial services. While this can be considered as a interactivity through computer vision, the consequences of this in this contemporary society could affect the lives of many in serious ways.

While reading the computer vision techniques, the notion of semantic understanding piqued my interests. Upon further research I found that unlike text-based data, which inherently carries semantic and symbolic information, digital video, in its raw form, consists of streams of rectangular pixel buffers with no intrinsic meaning therefore making the case for this problem. This lack of inherent semantics hinders a computer’s ability to interpret or extract meaningful information from the video content without additional programming. Bridging the semantic gap seems to be important for developing computer vision algorithms that can discern and interpret the content of video streams, enabling applications to answer even elementary questions about the presence of objects or individuals and the contextual details within a given scene.

From a young age, I always loved playing arcade games with my brother. We would get around 20 gold coins from the cashier and spend all of it on different games. Which is why for a midterm project, I have decided to make a small arcade game called Knighthood using sprites. I am using p5play library and incorporating it to p5js using cdn. For now, I have one character called knight who stay idle, walk, jump, and attack. Although I have not implemented it yet, I am planning to make the game seem as vintage as possible. Design wise, I am also planning to incorporate a background music and movement sounds as well. For instance, there will be sound when the knight attacks or when enemies die.

As for game logic, I am planning to make 3 types of difficulties so the number of enemies change alongside their life bar. As the user chooses the difficulty, the number of enemies with different life bars will be pre loaded and will spawn in random location is set interval and attack the character. the Character will die after 3 hits.

I believe that the challenging aspect of this project will be implementing the way in which the knight hits the enemies. The condition is that the enemy should be near the knight with the margin of 10 px and the key ‘K’ should be pressed. To mitigate this problem, I am planning making boolean variable is Attacking to check if the K key is pressed and if it is pressed, checks if the enemy is nearby and if they are, they will change animation to dead state and then vanish after few seconds. Another potential problem might be the interactive problem of indicating that the knight was hit. User has to know besides the health count that the knight has been hit. For this, I am thinking of implementing knock back effect for the knight when he is hit by an enemy.

function keyPressed() {
  if (keyCode === 66) {
    // 'B' key pressed, hide instructions
    showInstructions = false;
  }
  if (keyCode === 87) {
    mySprite.setSpeed(3, 270);
    mySprite.changeAnimation("walk");
    setTimeout(() => {
      mySprite.changeAnimation("stand");
    }, 500);
  } else if (keyCode === 83) {
    mySprite.setSpeed(3, 90);
    mySprite.changeAnimation("walk");
    setTimeout(() => {
      mySprite.changeAnimation("stand");
    }, 500);
  } else if (keyCode === 65) {
    mySprite.setSpeed(3, 180);
    mySprite.changeAnimation("walk");
    setTimeout(() => {
      mySprite.changeAnimation("stand");
    }, 500);
  } else if (keyCode === 68) {
    mySprite.setSpeed(3, 0);
    mySprite.changeAnimation("walk");
    setTimeout(() => {
      mySprite.changeAnimation("stand");
    }, 500);
  } else if (keyCode === 13) {
    mySprite.position.x = width / 2;
    mySprite.position.y = height -100;
    mySprite.setVelocity(0, 0);
    mySprite.changeAnimation("stand");
  } else if (keyCode === 32 && !isJump) {
    mySprite.changeAnimation("jumping");
    isJump = true;
    mySprite.velocity.y = -8;
    // Set a timeout to reset the jumping flag after a short duration
    setTimeout(() => {
      mySprite.velocity.y = +5;
      mySprite.changeAnimation("stand");
      isJump = false;
    }, 500); // Adjust the duration as needed
  } else if (keyCode === 75 && !isAttack) {
      mySprite.changeAnimation("attack");
      isAttack = true;
      setTimeout(() => {
        mySprite.changeAnimation("stand");
        isAttack = false;
      }, 500) ;
  }
}