Category: Spring 2024 – Aya (Section 002)

The exploration of computer vision in interactive artworks is truly fascinating, especially considering its roots in Myron Krueger’s Videoplace from the late 1960s. The idea that the entire human body should be involved in interactions with computers was revolutionary at the time. Videoplace’s ability to digitize a participant’s silhouette and synthesize graphics based on their posture and gestures was ahead of its time. I believe that the notion of involving human body with computer interaction through computer vision to be an essential yet dangerous idea. While it does produce valuable possibilities like the Videoplace or Sorting Daemon, but there could be dangerous implications for our society. For instance, computer vision algorithms may exhibit bias, leading to discriminatory outcomes, especially if the training data is not representative of diverse populations. Biases in such algorithms can result in unfair and discriminatory decisions, affecting individuals in areas like hiring, law enforcement, and financial services. While this can be considered as a interactivity through computer vision, the consequences of this in this contemporary society could affect the lives of many in serious ways.

While reading the computer vision techniques, the notion of semantic understanding piqued my interests. Upon further research I found that unlike text-based data, which inherently carries semantic and symbolic information, digital video, in its raw form, consists of streams of rectangular pixel buffers with no intrinsic meaning therefore making the case for this problem. This lack of inherent semantics hinders a computer’s ability to interpret or extract meaningful information from the video content without additional programming. Bridging the semantic gap seems to be important for developing computer vision algorithms that can discern and interpret the content of video streams, enabling applications to answer even elementary questions about the presence of objects or individuals and the contextual details within a given scene.

From a young age, I always loved playing arcade games with my brother. We would get around 20 gold coins from the cashier and spend all of it on different games. Which is why for a midterm project, I have decided to make a small arcade game called Knighthood using sprites. I am using p5play library and incorporating it to p5js using cdn. For now, I have one character called knight who stay idle, walk, jump, and attack. Although I have not implemented it yet, I am planning to make the game seem as vintage as possible. Design wise, I am also planning to incorporate a background music and movement sounds as well. For instance, there will be sound when the knight attacks or when enemies die.

As for game logic, I am planning to make 3 types of difficulties so the number of enemies change alongside their life bar. As the user chooses the difficulty, the number of enemies with different life bars will be pre loaded and will spawn in random location is set interval and attack the character. the Character will die after 3 hits.

I believe that the challenging aspect of this project will be implementing the way in which the knight hits the enemies. The condition is that the enemy should be near the knight with the margin of 10 px and the key ‘K’ should be pressed. To mitigate this problem, I am planning making boolean variable is Attacking to check if the K key is pressed and if it is pressed, checks if the enemy is nearby and if they are, they will change animation to dead state and then vanish after few seconds. Another potential problem might be the interactive problem of indicating that the knight was hit. User has to know besides the health count that the knight has been hit. For this, I am thinking of implementing knock back effect for the knight when he is hit by an enemy.

function keyPressed() {
  if (keyCode === 66) {
    // 'B' key pressed, hide instructions
    showInstructions = false;
  }
  if (keyCode === 87) {
    mySprite.setSpeed(3, 270);
    mySprite.changeAnimation("walk");
    setTimeout(() => {
      mySprite.changeAnimation("stand");
    }, 500);
  } else if (keyCode === 83) {
    mySprite.setSpeed(3, 90);
    mySprite.changeAnimation("walk");
    setTimeout(() => {
      mySprite.changeAnimation("stand");
    }, 500);
  } else if (keyCode === 65) {
    mySprite.setSpeed(3, 180);
    mySprite.changeAnimation("walk");
    setTimeout(() => {
      mySprite.changeAnimation("stand");
    }, 500);
  } else if (keyCode === 68) {
    mySprite.setSpeed(3, 0);
    mySprite.changeAnimation("walk");
    setTimeout(() => {
      mySprite.changeAnimation("stand");
    }, 500);
  } else if (keyCode === 13) {
    mySprite.position.x = width / 2;
    mySprite.position.y = height -100;
    mySprite.setVelocity(0, 0);
    mySprite.changeAnimation("stand");
  } else if (keyCode === 32 && !isJump) {
    mySprite.changeAnimation("jumping");
    isJump = true;
    mySprite.velocity.y = -8;
    // Set a timeout to reset the jumping flag after a short duration
    setTimeout(() => {
      mySprite.velocity.y = +5;
      mySprite.changeAnimation("stand");
      isJump = false;
    }, 500); // Adjust the duration as needed
  } else if (keyCode === 75 && !isAttack) {
      mySprite.changeAnimation("attack");
      isAttack = true;
      setTimeout(() => {
        mySprite.changeAnimation("stand");
        isAttack = false;
      }, 500) ;
  }
}

 

This week’s reading was interesting from the technical and social sides of things. The concept of creating something for experimentation, and even art, to then transform it into a weapon of mass surveillance (and even war) is something that can provoke many thoughts. Indeed, I know that my iPhone applies many processes internally, which allowed face recognition to happen in the first place, and probably surveillance of me without my consent (just probably).

I remember when I was a child. I was impressed when I first saw another human face through Skype in 2013. Yes, the machine was just displaying a set of pixels that were observable through my screen, but the fact that another human, who was residing in another country, was actually seen was truly impressive! Then the smartphone arrived, and more complex algorithms were starting to get implemented, which allowed for further experimentation and user engagement.

Therefore, I would argue that the reading really did not change that much my views on video image processing, aside from the thought-provoking fact that a war medium originated from an art experiment. And I have not used many of the algorithms that allow modifying the perception of you through the screens of other people.

In conclusion, I would not argue that the author is biased, as he presents factual information and techniques for students or professionals to implement in their careers. Nevertheless, most information could be considered outdated as it was presented around 2006–2007. In today’s world, we have Artificial Intelligences that are capable of understanding humans and even replicating them through simulations, which can then be shared as a small clip through social media.

I really enjoyed reading this text, particularly discovering the artworks on surveillance. I find it quite ironic (but in a good way) that a lot of artworks that want to talk about surveillance, employ some sort of surveillance themselves. Of course, here, surveillance has to be looked at in the wider sense. Usually, when we think surveillance we imagine CCTV or hidden cameras. However, as the text mentions, human tracking is also included under surveillance. So, even artworks that simply track your body movements, as basic as the Cheese installation by Möller, can be said to be «surveilling» you in a way. There are also the other forms of human tracking, like Lozano-Hemmer’s Standards and Double Standards, which involve more intense forms of surveillance and tracking. To have items following you takes the «scrutiny» to a different level. Reading about this artwork reminded me of the Manar Abu Dhabi exhibition which took place last fall. It included 35 artworks, many of which were installations. The one that Lozano-Hemmer’s work reminded me of is the Abu Dhabi Dots by Carsten Höller, which was exhibited on the shore of the Corniche beach. Essentially, there are projected spotlights on the sand, and when you stand on them and move, they move with you.

After reading about it now, I found out that it is actually a game. Here is the description:

Concept:

For my midterm project, I am inspired by my childhood game: a classic Tank Game in the small brick game console.

The game is basically divided into different levels. As you come to the next level, the speed come from the bullet will be faster and more component will spawn in the game. Also, the player only has a limited number of lives. When those lives are used up, it is game over. The user can move around to dodge the bullet. However, because the screen is relatively small to contain both player and enemies, it becomes more and more difficult to win.

Design:

I aim for a similar design as its original. I want player tank will have a brick-like display which is distinguished from the enemies. However, instead of doing a monochrome game similar to the original, I want to add colors and random movements to the enemies. It will be either the enemies will move close and close to the player’s tank or the tank will move and shoot bullets randomly.

Object Oriented Class:

• Tank – Need to have 2 designs, one for player and one for the enemies.
• Player – Lives, Points
• Game – Call all the player and enemies state and changes of levels.

Most Frightening Part:

• The tank movements. In the best case, I want the enemies tank to move based on the players movements. In other words, they would move close and close to the player’s tank. Since the player’s tank will constantly moving, the enemy target path will also change accordingly.
• Levels. I think it is quite difficult to have the correct parameters for each level. As the level increases, the parameters such as speed and number of enemies will also increase.

Risk Prevention:

• I will continue to come up with a way to have the tank targeting the player’s tank. However, in the worst case scenario, I can make the tank to move randomly. As I increase the number of tanks, the disadvantage of not having targeting tank will disappear.
• I will try to be careful with the game state control. It is really important to know when a player wins to move to the next level. Or when the player loses to allow replay sufficiently. Therefore, I will make sure these parameters are set up appropriately for different levels.

Reading through Golan Levin’s musings on computer vision feels like sitting down with a mad scientist who’s decided to use his powers for art instead of world domination. It’s like he’s taken this super complex tech thing and turned it into a playground. Imagine computers squinting at us, trying to make sense of what they’re seeing, and then artists stepping in saying, “Let’s make it fun.” Levin essentially tells us that with a bit of creativity, even the geekiest tech can be as expressive and unpredictable as a paint-splattered canvas. Who knew algorithms could be party tricks for the digital age? It’s a reminder that innovation isn’t just about serious business applications; it’s also about making cool stuff just because we can.

Computer Vision is full of bits and bytes. There is no way perfect algorithm to obtain the computer vision of the background. The computer vision seems to only see the whole back as a series of pixels. This means that without human to define what these pixels are, the computer does not know what to do with the image in their vision. In other words, the image data is a stream of nonsense to the computer.

The way the computer analyze the changes in its vision is similar to how the sensors work, I think. For example, in one of the techniques mentioned in the reading, “detecting motion” is done by comparing the pixels of the first frame (F1) and the subsequent frame (F2). Any changes in pixels detected by the processor can be concluded that was a movement in the vision. However, it is clear that this technique is unable to recognize what type of shape is it as well as the color of the moving objects. Similarly, since this algorithm is created to only detect motion, it does not understand the general image that it is capturing.

Therefore, computer vision can only do good at the jobs that t is assigned.

The article provides a comprehensive overview of the history, techniques, and applications of computer vision in the interactive media arts. It is interesting to see how artists have used computer vision to create novel and expressive experiences, such as Myron Krueger’s Videoplace, Rafael Lozano-Hemmer’s Standards and Double Standards, and David Rokeby’s Sorting Daemon. The article also explains some of the basic algorithms and tools for implementing computer vision, such as frame differencing, background subtraction, brightness thresholding, and object tracking. I find these algorithms useful and inspiring for my own projects, as they enable me to detect and analyze the movements and gestures of people and objects in real time.

Messa di Voce by Golan Levin and Zachary Lieberman was very interesting to me. This is an audiovisual performance that uses computer vision to track the locations of the performers’ heads and analyze the sounds they produce. The system then generates graphics that are projected from the performers’ mouths, creating a dynamic and expressive dialogue between sound and image. I think this artwork is a creative and engaging example of how computer vision can augment and transform human communication and expression.

I found Cheese by Christian Möller pretty funny. This is an installation that uses computer vision to measure the sincerity of the smiles of six actresses. The system uses an emotion recognition algorithm to detect the changes in the facial expressions of the actresses, and alerts them when their smiles fall below a certain threshold. I think this artwork is a humorous and critical commentary on the social and cultural expectations of happiness and beauty, as well as the limitations and biases of computer vision.

The reading for this week was about what computer vision is and how it is integrated into the field of interactive media. First of all, the book mentions Videoplace, the first interactive artwork of Mysron Krueger to incorporate computer vision, which was also one of the first interactive artworks. This fact made me realize that the development of technology can also fuel the expansion of other fields. Also, looking at different examples of the integration of computer vision and art, I was just amazed at how the development of technology can also enable artists more creative and innovative ways to create artwork or projects.

I particularly found Sorting Daemon (2003) by David Rokeby intriguing among the examples. This artwork was made by capturing and extracting people’s images who are on the street and dividing them according to areas of similar color. While I truly found his idea of creating such artwork very innovative, this artwork made me question the captured people’s portrait rights. To me, this looked like a violation of people’s portrait rights, which is the right of people to oppose the publication of his/her portrait, as we can see the faces of captured people very clearly in the artwork. Looking at the massive number of captured portraits in the artwork and the description of the artwork, it seemed like the artist just placed the camera somewhere without people knowing the existence of the camera in that location. Then, I came to wonder where we should draw the line between “artwork” and “crime” when the theme of the artwork involves such surveillance and release of the captured portraits without the allowance of the captured people.

Another insight I had after finishing the reading is how seemingly very difficult thing is not that difficult and it is something that everyone could do if he/she knows the logic. At first, when the author talked about how computer vision is actually not that hard concept, I doubted the author. However, after reading the entire article, I came to realize that the author was right. Every time I passed by the IM lab, I was amazed at the interactive installation with the changing Chinese characters capturing our moves and thought of it as a result of very complicated programming that only experts could make. While I do not know the “real” logic behind the programming of that installation, I realized that maybe this could be the application of one of the elementary computer vision techniques that the book mentions. I guess that the installation is the application of the detection through brightness thresholding or frame differencing method. I am leaning more toward brightness thresholding because it seems like there is an assigned Chinese character for each level of brightness if you look at the installation closer. Likewise, this book made me think that sometimes, seemingly complicated things could be in fact not that complicated.

Concept

My concept for the midterm project is to recreate and build upon the 1978 Atari game Asteroids. As an iconic arcade classic, Asteroids presents an exciting challenge for modern reinterpretation and expansion.

Design

Here’s a breakdown of a possible architecture:

Main Class:

• The main class serves as the entry point of the game and manages the overarching game loop. It initializes key components such as the player, asteroids, bullets, and handles user input and game state transitions. This class orchestrates the flow of the game, updating and rendering elements accordingly.

Player Class:

• The player class encapsulates the behavior and attributes of the player’s spaceship. It includes properties such as position, velocity, rotation, and health. Methods within this class handle player movement, shooting bullets, and responding to collisions with asteroids or other game entities.

Asteroid Class:

• The asteroid class represents the asteroids that populate the game world. Each asteroid object possesses properties such as position, size, velocity, and rotation. Methods within this class manage asteroid movement, splitting into smaller asteroids upon destruction, and rendering.

Bullet Class:

• The bullet class defines the behavior and properties of the bullets fired by the player’s spaceship. It includes attributes such as position, velocity, and lifespan. Methods within this class handle bullet movement, collision detection with asteroids, and removal from the game world upon reaching the end of their lifespan or hitting a target.

Collision Detection:

• A separate module or class is dedicated to collision detection algorithms. This component detects collisions between various game entities, such as player-asteroid collisions, bullet-asteroid collisions, and potentially asteroid-asteroid collisions if asteroids can collide with each other. Implementing efficient collision detection algorithms, such as bounding box or circle intersection tests, is crucial for optimizing game performance.

Game Manager Class:

• The game manager class oversees high-level game logic and state management. It handles tasks such as initializing the game world, updating game entities, checking for win or loss conditions, and transitioning between different game states (e.g., main menu, gameplay, game over). This class facilitates modularization and separation of concerns, promoting code readability and maintainability.

User Interface (UI) Components:

• UI elements such as score displays, health bars, and menu screens may be implemented as separate classes or modules. These components interact with the game manager and player classes to reflect changes in game state and provide feedback to the player.

Risk Management and Mitigation

One challenge lies in implementing the collision detection algorithms, especially considering the dynamic nature of objects in the game world. To mitigate this risk, I will allocate time to research and experiment with different collision detection techniques. By prototyping and testing these algorithms early on, I aim to identify the most effective approach and integrate it into the project.