Month: February 2024

The reading for this week was about what computer vision is and how it is integrated into the field of interactive media. First of all, the book mentions Videoplace, the first interactive artwork of Mysron Krueger to incorporate computer vision, which was also one of the first interactive artworks. This fact made me realize that the development of technology can also fuel the expansion of other fields. Also, looking at different examples of the integration of computer vision and art, I was just amazed at how the development of technology can also enable artists more creative and innovative ways to create artwork or projects.

I particularly found Sorting Daemon (2003) by David Rokeby intriguing among the examples. This artwork was made by capturing and extracting people’s images who are on the street and dividing them according to areas of similar color. While I truly found his idea of creating such artwork very innovative, this artwork made me question the captured people’s portrait rights. To me, this looked like a violation of people’s portrait rights, which is the right of people to oppose the publication of his/her portrait, as we can see the faces of captured people very clearly in the artwork. Looking at the massive number of captured portraits in the artwork and the description of the artwork, it seemed like the artist just placed the camera somewhere without people knowing the existence of the camera in that location. Then, I came to wonder where we should draw the line between “artwork” and “crime” when the theme of the artwork involves such surveillance and release of the captured portraits without the allowance of the captured people.

Another insight I had after finishing the reading is how seemingly very difficult thing is not that difficult and it is something that everyone could do if he/she knows the logic. At first, when the author talked about how computer vision is actually not that hard concept, I doubted the author. However, after reading the entire article, I came to realize that the author was right. Every time I passed by the IM lab, I was amazed at the interactive installation with the changing Chinese characters capturing our moves and thought of it as a result of very complicated programming that only experts could make. While I do not know the “real” logic behind the programming of that installation, I realized that maybe this could be the application of one of the elementary computer vision techniques that the book mentions. I guess that the installation is the application of the detection through brightness thresholding or frame differencing method. I am leaning more toward brightness thresholding because it seems like there is an assigned Chinese character for each level of brightness if you look at the installation closer. Likewise, this book made me think that sometimes, seemingly complicated things could be in fact not that complicated.

Concept

My concept for the midterm project is to recreate and build upon the 1978 Atari game Asteroids. As an iconic arcade classic, Asteroids presents an exciting challenge for modern reinterpretation and expansion.

Design

Here’s a breakdown of a possible architecture:

Main Class:

• The main class serves as the entry point of the game and manages the overarching game loop. It initializes key components such as the player, asteroids, bullets, and handles user input and game state transitions. This class orchestrates the flow of the game, updating and rendering elements accordingly.

Player Class:

• The player class encapsulates the behavior and attributes of the player’s spaceship. It includes properties such as position, velocity, rotation, and health. Methods within this class handle player movement, shooting bullets, and responding to collisions with asteroids or other game entities.

Asteroid Class:

• The asteroid class represents the asteroids that populate the game world. Each asteroid object possesses properties such as position, size, velocity, and rotation. Methods within this class manage asteroid movement, splitting into smaller asteroids upon destruction, and rendering.

Bullet Class:

• The bullet class defines the behavior and properties of the bullets fired by the player’s spaceship. It includes attributes such as position, velocity, and lifespan. Methods within this class handle bullet movement, collision detection with asteroids, and removal from the game world upon reaching the end of their lifespan or hitting a target.

Collision Detection:

• A separate module or class is dedicated to collision detection algorithms. This component detects collisions between various game entities, such as player-asteroid collisions, bullet-asteroid collisions, and potentially asteroid-asteroid collisions if asteroids can collide with each other. Implementing efficient collision detection algorithms, such as bounding box or circle intersection tests, is crucial for optimizing game performance.

Game Manager Class:

• The game manager class oversees high-level game logic and state management. It handles tasks such as initializing the game world, updating game entities, checking for win or loss conditions, and transitioning between different game states (e.g., main menu, gameplay, game over). This class facilitates modularization and separation of concerns, promoting code readability and maintainability.

User Interface (UI) Components:

• UI elements such as score displays, health bars, and menu screens may be implemented as separate classes or modules. These components interact with the game manager and player classes to reflect changes in game state and provide feedback to the player.

Risk Management and Mitigation

One challenge lies in implementing the collision detection algorithms, especially considering the dynamic nature of objects in the game world. To mitigate this risk, I will allocate time to research and experiment with different collision detection techniques. By prototyping and testing these algorithms early on, I aim to identify the most effective approach and integrate it into the project.

Concept

Since I love the concept of interactivity, I thought to myself: “It is possible to simulate physics and collisions between objects in a simple art?”. Many ideas came into my mind, but I fixed into creating something simple and, arguably, realistic: A interactable painting that displays a glass cup and ice cubes, that you can interact with and displays a sense of physics and collisions.

Design

When I arrived at my idea, I first started to narrow my vision on what is possible to make. In my previous projects, I had done things such as abstract paintings with the use of figures, and “physics” like simulation where diamonds fall and stack on top of each other (although honestly they look more like ice). Given this, I made a sketch to understand how can I implement these physics:

Me trying to figure out the collisions and physics.

And also searched a reference in Google Images for the ideas for the art, thus I arrived at this image created by crayon.ai:

In order to interact with the painting, the user only needs to point at the objects and press at them, either only clicking or holding it to be able to move the object freely. Also, the sound design for this art will be kept minimum, as art, in my opinion, should be a relaxing experience. If a glass cup falls on the wooden table, the appropriate sound will play, if an ice cube falls into the glass cup, the appropriate sound will also play.

Therefore, I found the following references for the music used is the following:

This makes me want to create a nostalgic-like art.

Or also this one, which is more classical and makes me go through a more conventional art style:

Challenges

• Physics are hard to program and simulate, due to the many conditions one has to create and also ensure to keep the illusion that the properties could be seen in real life. Due to this, I had to ignore the idea of also adding rotations since it would be difficult to complete in a timely manner.

For example, my current code for collisions and gravity looks like this:

//Display cubes and apply collision and gravity.
for (let i = 0; i < cubes.length; i++) {
  cubes[i].display();

  //We check collision with the glasses. (Since we are simulating glass cups, the collision for the top is disabled for the glass cups}.
  for (let c = 0; c < glasses.length; c++) {
    //Check if it is on the X range of the table.

    //Check if it is on the X range of a cup.
    if (
      cubes[i].x < glasses[c].x + glasses[c].w &&
      cubes[i].x + cubes[i].w > glasses[c].x
    ) {
      //I am standing on an edge of a cup on the right side?
      if (
        cubes[i].x < glasses[c].x + glasses[c].w &&
        cubes[i].x + cubes[i].w > glasses[c].x + glasses[c].w
      ) {
        print("right corner detected");
        if (cubes[i].y + cubes[i].h >= glasses[c].y) {
        }
      }

      //What about the left side?
      else if (
        cubes[i].x < glasses[c].x &&
        cubes[i].x + cubes[i].w > glasses[c].x
      ) {
        if (cubes[i].y + cubes[i].h >= glasses[c].y) {
          print("left corner detected");
        }
      } else {
        cubes[i].gravity();
      }

      //Please do not pass the cup.
      if (cubes[i].y + cubes[i].h >= glasses[c].y + glasses[c].h) {
        print("Im here 3");
      }

      //Nothing else we can check.
      else {
        cubes[i].gravity();
      }

      //Please do not pass

      //If the cube is on the X range, then check Y axis to apply collision
    } else {
      cubes[i].gravity();
    }
  }

  //We check collision with the cubes. Also, it is important to add the number 1 to avoid comparing the current index.
  for (let c = 0; c < cubes.length; c++) {
    //Check if it is on the X range of a cube.
    if (
      cubes[i].x < cubes[c].x + cubes[c].w &&
      cubes[i].x + cubes[i].w > cubes[c].x
    ) {
      //I am standing on top of the cube?
      print("top of a cube detected");
      if (cubes[i].y + cubes[i].h >= cubes[c].y) {
        //Make a comparison.
      } else {
        cubes[i].gravity();
      }

      //If the cube is on the X range, then check Y axis to apply collision
    } else {
      cubes[i].gravity();
    }
  }

• Simulating the art seems challenging, but in theory, it should be possible to finish if I add images that are similarly.
• Due to the nature of this project, I want to keep the player immerse in the art while also understanding (intuitively through the elements of the painting) that it is interactable.
• Keeping all the classes consistent among the shared properties. Likewise, in order to keep best programming practices, I decided to divided it into the following way:
  • • Items:
      • • Glass.js (Glass cup)
        • Cubes.js (Cubes)
        • Items that are part of Scenario.js:
          • Table (A wooden table)
          • GlassSpawner (Still deciding on how to represent it, because I want to keep consistency with the art)
    • Menu & HUD:
      • Menu.js (Can display both the start menu and instructions)
      • UI.js (Buttons that will be displayed which allows the player to return to the main menu or disable the music and/or SFX).
    • Audio:
      • Audio.js (For both music and SFX).

Risk Prevention

• In order to ensure I finish the project, I need to work at least per day on it. The hours can vary since at the time I am writing this I am still a college student.
• The menus are possible to make without issues, the only difficult part that risks the entire project is the creation of the physics and collisions.
• Keep in mind that, if difficulties present, I can ask the professor to clarify or suggest new ideas on how to approach certain part of the code.
• Used code that I made before for the physics and collisions in order to reduce time spent.

Midterm project in its current form

Here is a quick showcase of how my midterm looks as of February 26th, 2024. This will be kept as it is, to showcase the progress once it is finished:

I have not had the time to fix the cube physics, sadly. Also, keep in mind, you can move the red square with the mouse if you hold the left click.

Conclusion

This idea was born due to the need of experimenting with new, challenging ideas. Simulating these properties in a canvas is hard without using any reference, but at the end, this will help me increase my confidence as a programmer.

For the midterm, I plan to make a game called Pyro Dancer. It is a single-player endless runner game, where players assume the role of a knight trying to save the princess. Previous notable titles, such as Castlevania and Metroid inspire the game.

Concept

Pyro Dancer uses pixel art as its main artistic choice. I believe that this is the element that captures the experience and feeling of playing similar titles. Paired with 8-bit sound and theme by Haynes, Pyro Dancer tries to mimic the retro feel of older games.

Knight Sprite

Sketches, Ideas, Drawings!

As the name suggests, Pyro Dancer will feature a knight running through a supervillain castle (unknown for now) to save the princess. However, he is greeted by dozens of fireballs in the process.

Gameplay

Players will have to dodge the fireballs by going up or down. They also have the choice to speed up their running by pressing the D key. As the score increases, so does the difficulty.

Above a certain score, players will be greeted by stationary monsters that instantly kill them upon touching. This would balance out the difficulty and create varied gameplay, as dodging only fireballs would be boring!

To win the game, players must reach a score of 1000 and proceed to the boss battle. (It is still in development, and I am unsure whether I will have the time to code it!)

Development Process

While trying to program the game, I ran across a dilemma.

I could program things from scratch, but it will take more time and the chances of running into bugs is higher.

Or, I could use some external libraries to help me develop the game, where I could devote my time and energy to create a better game instead.

I used an external Javascript library called p5 Play to help me develop the game. The library introduces a sprite class that allows me to easily manage the characters, obstacles, and even the game environment with a built-in physics engine.

Asset-wise, I have compiled a lot of the resources in this document. Please refer to it for the assets that I use.

Programming-wise, I have laid down the basics and finished testing the library itself.

Main Menu Screen (Still in Development)

Debugging Game Screen

Hence, I only need to clean up the code and mash them all together plus adding some sound effects to make sure they are neatly packed as a game.

Concerns & Optimisms

Because of how busy and tight the deadlines are, I am concerned that I might not be able to play test the game extensively. By the end product, there might be a lot of bugs or unexpected exploits (could be a feature/mechanic too!) that would go unnoticed. However, I am optimistic that I will be able to deliver an enjoyable game in the end.

For some reason,  I could not import the p5 sketches into this WordPress without it spiraling out of control. I am posting my reference links below instead!

References

p5.js Web Editor | Pyro Dancer – Main Menu Cleaned (p5js.org)

p5.js Web Editor | Pyro Dancer – Game Screen (p5js.org)

p5.js Web Editor | Pyro Dancer – Knight Animation copy (p5js.org)

p5.js Web Editor | Pyro Dancer – Tutorial Screen (p5js.org)

The article talks about computer vision which is a field of computer science responsible for designing algorithms that enable computers to provide some level of analysis for digital content. The article mainly talks about new ways of using computer vision, in digital media, art, and more which are nowadays accessible to a much wider range of consumers. 

There were many points that I found interesting in the article, starting from the methodologies behind vision algorithms to social implications. Myron Krueger’s legendary Videoplace reminded me of a project I did in last year’s class virtual body performance. I created an environment in Unity similar to how Videoplace used human canvas and allowed interaction with elements through movement. I also used bodily movements to signify unlocking chakras and connecting to the outer world through physical movements reflected in VR using body capture technology. While methodologies are different, at the core, they are both interactive performances that require a combination of computer technology, human creativity, and physical involvement.

The second and far more interesting point involves ethical considerations coming from the Golden Gate Bridge incident. How morally acceptable is it for us to record people’s final moments for all the world to see? And all this happening without consent is the most problematic aspect. In this case, there is a fine line between artistic expression and socially acceptable expectations. In the future, this line will be blurred further as various cameras around us start to capture more and more data, with better, more sophisticated algorithms. Soon, an immense amount of data concerning our daily routines, choices, feelings, and other personal information will be collected and stored in massive databases. This information will include data on our facial expressions, movements, and more. Unfortunately, there is potential for this data to be exploited by corrupt governments or large technological conglomerates.

On a final note, I agree that computer vision technology has enormous potential for innovation and creativity, but it must be used ethically and responsibly to ensure privacy/ethical concerns are addressed going forward into the future.