This week’s reading was interesting from the technical and social sides of things. The concept of creating something for experimentation, and even art, to then transform it into a weapon of mass surveillance (and even war) is something that can provoke many thoughts. Indeed, I know that my iPhone applies many processes internally, which allowed face recognition to happen in the first place, and probably surveillance of me without my consent (just probably).
I remember when I was a child. I was impressed when I first saw another human face through Skype in 2013. Yes, the machine was just displaying a set of pixels that were observable through my screen, but the fact that another human, who was residing in another country, was actually seen was truly impressive! Then the smartphone arrived, and more complex algorithms were starting to get implemented, which allowed for further experimentation and user engagement.
Therefore, I would argue that the reading really did not change that much my views on video image processing, aside from the thought-provoking fact that a war medium originated from an art experiment. And I have not used many of the algorithms that allow modifying the perception of you through the screens of other people.
In conclusion, I would not argue that the author is biased, as he presents factual information and techniques for students or professionals to implement in their careers. Nevertheless, most information could be considered outdated as it was presented around 2006–2007. In today’s world, we have Artificial Intelligences that are capable of understanding humans and even replicating them through simulations, which can then be shared as a small clip through social media.
I really enjoyed reading this text, particularly discovering the artworks on surveillance. I find it quite ironic (but in a good way) that a lot of artworks that want to talk about surveillance, employ some sort of surveillance themselves. Of course, here, surveillance has to be looked at in the wider sense. Usually, when we think surveillance we imagine CCTV or hidden cameras. However, as the text mentions, human tracking is also included under surveillance. So, even artworks that simply track your body movements, as basic as the Cheese installation by Möller, can be said to be «surveilling» you in a way. There are also the other forms of human tracking, like Lozano-Hemmer’s Standards and Double Standards, which involve more intense forms of surveillance and tracking. To have items following you takes the «scrutiny» to a different level. Reading about this artwork reminded me of the Manar Abu Dhabi exhibition which took place last fall. It included 35 artworks, many of which were installations. The one that Lozano-Hemmer’s work reminded me of is the Abu Dhabi Dots by Carsten Höller, which was exhibited on the shore of the Corniche beach. Essentially, there are projected spotlights on the sand, and when you stand on them and move, they move with you.
After reading about it now, I found out that it is actually a game. Here is the description:
“Operating as a reward and punishment game, you can upgrade from red to blue by overlapping your dot with a blue dot, and can then move to green, and white. People you are playing with who are overlapped are downgraded.
But if a series of red dots join an interrupted chain or cluster, the dots all become white. If not, the game continues until 7 or more white dots don’t overlap with other white dots any more.”
Before reading this text, I hadn’t thought much about it, but now I realize that it is a form of tracking after all. And although it seems like a game for children, the whole «reward and punishment» part of it actually reminds me a bit of surveillance and scrutiny, where your actions can favor you but also disadvantage you, while also impacting other people around you. It is therefore interesting to see how computer-based tracking artworks feed into the idea of surveillance, and potentially how they use our data (and store them?).
For my midterm project, I am inspired by my childhood game: a classic Tank Game in the small brick game console.
The game is basically divided into different levels. As you come to the next level, the speed come from the bullet will be faster and more component will spawn in the game. Also, the player only has a limited number of lives. When those lives are used up, it is game over. The user can move around to dodge the bullet. However, because the screen is relatively small to contain both player and enemies, it becomes more and more difficult to win.
Design:
I aim for a similar design as its original. I want player tank will have a brick-like display which is distinguished from the enemies. However, instead of doing a monochrome game similar to the original, I want to add colors and random movements to the enemies. It will be either the enemies will move close and close to the player’s tank or the tank will move and shoot bullets randomly.
Object Oriented Class:
Tank – Need to have 2 designs, one for player and one for the enemies.
Player – Lives, Points
Game – Call all the player and enemies state and changes of levels.
Most Frightening Part:
The tank movements. In the best case, I want the enemies tank to move based on the players movements. In other words, they would move close and close to the player’s tank. Since the player’s tank will constantly moving, the enemy target path will also change accordingly.
Levels. I think it is quite difficult to have the correct parameters for each level. As the level increases, the parameters such as speed and number of enemies will also increase.
Risk Prevention:
I will continue to come up with a way to have the tank targeting the player’s tank. However, in the worst case scenario, I can make the tank to move randomly. As I increase the number of tanks, the disadvantage of not having targeting tank will disappear.
I will try to be careful with the game state control. It is really important to know when a player wins to move to the next level. Or when the player loses to allow replay sufficiently. Therefore, I will make sure these parameters are set up appropriately for different levels.
Reading through Golan Levin’s musings on computer vision feels like sitting down with a mad scientist who’s decided to use his powers for art instead of world domination. It’s like he’s taken this super complex tech thing and turned it into a playground. Imagine computers squinting at us, trying to make sense of what they’re seeing, and then artists stepping in saying, “Let’s make it fun.” Levin essentially tells us that with a bit of creativity, even the geekiest tech can be as expressive and unpredictable as a paint-splattered canvas. Who knew algorithms could be party tricks for the digital age? It’s a reminder that innovation isn’t just about serious business applications; it’s also about making cool stuff just because we can.
Computer Vision is full of bits and bytes. There is no way perfect algorithm to obtain the computer vision of the background. The computer vision seems to only see the whole back as a series of pixels. This means that without human to define what these pixels are, the computer does not know what to do with the image in their vision. In other words, the image data is a stream of nonsense to the computer.
The way the computer analyze the changes in its vision is similar to how the sensors work, I think. For example, in one of the techniques mentioned in the reading, “detecting motion” is done by comparing the pixels of the first frame (F1) and the subsequent frame (F2). Any changes in pixels detected by the processor can be concluded that was a movement in the vision. However, it is clear that this technique is unable to recognize what type of shape is it as well as the color of the moving objects. Similarly, since this algorithm is created to only detect motion, it does not understand the general image that it is capturing.
Therefore, computer vision can only do good at the jobs that t is assigned.
The article provides a comprehensive overview of the history, techniques, and applications of computer vision in the interactive media arts. It is interesting to see how artists have used computer vision to create novel and expressive experiences, such as Myron Krueger’s Videoplace, Rafael Lozano-Hemmer’s Standards and Double Standards, and David Rokeby’s Sorting Daemon. The article also explains some of the basic algorithms and tools for implementing computer vision, such as frame differencing, background subtraction, brightness thresholding, and object tracking. I find these algorithms useful and inspiring for my own projects, as they enable me to detect and analyze the movements and gestures of people and objects in real time.
Messa di Voce by Golan Levin and Zachary Lieberman was very interesting to me. This is an audiovisual performance that uses computer vision to track the locations of the performers’ heads and analyze the sounds they produce. The system then generates graphics that are projected from the performers’ mouths, creating a dynamic and expressive dialogue between sound and image. I think this artwork is a creative and engaging example of how computer vision can augment and transform human communication and expression.
I found Cheese by Christian Möller pretty funny. This is an installation that uses computer vision to measure the sincerity of the smiles of six actresses. The system uses an emotion recognition algorithm to detect the changes in the facial expressions of the actresses, and alerts them when their smiles fall below a certain threshold. I think this artwork is a humorous and critical commentary on the social and cultural expectations of happiness and beauty, as well as the limitations and biases of computer vision.
The reading for this week was about what computer vision is and how it is integrated into the field of interactive media. First of all, the book mentions Videoplace, the first interactive artwork of Mysron Krueger to incorporate computer vision, which was also one of the first interactive artworks. This fact made me realize that the development of technology can also fuel the expansion of other fields. Also, looking at different examples of the integration of computer vision and art, I was just amazed at how the development of technology can also enable artists more creative and innovative ways to create artwork or projects.
I particularly found Sorting Daemon (2003) by David Rokeby intriguing among the examples. This artwork was made by capturing and extracting people’s images who are on the street and dividing them according to areas of similar color. While I truly found his idea of creating such artwork very innovative, this artwork made me question the captured people’s portrait rights. To me, this looked like a violation of people’s portrait rights, which is the right of people to oppose the publication of his/her portrait, as we can see the faces of captured people very clearly in the artwork. Looking at the massive number of captured portraits in the artwork and the description of the artwork, it seemed like the artist just placed the camera somewhere without people knowing the existence of the camera in that location. Then, I came to wonder where we should draw the line between “artwork” and “crime” when the theme of the artwork involves such surveillance and release of the captured portraits without the allowance of the captured people.
Another insight I had after finishing the reading is how seemingly very difficult thing is not that difficult and it is something that everyone could do if he/she knows the logic. At first, when the author talked about how computer vision is actually not that hard concept, I doubted the author. However, after reading the entire article, I came to realize that the author was right. Every time I passed by the IM lab, I was amazed at the interactive installation with the changing Chinese characters capturing our moves and thought of it as a result of very complicated programming that only experts could make. While I do not know the “real” logic behind the programming of that installation, I realized that maybe this could be the application of one of the elementary computer vision techniques that the book mentions. I guess that the installation is the application of the detection through brightness thresholding or frame differencing method. I am leaning more toward brightness thresholding because it seems like there is an assigned Chinese character for each level of brightness if you look at the installation closer. Likewise, this book made me think that sometimes, seemingly complicated things could be in fact not that complicated.
My concept for the midterm project is to recreate and build upon the 1978 Atari game Asteroids. As an iconic arcade classic, Asteroids presents an exciting challenge for modern reinterpretation and expansion.
Design
Here’s a breakdown of a possible architecture:
Main Class:
The main class serves as the entry point of the game and manages the overarching game loop. It initializes key components such as the player, asteroids, bullets, and handles user input and game state transitions. This class orchestrates the flow of the game, updating and rendering elements accordingly.
Player Class:
The player class encapsulates the behavior and attributes of the player’s spaceship. It includes properties such as position, velocity, rotation, and health. Methods within this class handle player movement, shooting bullets, and responding to collisions with asteroids or other game entities.
Asteroid Class:
The asteroid class represents the asteroids that populate the game world. Each asteroid object possesses properties such as position, size, velocity, and rotation. Methods within this class manage asteroid movement, splitting into smaller asteroids upon destruction, and rendering.
Bullet Class:
The bullet class defines the behavior and properties of the bullets fired by the player’s spaceship. It includes attributes such as position, velocity, and lifespan. Methods within this class handle bullet movement, collision detection with asteroids, and removal from the game world upon reaching the end of their lifespan or hitting a target.
Collision Detection:
A separate module or class is dedicated to collision detection algorithms. This component detects collisions between various game entities, such as player-asteroid collisions, bullet-asteroid collisions, and potentially asteroid-asteroid collisions if asteroids can collide with each other. Implementing efficient collision detection algorithms, such as bounding box or circle intersection tests, is crucial for optimizing game performance.
Game Manager Class:
The game manager class oversees high-level game logic and state management. It handles tasks such as initializing the game world, updating game entities, checking for win or loss conditions, and transitioning between different game states (e.g., main menu, gameplay, game over). This class facilitates modularization and separation of concerns, promoting code readability and maintainability.
User Interface (UI) Components:
UI elements such as score displays, health bars, and menu screens may be implemented as separate classes or modules. These components interact with the game manager and player classes to reflect changes in game state and provide feedback to the player.
Risk Management and Mitigation
One challenge lies in implementing the collision detection algorithms, especially considering the dynamic nature of objects in the game world. To mitigate this risk, I will allocate time to research and experiment with different collision detection techniques. By prototyping and testing these algorithms early on, I aim to identify the most effective approach and integrate it into the project.
Since I love the concept of interactivity, I thought to myself: “It is possible to simulate physics and collisions between objects in a simple art?”. Many ideas came into my mind, but I fixed into creating something simple and, arguably, realistic: A interactable painting that displays a glass cup and ice cubes, that you can interact with and displays a sense of physics and collisions.
Design
When I arrived at my idea, I first started to narrow my vision on what is possible to make. In my previous projects, I had done things such as abstract paintings with the use of figures, and “physics” like simulation where diamonds fall and stack on top of each other (although honestly they look more like ice). Given this, I made a sketch to understand how can I implement these physics:
Me trying to figure out the collisions and physics.
And also searched a reference in Google Images for the ideas for the art, thus I arrived at this image created by crayon.ai:
Prompt: Two near transparent empty cups of glass on a messa table, oil painting style.
In order to interact with the painting, the user only needs to point at the objects and press at them, either only clicking or holding it to be able to move the object freely. Also, the sound design for this art will be kept minimum, as art, in my opinion, should be a relaxing experience. If a glass cup falls on the wooden table, the appropriate sound will play, if an ice cube falls into the glass cup, the appropriate sound will also play.
Therefore, I found the following references for the music used is the following:
This makes me want to create a nostalgic-like art.
Or also this one, which is more classical and makes me go through a more conventional art style:
Challenges
Physics are hard to program and simulate, due to the many conditions one has to create and also ensure to keep the illusion that the properties could be seen in real life. Due to this, I had to ignore the idea of also adding rotations since it would be difficult to complete in a timely manner.
For example, my current code for collisions and gravity looks like this:
//Display cubes and apply collision and gravity.
for (let i = 0; i < cubes.length; i++) {
cubes[i].display();
//We check collision with the glasses. (Since we are simulating glass cups, the collision for the top is disabled for the glass cups}.
for (let c = 0; c < glasses.length; c++) {
//Check if it is on the X range of the table.
//Check if it is on the X range of a cup.
if (
cubes[i].x < glasses[c].x + glasses[c].w &&
cubes[i].x + cubes[i].w > glasses[c].x
) {
//I am standing on an edge of a cup on the right side?
if (
cubes[i].x < glasses[c].x + glasses[c].w &&
cubes[i].x + cubes[i].w > glasses[c].x + glasses[c].w
) {
print("right corner detected");
if (cubes[i].y + cubes[i].h >= glasses[c].y) {
}
}
//What about the left side?
else if (
cubes[i].x < glasses[c].x &&
cubes[i].x + cubes[i].w > glasses[c].x
) {
if (cubes[i].y + cubes[i].h >= glasses[c].y) {
print("left corner detected");
}
} else {
cubes[i].gravity();
}
//Please do not pass the cup.
if (cubes[i].y + cubes[i].h >= glasses[c].y + glasses[c].h) {
print("Im here 3");
}
//Nothing else we can check.
else {
cubes[i].gravity();
}
//Please do not pass
//If the cube is on the X range, then check Y axis to apply collision
} else {
cubes[i].gravity();
}
}
//We check collision with the cubes. Also, it is important to add the number 1 to avoid comparing the current index.
for (let c = 0; c < cubes.length; c++) {
//Check if it is on the X range of a cube.
if (
cubes[i].x < cubes[c].x + cubes[c].w &&
cubes[i].x + cubes[i].w > cubes[c].x
) {
//I am standing on top of the cube?
print("top of a cube detected");
if (cubes[i].y + cubes[i].h >= cubes[c].y) {
//Make a comparison.
} else {
cubes[i].gravity();
}
//If the cube is on the X range, then check Y axis to apply collision
} else {
cubes[i].gravity();
}
}
Simulating the art seems challenging, but in theory, it should be possible to finish if I add images that are similarly.
Due to the nature of this project, I want to keep the player immerse in the art while also understanding (intuitively through the elements of the painting) that it is interactable.
Keeping all the classes consistent among the shared properties. Likewise, in order to keep best programming practices, I decided to divided it into the following way:
Items:
Glass.js (Glass cup)
Cubes.js (Cubes)
Items that are part of Scenario.js:
Table (A wooden table)
GlassSpawner (Still deciding on how to represent it, because I want to keep consistency with the art)
Menu & HUD:
Menu.js (Can display both the start menu and instructions)
UI.js (Buttons that will be displayed which allows the player to return to the main menu or disable the music and/or SFX).
Audio:
Audio.js (For both music and SFX).
Risk Prevention
In order to ensure I finish the project, I need to work at least per day on it. The hours can vary since at the time I am writing this I am still a college student.
The menus are possible to make without issues, the only difficult part that risks the entire project is the creation of the physics and collisions.
Keep in mind that, if difficulties present, I can ask the professor to clarify or suggest new ideas on how to approach certain part of the code.
Used code that I made before for the physics and collisions in order to reduce time spent.
Midterm project in its current form
Here is a quick showcase of how my midterm looks as of February 26th, 2024. This will be kept as it is, to showcase the progress once it is finished:
I have not had the time to fix the cube physics, sadly. Also, keep in mind, you can move the red square with the mouse if you hold the left click.
Conclusion
This idea was born due to the need of experimenting with new, challenging ideas. Simulating these properties in a canvas is hard without using any reference, but at the end, this will help me increase my confidence as a programmer.
For the midterm, I plan to make a game called Pyro Dancer. It is a single-player endless runner game, where players assume the role of a knight trying to save the princess. Previous notable titles, such as Castlevania and Metroid inspire the game.
Concept
Pyro Dancer uses pixel art as its main artistic choice. I believe that this is the element that captures the experience and feeling of playing similar titles. Paired with 8-bit sound and theme by Haynes, Pyro Dancer tries to mimic the retro feel of older games.
Knight Sprite
Sketches, Ideas, Drawings!
As the name suggests, Pyro Dancer will feature a knight running through a supervillain castle (unknown for now) to save the princess. However, he is greeted by dozens of fireballs in the process.
Gameplay
Players will have to dodge the fireballs by going up or down. They also have the choice to speed up their running by pressing the D key. As the score increases, so does the difficulty.
Above a certain score, players will be greeted by stationary monsters that instantly kill them upon touching. This would balance out the difficulty and create varied gameplay, as dodging only fireballs would be boring!
To win the game, players must reach a score of 1000 and proceed to the boss battle. (It is still in development, and I am unsure whether I will have the time to code it!)
Development Process
While trying to program the game, I ran across a dilemma.
I could program things from scratch, but it will take more time and the chances of running into bugs is higher.
Or, I could use some external libraries to help me develop the game, where I could devote my time and energy to create a better game instead.
I used an external Javascript library called p5 Play to help me develop the game. The library introduces a sprite class that allows me to easily manage the characters, obstacles, and even the game environment with a built-in physics engine.
Asset-wise, I have compiled a lot of the resources in this document. Please refer to it for the assets that I use.
Programming-wise, I have laid down the basics and finished testing the library itself.
Main Menu Screen (Still in Development)
Debugging Game Screen
Hence, I only need to clean up the code and mash them all together plus adding some sound effects to make sure they are neatly packed as a game.
Concerns & Optimisms
Because of how busy and tight the deadlines are, I am concerned that I might not be able to play test the game extensively. By the end product, there might be a lot of bugs or unexpected exploits (could be a feature/mechanic too!) that would go unnoticed. However, I am optimistic that I will be able to deliver an enjoyable game in the end.
For some reason, I could not import the p5 sketches into this WordPress without it spiraling out of control. I am posting my reference links below instead!
Me trying to figure out the collisions and physics.
Sketches, Ideas, Drawings!
