Category: F2023 – Mang

Computer Vision for Artists and Designers

We always see the stage but the backstage remains secret for most of us until we get interested and look at it by ourselves. Today’s material opened the curtains and showed the backstage of the computer vision. We all have been imposed on face and movement recognition to some extent, starting from the face masks in the Instagram filters and the face ID of the iPhone to the movement recognition in various artworks. Although I always wondered how it works, I didn’t have a chance to look at it by myself until today. 

Although I could predict beforehand that the motion is detected by the comparison of the pixels in each subsequent frame, the information about the detection through brightness threshold was new for me, yet so simple. It was interesting to read all these operations made by the computer, which is hidden from our eyes. After reading the article, I have a question whether the advanced technologies still operate the same way, or whether their way of detection might be changed over time. 

I am surprised by the amount of constraints and limitations of the detection of presence and motion, object tracking, etc. Although the technologies have developed rapidly since the day of the first interactive media artworks such as Videoplace, there are still many things, that need to be developed. For instance, simply the face ID in my iPhone recognizes me and my 11-year-old sister as one person. 

Last but not least, I would like to mention examples of artwork using computer vision. While some create an entertaining and aesthetically pleasing experience with voice recognition as Messa di Voce or movement recognition as Videoplace, others use computer vision to raise social issues such as Standards and Double Standards and Suicide Box. I liked the idea of Suicide Box in a way that this project could be developed further as a safety tracker. Every time the vertical motion is recognized, the net can be released under the bridge to catch the falling person and the signal can be sent to the police about possible suicide. Considering the statistics of 17 suicides in 100 days of deployment, the issue of suicide remains significant and needs a solution.

Design and Concept

Over the first few weeks of this class and our class projects, I’ve found myself leaning towards pieces that would work well in an art installation setting ( or at least I think so ), ones that don’t require much user interaction but is just pleasant to look at.

For my midterm, I wanted to do something similar, but at the same time incorporating some more interaction in it while retaining the essence of being an ‘installation piece’ . I was inspired by Chris Milk’s “The Treachery of Sanctuary”, Romy Achituv & Camille Utterback’s “Text Rain”, and more than the others, Scott Snibbe’s “Deep Walls”. I loved the concept of leaving behind little snippets of yourself, seeing the snippets left behind by other people, and the idea that for every memory that is added, a memory is lost.

Picture of Scott Snibbe’s “Deep Walls”

For my midterm project, I’d like to do something similar with a camera and a keyboard. On the screen, there will be a few black and white photo threshold’d to be fully black or white so that the person is not identifiable, along with some text that was left behind by that person. The next viewer will be given one of a few prompts, such as “What is your favorite memory?” , “Who do you miss the most?”, “What are you looking forward to?”, and the viewer can type their answers with the keyboard while the camera takes a photo of them without a prompt. Their memory and photo will be left on the screen, replacing one of the existing ‘memories’, causing it to be lost forever.

I’ve been thinking about the concept of memory for a while, and I thought that this will be a good project to incorporate my thoughts into a physical piece. The biggest challenge for this project I think will be getting the camera and photo-taking to work with Processing, and doing so in a meaningful way. I would also have to think of some good prompts for the piece, as some prompts would be too simple, while having them be too complicated might discourage viewers from typing.

It’s fascinating how many technologies were developed first for military use, before finding it’s way into the commercial space! This was the first time I’ve heard of computer vision originating from military origins, but I knew that the Internet was first invented for war-time communications, and super-glue was an accidental invention from other military-based research. Those inventions that were first made for something as violent as harm has eventually settled down and turned into technologies that we use in our everyday lives, and even in art.

I still think it’s funny how some tasks that are so simple for humans are extremely difficult problems in computation, and some tasks that are nigh-impossible for humans is easy for computers. Computers can calculate 3 to the power of 100 almost instantly, a task that no sane human can do. In the case of computer vision, any human can identify a pigeon in a picture, yet it took computers decades of research before it started being able to identify objects well.

I also found it interesting how many of the early applications of computer vision in projects has a theme of surveillance around it, and I think it reflects how the scene viewed cameras at the time — tools of surveillance. Standards and Double Standards and Suicide Box both use computer vision to highlight societal observations through the lens of an array of pixels.

Concept

For people who know me, I was a gamer my entire life, and I loved the concept of video games as art. I always believed that games themselves can convey meaning like any other artwork, and for some their aesthetic values might even exceed what people traditionally identify as art. Also due to games’ nature of being an interactive medium, it can achieve much more that traditional art forms cannot. For example, the immersion that games bring to people playing them goes far beyond just viewing them, but actually experiencing them yourselves. Therefore, I decided to look for these artistic values in the earliest games, played on the arcades. When I was reading the webpage on The Art of Video Games exhibition in 2012 on the Smithsonian Art Museum website, I came across this game that I had heard of but never played before (because it was too old, obviously), Pac-Man. I knew it was a famous game and almost as old as Tetris, and then soon found out it was one of the first video games that “took this leap beyond the arcade, beyond the screen, into the broader popular culture. That changed the way the world looked at video games and understood that there was something bigger happening here than just a place to waste a quarter.” I took a great interest in this and decided to modify it by combining some other game elements into it and upgrading its interface for my midterm assignment.

I plan to leave the core gameplay of Pac-Man alone and focus on other aspects of the game. In the original version of the game, The map is always the same. I want to combine some roguelike game elements into my project by randomly generating the map every time the player resets the game. therefore the player would always get a new experience when in the game. I also want to make it an endless mode, therefore the player can play to update their personal records. I plan to achieve that by using the tunnels on the left and right sides of the map. Once the player finishes eating all the dots in one map, the tunnels turn into portals that teleport them to other maps with different backgrounds and different theme songs.

Functions and Classes

In this game, I think the main classes are Ghosts, Map, Player, Dots, and Game.

For ghosts, the four types of ghosts (Blinky (red), Pinky (pink), Inky (cyan), and Clyde (orange)) inherit the Ghost class. The Ghost class has certain attributes: intposition for identifying its absolute position, also responsible for calculating destination position; realposition for identifying the true position of the ghost and for determining collisions; targetposition for identifying the target the ghost tries to reach; speed for the speed of the ghosts, and state for determining the current state the ghost is in for it to have different actions. The functions the Ghost class includes are checkCollision, updateTarget, stateChange, directionUpdate, etc. In these functions, updateTarget is different for different types of ghosts.

For the map, it will only have a few attributes. One is a 2D array used to represent the map, and one is its state. it will only have one function, that is mapGen, called every time it is initialized.

For Player, it will have attributes: intposition, realposition, direction, and speed. The only functions it will have are movement and checkDot.

For Dots, the only attribute is its position and type, and the only function is to destroy itself after being touched.

Lastly, for the game class, will have attributes such as points, lives, gamestate(pause, ongoing, in between, etc). It will take care of loading each state of the game and contain data necessary to keep the game going.

This is just a brief outlook of what the system would be like. In practice, some parts may be modified.

Uncertain Part of Project

I find the most uncertain part to be the random map generator. I need to come up with a set of rules so that the map is playable and won’t include too many long corridors the player can be cornered, long dead ends, or areas the player cannot access. However, I have made a preliminary generator that follows one set of rules. This set of rules makes sure that the map is different every time and will not have corridors with width 2 in it. However, this set of rules could result in inaccessible areas. Right now I have an idea about another set of rules to be applied after this so that the inaccessible areas and dead ends can be connected to the other corridors so that the map could be more reasonable. I plan to make any tile that should be accessible with only one or fewer connections with other accessible tiles have at least two connections with other accessible tiles. There will be tiles that cannot find two connection points. Then this tile will be discarded into a wall tile. This process will repeat until no action can be done, and a usable map is then created. I haven’t implemented the second rule set yet, but I have a prototype of the first rule set and it yields satisfying results. It generates grids shown below:

Therefore I am confident that this map generator will be achievable, and I only need to tweak some details.

DiDecor- decorate the room and interact with the character

My concept: When I was a child, I used to play games involving the decoration of rooms of different kinds. When Professor Mang showed the example of a midterm coffee shop, I recalled this experience as the coffee shop looked a lot like one of those games. While looking at that project, I really wanted to change the furniture and play around with the different images. At that moment, I realized that I would like to create my own game with room decoration, so the furniture such as sofas, cupboards, tables, etc could be changed as shown in Fig 1. Fig 1. Room decor game example. Source: 1,334 × 750

The interactivity of the game: I will create an array of images of each of the furniture types, which can be changed by clicking the mouse on top of that particular piece of furniture, and the last shown picture will remain in the game until the user clicks one more. Additionally, I would like to create a character, who lives in this room. The user can move the character and make her sit, and talk with another character (this one isn’t movable) when clicking on that character. I liked the idea of the radio, which changes the music by clicking or stopping it. I would like to implement that with the TV or radio as well because the music sets the vibe of the room too. Additionally, if I have time, I would like to add the story at the beginning of the game how the young lady bought the house and wants the interior designer (the user) to create a design of the room and explain the rules of the game such as which buttons to click. 

Complex part of the game and minimization of risk: First, one of the challenging parts would be finding the images of pieces of furniture. Secondly, talking about the code, it seems hard for now to set the conversation between the characters. For now, I am thinking about making conditionals, so the user can choose what to answer from the given options and the reply of another character changes depending on that answer. If this is too complicated, I will make a random conversation. Each time the user clicks on the second character, a random conversation is generated between the characters, so each time, they are talking about different things but without the interaction of the user. 

 

Concept

For my midterm project, I have decided to implement a puzzle for the user to solve, “Black and White Hat Puzzle”. The puzzle typically goes something like this:

There are 5 hats: 3 black and 2 white. Three characters can see each other’s hats but not their own. They must figure out the color of their own hat based on what they see on the others’ heads, using logic and deduction.

Design

I will implement this puzzle as the user to be one of the three characters wearing a hat, and can see the other two people’s hat. The user needs to correctly guess the hat color he/she is wearing. The user can make only 2 guesses.

If the user can’t make the right guess, the solution will be presented to the user.

A simple wireframe:

Challenging Part

• Drawing the each elements and characters on the canvas
• Assigning the hats randomly to the characters and displaying it
• Building the algorithm behind explaining how the user should solve the puzzle correctly.

Risk Prevention

• I will not draw everything on the canvas, but I will draw them on my ipad and insert them as images to the canvas.
• Assign each of the hats values (and an image of white or black hat) and randomly choose one of them and display it on the screen (on each characters’ head). But the hat of the user must be hidden until user makes the correct guess or two guesses.
• Depending on the other two characters’ hat, the guessing logic must be one of the following:
  1. If a character sees that both of the other characters are wearing white hats: In this case, the character deduces that their own hat must be black. This is because if they were also wearing a white hat, it would mean that there are two white hats in total, which contradicts the given hat distribution.
  2. If a character sees one white hat and one black hat on the other two characters: In this scenario, the character cannot deduce the color of their own hat based solely on this information. They don’t have enough information to determine their own hat color. Their guess would be uncertain.
  3. If a character sees that both of the other characters are wearing black hats: In this situation, the character can deduce that their own hat must be white. This deduction is made based on the knowledge that there are only two white hats and three black hats in total. If they were also wearing a black hat, it would result in three black hats, which contradicts the given hat distribution.

The author outlines several ways computer vision can be enhanced to work on identifying human beings which include but are not limited to using retro reflective marking material. This raises multiple questions in my head about the efficiency of the programs if it only works under certain conditions. I know that this is considered a huge improvement when compared to earlier solutions however I find it interesting that the author is overlooking the main problem which is that having people wear retro reflective film to be detected by the program makes the program non reusable which in turn makes it inefficient. 

The author also mentions how the program would only be able to detect objects that can be described using code or by relating it to other objects the computer understands which further makes me question whether it is reliable to use in all cases. 

The reading made me wonder how much this technology has advanced since this article has been written considering the amount of facial recognition, virtual reality and augmented reality softwares that have emerged and are being used more frequently and more accurately.

concept

My favorite thing about media is its ability to tell stories. Particularly, in interactive media such as video games, the storytelling is incredibly immersive as the viewer has some agency in navigating the way they experience those stories. For this reason, for my midterm project, I wanted to engage in some sort of storytelling. What I envision is something that is at the intersection of being a game/a visual story. I want to have a character that moves in an explorable region and interacts with different objects in the scene to get bits and pieces of the story.

Ideally, it would look similar to something like below, but it would have more distinct objects and probably only one playable character. I also want the setting and the story I’m trying to convey to be somewhat personalized to me and some of my own experiences, though I’m not exactly sure what yet.

design

For the design, I think I am going to heavily utilize images for more rich scene-setting/worldbuilding. I would like the interactive objects, however, to be made in p5.js, so it’s easier to allow the player to interact with.  I also would make visuals to convey the story, that would need to be drawn.

challenging

My main worry in this project is that it relies very heavy on visuals. I’m not exactly the best artist, so I’d need to find specific sprites for the story that I want to tell and background images. In addition, because I want some element of story telling through visuals similar to graphic novels, I’ll have to think about how to get those as well.

In addition, I’m worried about how to handle all these different aspects, especially transitions between different states (as this was something I struggled with in our last assignment as well).

how I plan to handle it 

I think some part of it would be through finding assets online, and other would be finding creative ways to develop those as well, such as perhaps drawing them myself to make the project more personal.

For the more complicated/technical challenges, I think the best solution is to make the story as succinct as possible and start small. I’ll implement the small functionality first – such as making a few objects interactive – and only having one animating object. If I figure out how to do that, the rest should be easy hopefully.

The interactive artworks seemed quite interesting but what caught my attention more were the elementary computer vision techniques. I’m always curious to know what goes on behind the scenes so understanding how motion and presence are detected was fascinating. Just as I was thinking about real time issues these could pose with reference to the physical environment, the next paragraph highlighted these and the possible optimization solutions like dim light, thermal imaging and more, I wonder how plausible and multifaceted these are considering the diversity of environments users might be operating from.

However, amidst the technical intricacies of this reading, there was a line that struck me.’In designing systems to “see for us,” we must not only become freshly awakened to the many things about the world which make it visually intelligible to us, but also develop a keen intuition about their ease of computability.‘ This encapsulated the essence of Computer Vision to me and also added some meaning to it beyond code. This whole concept of being awakened to our surroundings, takes me back to something that I try to practice – being mindful about simple things around me. Combining this with code seems like the perfect combination to work on.

As I find this field of Computer Vision quite interesting and fun, I definitely want to learn more and explore its applications in further projects. I would also read up more in-depth about this specifically, the algorithms, the issues that may come up and possible ways to solve them.

This week’s reading particularly piqued my curiosity, as I have often heard/read about computer vision, but never in the context of Interactive Art. The reading adequately introduces the difficulties within the field and brings up notable artworks that utilize technology like this. It was interesting to see the different ways computer vision could be used.

The pieces that caught my attention were the ones about surveillance. My favorite piece was Sorting Daemon. The composition of the given snapshot generated by the project reminds me of many avant-garde artworks, which is not only aesthetically pleasing – but also eerie, considering that the artwork is being generated from real people’s faces, regular unaware passers-by. I can not imagine how odd it must be to see a machine reconstruct your physical self, using your likeness to construct silhouettes that only appear vaguely human. The existence of this artwork in of itself feels like a paradox – critiquing surveillance while enabling it as well.

Another piece that causes similar conflicting feelings is the morbid Suicide Box. I think the art piece, while exploring themes of surveillance, presents ethical questions at the cross-section of technology, art, and interactive media. Does the piece’s intention – measuring the ‘despondency index’ – justify the technology used?

Both these pieces inspire a conversation about computer vision in interactive media art – not only raising the question of how far we should go, but also if we should.