In the article “Computer vision for artists and designers” Gloan Levin proposes an important idea – the field of human-computer interaction has expanded, including now not only researchers and engineers, but also people from creative industries. As such, the scope of projects and inventions in this area grows as well, especially considering the increasing accessibility of software and hardware that allows experimentation for interactive media artists and even undergraduates.
Computer vision is essential for the machines to interpret the visuals, just as human vision is crucial for us to see what is going on around. However, there is a fundamental difference between these two – humans analyse the picture subconsciously, while computers depend on algorithms. Both are dependent on assumptions, but while human vision can interpret any new unexpected object by relying on their previous knowledge, a computer will likely break down. In order to help the computer to see what we want them to see, certain techniques have been developed to facilitate the machine’s analysis of the visuals – frame differencing, background subtraction, brightness thresholding, object tracking. All these methods allow to set up a path which the computer will follow when it analyses what it “sees”, creating specific conditions for the machine to make its output more reliable.
In addition to that, Levin touches upon the idea of surveillance in interactive art. Thinking about rapid technological development in the last decade, I am always anxious about the emerging tools for surveillance, especially facial recognition systems. Although I understand the necessity behind these innovations, the issue of privacy remains. As I was reading the article, I remembered a video that explained how the AI human recognition algorithm can be fooled – a special pattern for clothing was developed, which confuses the AI when it analyses the image, making it lose the human face. I think this is an interesting area for investigation in interactive media arts – how can patterns disturb or help the computer vision? And what are the boundaries of intervening in the surveillance process?

Concept

What are the major facial features? Forehead, eye, nose, mouth, lips, chin, and skin. We interact with them everyday by looking into the mirror. One facial feature that is essential but always ignored visually is, EARS. This project aims to decontextualize one’s relationship with their body parts by putting it in an unfamiliar context

The game will allow the users to piece jigsaw puzzle together. They will be able to click on the 4 to 10 pieces of segmented images on the canva to rotate it. The number of pieces are randomly generated. When all pieces is rotate to a certain position that match the original images,  the users will score and enters the next page. If they can’t finish, there will be a refresh button

Images

Ideally I’d take the ear pictures, but I also found something on pinterest.

I also found this website that can pixelate the picture to make the ears look more odd.

Implementation

There are two directions that I could design the game. For option 1, users can only rotate the puzzles but not moving them. For option 2, users have the freedom to move around the puzzle and rotate them. We will see which one works better,

Option 1: rotate

Option 2: rotate and move

Below are some functions that I think I need.

Preload

• load image
• Load sound

Rotation

• When mousepressed, rotate

Win condition

• Check position
• Check correct angle
• Display text

Possible implementation & challenges:

User can upload their own pictures of the ears and it will be pixelated and randomly show up in one of the ears. 

One way computer vision differs from human vision is in which computer vision uses sensors, which is almost similar to the human eyes, except their ‘vision’ is not an image but more like bits of information. Though both ‘visions’ need to be understood, the process for a computer would be longer and more complicated in which information has to be internalized and run through a series of program.

To help the computer see we must make the object of attention to be easily visible by either having it be a very bright light, so the computer can track the brightest pixel, or have it in an environment that contrasts the desired object to be tracked/seen (like against a green screen)

As much as it is interesting and cool, the idea of computer vision in relation to surveillance will always come with privacy concerns, especially as seen with the Suicide Box project by the Bureau of Inverse Technology. I think there should be some type of consent from both the artist and those that interact with the art. However with art like the Suicide Box, there is no consent and it all just seems like an invasion of privacy.

Concept

Keychains. Trinkets. And other knick-knacks. The Holy Trinity of my obsessions. I love collecting them. The keychain of my house key always jingles wherever I go because of how many keychains I’ve attached to it. I want to encapsulate this experience into a game in which the user will personalize a bag of their choice with keychains, trinkets and other decorative items of their choice!

Design

• Using pictures of the different keychains and storing them in an array that will randomize when the user clicks the randomize button but it will stay on screen/user can move the object onto their bag if they wish to take the object.
• Implement sound effects of the jingle of keychains whenever the user clicks on a keychain.
• User can also randomize the bag until they choose a bag of their choice.
• use OOP for ribbons so that the user can put however many ribbons as they like on their bag.

Challenges

• Figuring out the logic in how to click-move-click-place the keychains based on mouse input could be tricky. This would require a function.
• Using shapes to illustrate the illustrations and design of the game. I’m using pictures for the more detailed keychains, but I would like to try designing my own keychains using the shapes on p5.

Risk prevention

• Use different files for different parts of the code so that it is more organized and I can easily refer to a specific part of the code without having to scroll through long blocks of code.
• Have a ‘miscellaneous’ file where I would do my testing at.
• Create copies of files so I never lose my previous work after I’ve edited some parts of the code.

Concept:

Coming up with this midterm progress took a long time of just sitting in my room and doing random things in my dorm to come up with a concept. I couldn’t choose whether I wanted to create an art piece, a game or something else. So like I always do, I decided to listen to some music, started looking at my clothes in my closet, and playing some random games on my computer.

Then that became my concept. A virtual interactive experience of my dorm. My concept takes inspiration from the class example of the the coffee shop experience, with my own personal twist.

I drew some sketches and also used DALL-E to create some images to show the experience I want to portray.

We enter the experience by entering my dorm door(will draw better or use AI for final).

 

 

 

 

 

Upon entering the dorm, we are greeted to a closet, speakers and a computer.

I generated the below two images using DALL-E with the prompt: “create an image of a dorm room view from the door with a large window on the very left. To the right of the window is a bed. To the right of the bed is a table with a speaker, desktop computer, monitor, keyboard, and mouse. to the right of that is a closet.”

When clicking on the computer, I want there to be 2 tabs available, one for the a snake game, and one for a beat sequencer/drum machine.

Below is a rough sketch of what the beat sequencer or drum machine would look like. I intend for it to create a drum sequence, based on the 4 bars that continuously loop.

 

 

 

 

 

On the second tab would be a recreation of the snake game.
I’m also considering an option to click into the closet, where you can just scroll through different clothing items.

Currently this is what the main page and the dorm room looks like.

Challenges: 

• Producing the beat sequencer
  • I have been looking into the Tone.js library in order to help with making the beat sequencer.
  • Incorporating the selection method of items in the room(closet or computer) and also the switching of tabs on the computer for either the beat sequencer or the snake game.

 

Golan Levins work makes me rethink the role of technology in the creative process. He puts much emphasis on the accessibility of computer vision tools which are often seen as complex to inexperienced programmers. His idea personally excites me as a student of computer science and art. It makes me consider how I can use my programming to enhance my art projects especially in interactive media. I often focus on technical aspects of my work but Levins work has taught me that technology should also work to deepen emotional impact of art works.

I personally resonated and liked his exploration of Myron Krueger’s Videoplace as it was a great example of how one can merge physical presence with digital interactivity. Krueger’s belief in using the whole human body in interaction counters the notion of passive use of technology. I hope to use this idea of full-body engagement with digital systems in my future work.  It makes me question the boundary I set when making interactive systems – am I too focused on screens and isolated user inputs? His work makes me think beyond the basic interactions and imagine more immersive experiences in digital art.

I was  triggered by the social implications of surveillance in artworks like the Suicide Box. The controversy around the suicide box made think of my responsibility as an artist to become mindful of the impact my work has on other people .

Concept

I honestly felt lost about what kind of project I want to create for my Midterm. I did consider card games, artworks, but finally I settled on a concept of musical game. I took an inspiration from a game that I used to play non-stop as a kid, which is called Piano Tiles.

When I played this game, it made me feel like I was creating music myself, while I was just pressing the rectangles and it initiated sounds of a song. So, I transferred this idea to my own project. I decided that I want to change the interface and introduce piano keys, so it would almost seem like a piano tutorial (like in the photo below).

I imagined that I will have only seven piano keys on the screen (hence just an octave).  And there will be bubbles that will fall down on specific piano keys, depending on the part of the song. The user will be able to press on those bubbles by pressing specific keys on keyboard that will correspond to piano keys.

I think I can assign Q W E R T Y keys on the keyboard to the piano ones, so it would feel more intuitive to the user. So, basically, when the user presses one of those keys, he will be able to pop the bubble. And importantly, user have to blow correct bubble to get a score and follow the song.

Design

As a part of the draft version, I designed piano keyboard using loops. Also, I added OOP for the bubbles, creating class, so it would randomly fall downwards. I sort of worked out on the color palette of my game. I added one song, that is stopped when you press on the screen. Furthermore, I am going to work on the opening page for the project that will introduce the game to the user, also I will set different background, score metric and adjust the movement of the bubbles according to the song.

Frightening aspects

Here is the most interesting part

• I will have to figure out how to connect pressing on keys of the piano to the blow of the circle, so it disappears. Also I have to think about whether the circle will need a specific duration needed to be pressed so it would suit to the song. If that’s the case, I will have to change circular shape either to the ellipse of rectangle with rounded edges.
• But the hardest part is to understand how to tie the music (beat and rhythm of the song)  to the falling of the circles, so basically they should follow the order of beats in the song and be adjusted to how strong the beat is or how long it is, which I don’t particularly understand how to do. Moreover, the circles should fall directly at the place beyond a particular key, which lead to more specification in the code.
• I will also need to work on interactivity of the piano key board, so it would be more intuitive to the audience that the keys should be and can be pressed in order to follow the song.
• I need to figure out as well how to calculate the score of the user if the duration of the pressing will be counted to the score. In that case it will lead to more complicated code.
• Finally, entering and exiting from the game should be planned and coded as well, so it would be accessible to the user.

Risk prevention

• To prevent risk, I looked at the works of other students, particularly with the purpose of identifying how they worked with the “score” metrics, how it appeared on the code and how the score was counted.  This helped me to get more clear on my project’s score calculation
• Moreover, I decided to watch series of lesson on sound in p5.js to figure out how to tie rhythm of the song to how the bubbles will fall down. Here is the link.
• Also, to mitigate the risk, I am planning to take an office hour once I am finished with implementing all the elements.

I found Golan Levin’s article on computer vision to be extremely informative and useful for my future artworks, as it is highly illustrative and detailed in explaining what computer vision actually is. I particularly appreciated how the examples Levin provides demonstrate the vast array of forms in which computer vision is used in interactive media arts: from the sociopolitical Sorting Daemon by David Rokeby to the metaphorical Standards and Double Standards by Rafael Lozano-Hemmer. I believe, as the article suggests, that there is still significant room for improvement in computer vision. It undoubtedly cannot be compared to human vision, as it requires meaning to be assigned to it (a thing human eye has no need to do) through the careful design of algorithms that analyze individual pixels and patterns using techniques like differencing, background subtraction, and brightness thresholding for detection tasks.

Beyond the obvious potential for growth, one concerning issue with computer vision is its possible use as a tool for surveillance, and thus as a powerful method of control. The ethical concerns here are significant: computer vision in art has the potential not only to create but also to exploit those who interact with it without their consent (as in Suicide Box by the Bureau of Inverse Technology).

Therefore, while Golan Levin successfully demystifies the concept of computer vision, it remains unclear how we, as artists, should address the ethics of its use in relation to surveillance. How do I make an interactive artwork that is engaging and immersive without being intrusive and exploitative? That is one of the primary questions an artist should ask when working with computer vision.

 

Concept

For my midterm project, I initially considered creating an interactive experience centered around pet care in some way. While I wanted to explore the emotional connections and responsibilities that come with caring for a living creature, I thought this idea seemed kind of predictable. To add a touch of humor and absurdity, I decided to create an interactive experience where the user will cares for a pet rock (named Rocky) by feeding, playing and interacting with him to maintain his mood over a short period of time (1-2 minutes). The concept explores how minimalistic interactions, like feeding or playing, affect a virtual pet’s state of well-being.

While I still haven’t fully decided on the specifics, the user will have a few options to care for Rocky (feeding and playing for instance). Each interaction will influence the rock’s mood, with the goal of keeping him as happy as possible. For example, feeding might involve choosing from different food types, each of which impacts his mood differently while playing might involve a mini game where Rocky must avoid obstacles.

Code Design and Interactivity

In terms of coding, I’ve broken the project into several key components (I will probably be adding more later as different functionalities arise):

1. Rocky : A class that handles the rock’s mood and his appearance on screen.
2. Mood Meter: This reflects Rocky’s mood visually, giving feedback to the player.
3. Buttons for Actions: Three buttons allow the user to trigger different actions (such as feeding and other care actions).
4. Playing the Mini Game: This involves collision detection where Rocky avoids obstacles.

The game will end after the specified number of minutes, and Rocky’s final mood will be displayed.

What I’m Uncertain Of

The most challenging and uncertain part of this project is implementing the mini-game where Rocky plays by avoiding obstacles. Collision detection and ensuring the game flows smoothly without lag were concerns. To minimize these risks, I will start by prototyping the mini-game early. I will implement a basic version where Rocky moves up and down, avoiding obstacles that scroll from right to left. This will help me better understand how I can handle object collisions and movement dynamics. I also need to fine-tune ideas regarding the options available to the user to take care of Rocky.

Next Steps

Moving forward, I will continue implementing the basic features of the game, refining the interaction design, adding sound effects for feedback, and enhancing the overall user experience. I also plan to improve Rocky’s animations and ensure the game has a restart function. Overall, I’m pretty excited to see how this project turns out :))

Computer Vision is a set of algorithms that are developed to be able to interpret information that are derived from a digital image or video. Human vision is able to recognize items based on past interactions with such items and its traits. Computer vision defers from human vision in that it doesn’t see things as a whole, but rather a collection of pixels, it is due to this that algorithms have to be developed to help.

For example, using pixel and color and brightness, it is able to detect motion. Also unlike human vision, computer vision is not a thing that can be widely used, it is an umbrella term for different algorithms that possesses the functions of tracking, or understanding information from images or videos.

Computer vision’s ability to track and focus on objects allows interactivity between human and art pieces. It allows for artists to create pieces that are able to have a layer of interactivity that doesn’t even need the user to be touching anything physical. As an example, in a project for a different class, using simple computer vision code, we were able to produce a program that detects hand gestures, which then translates to Morse Code.