Category: F2023 – Shiloh

Having conducted research in computer vision, and considering it as one of my areas of specialization, I believe this reading wasn’t specifically targeted towards me. Nonetheless, it was a fun reading and exploratory review of how computer vision can be used for the purpose of interactive medium. But in another way, it has always seemed to me as if those who are closest to a work often fail to recognize it beyond the framework they use it for. In this way, I feel like it may be that the exploration of computer vision and AI in interactive media which is explored in this reading (and that we will explore further in class) may perhaps be the most enlightening for me.

For this reason however, in this reading response, I would like to go a bit beyond the reading. One of the first things that surprised me here was the date of the publication of this article. From what I know, most modern computer vision techniques evolved in the 2010s – thus it was surprising for me to see such creative use of relatively primitive technology going back to the previous century. This also shows that the far more advanced technology available now can probably help us to even more boundary-crossing media pieces.

At this point, I also want to go on an aside about the usage of AI in art. The topic of the ethics of generative art has been in contention over the past year (and rightly so). But if we can somehow have such technology ethically created -then this adds several new dimensions to what art can do and show. Let’s take for example – a technology like ChatGPT. Even though this class rightly discourages its use for generating code, there is still something to be said of its power in being a tool to learn, and how it makes computer art more accessible even to those who have less of a background in coding.

 

Sketch link: https://editor.p5js.org/ff2185/sketches/J9qU9jniU

After tinkering around with different ideas, I decided on the concept of my midterm project: The Interactive Bar.

Here, I will feature different parts of a bar that will allow the user to have 3 actions that intend to simulate a bar experience:
– Make your own drink.
– Play the piano.
– Sit down and enjoy a performance.

For this midterm progress, I have managed to do the front view of the bar and the piano, which was one of the most intimidating experiences.
The inspiration for the bar front was taken from “Moe’s Bar”, an iconic spot in the popular TV show “The Simpsons”

In addition to this, the piano is made using sounds generated directly from the oscillator function in P5js, which creates a signal that oscillates between -1.0 and 1.0. Using this, we can replicate the piano notes, at least the most basic ones.

for (var j = 0; j < 17; j++) {
    envo.push(new p5.Env()); //p5.Envelope,  controls the output volume of oscillator.
    envo[j].setADSR(0.01, 0.05, 1, 0.1);
    envo[j].setRange(1, 0);
    osc.push(new p5.Oscillator());
    osc[j].amp(envo[j]);
  }

  for (var i = 0; i < 10; i++) {
    rSide.push(new rSideKey(i, space, rKee[i]));
    black.push(new BlackKey(i + 0.667, space, blKee[i]));
    mid.push(new MidKey(i, space, midKee[i]));
    lSide.push(new lSideKey(i, space, lKee[i]));
  }

Potential challenges and next steps:

Starting from here, the most challenging parts of the project will be designing the interior of the bar. In order to make this functional, we have to allow the user to click in a certain object to start the experience. In addition to this, the project needs to include navigation options. By this I mean that every new screen should allow the user also to go back.
For this, I included a state management system to organize every screen and make navigation easier.

Reading afterthoughts:

In “Computer Vision for Artists and Designers”, there are multiple interesting concepts and projects presented that piqued my interest heavily.

For starters, one term that caught my attention instantly was “computer vision techniques”. I did not expect the text to dive into this topic because my expectations when I dived into this was that the concept mentioned would be more related to the uses and applications of computer vision rather than proper explanation of how it works. As a Computer Science major, I was immediately shocked to discover that “there is no widely agreed-upon standard for representing the content of video”. Having worked with video myself, it never crossed my mind that there are multiple standards for handling the information of videos, which for many cases might be as bad as having no standard at all. Why? The problem relies on the level of understanding computers have. They have no knowledge on the content itself, but rather, they only know how to show it. In better ways, blurred, black and white, or whatever way you want, but they are not able to know if a person or a dog is there.

Instead of diving into AI and model training, I wanted to mention how interesting it is that older projects were able to work with computer vision even when there was no concept of artificial intelligence. It is not needed to teach the computer how to understand it, but we can start working by simply telling it how to recognize it. The simplest idea is detecting motion. By doing something as trivial as checking whether the frames changed, we can properly detect motion and movement. Even better, we can check how the pixels changed: if there was pixel A in position X, and that same pixel and neighbor pixels are found in position Y, then we can confidently guess that what pixel A and its neighbors portrayed moved from X to Y.

Overall, this reading was very enjoyable from the point of view of someone who loves coding and working with computer vision. It was interesting to see concepts that I could recognize, but at the same time exciting because I could learn new information about these exact topics.

In his article Computer Vision for Artists and Designers, Golan Levin writes about the progression of computer vision and how it has played a crucial role in shaping what we perceive to be interactive tech, art or not. It is interesting to acknowledge that prior to people experimenting with using computer vision for artistic endeavors, the “application development for computer vision technologies, perhaps constrained by conventional structures for research funding, has generally been limited to military and law-enforcement purposes” (Levin). Nevertheless, in our fast-paced and exponentially growing society, it is bizarre to note that with each decade, our computer vision capabilities expand vastly.

In his article, Levin demonstrates multiple examples of where computer vision met artistic and interactive ideas, all ranging in creation times spanning the past few decades. Levin also focused on the different techniques that are used to allow the computing of visual files, mainly by pixel analysis, and this led me to reflect on how, throughout my childhood, we took these technologies for granted. When I was 10, like any other young aspiring boy who liked to play video games, I dreamt of creating the perfect set-up to record myself playing my favorite games. Green screens were extremely popular during that time amongst various creators, and they allowed them to capture and project only the subject of the video onto a different layer. This effect was ultimately used to achieve a better immersive experience for the viewers; however, it is only now that I realize how these applications function and what algorithms and processes are involved to create the seamless effect of being able to change your background. And with each month, we see more implementations of these techniques; for instance, Zoom allows people to change their backgrounds, even without a proper green screen.

In conclusion, I believe that it is a fascinating topic for many to explore, and understanding the complexities behind all the computer vision algorithms is substantially brought into a simpler context in Levin’s article.

Midterm – Pride Dragon Generator

Inspired by the simple yet compelling cartoons of @dinosaurcouch, and the fun and customizability of avatar designing websites such as Picrew, I set out to make a “Pride Dragon Generator” in which users can select a LGBTQ+ identity and receive a dragon with spike colors of the corresponding pride flag. I wanted to incorporate an educational element as well, and what came to mind was using the generator to teach people LGBTQ+ terms in Chinese. When the user hovers their mouse over each of the buttons, an audio will play pronouncing the Chinese term. When they click on a button, they will get a picture of a dragon with, for example, lesbian flag spike colors. They can then save this image to their device.

Dinosaurcouch comic, featuring dinosaurs with lesbian and bi flag color spikes

One of the many customizable avatars in Picrew image maker

Most Frightening Part & Risk Reduction

The most frightening parts of this midterm are 1) the underlying logic of the “hover over button” interactions how users will go forward and go back to the homepage, 2) the sound playing and image saving functionalities, and 3) the complexities of drawing a dragon.

To address risk #1, I first tried to make interactive buttons on my own, and then went to IM Lab hours. With help from Coding Train and Zion, the IM lab assistant, I now have the basic logic of my program, and buttons that enlarge when you hover over them. The next steps are adding the sound and customization options.

To address risk #3, I went to this website to try to better understand bezierVertex, and played around with it a bit to get the hang of which numbers control which aspects of the shape.

Concept Design

As we were looking at previous Midterm projects in class, one of them reminded me of a type of game I used to highly enjoy. The project that reminded me of this was the Coffee shop experience. A few years back, I really enjoyed playing mobile 2D puzzle / escape room games, which had a very similar structure to how the Coffee shop experience functioned. You would be able to zoom into certain parts of the walls, where the objects were placed, and solve interactive puzzles, which would eventually lead to you figuring out some final code to opening the door and passing the game. Thus, I decided that I would attempt and create a virtual 2D escape room game of my own. I am still debating on whether I should create the images myself or find references online. I began by sketching out two versions of how I would want the game to function.

 

 

 

 

 

 

 

I am still not fully decided on the overall theme of the experience/puzzle game; however, I will shortly begin sketching possible ideas for the visuals and take it from there.

Code Design

In order to piece all of the walls and “zoom-in puzzles” together, I am sure that I will need to come up with some sort of layers and switch them according to some indicators.

As of now, I think the best approach for this would be to create a class that would help differentiate between the different scenes. Additionally, I need to consider where I will include each of these elements:

1. At least one shape – Perhaps the door, maybe create shapes as objects underlying the images for them to detect if they are being selected
2. At least one image – Images for puzzles, keys, characters, background, etc.
3. At least one sound – Some theme song playing in the background.
4. At least one on-screen text – One of the puzzles will be a riddle which will include on-screen text.
5. Object-Oriented Programming – I will create a “Layer Manager” class which will help me switch to different layers, such as the overall wall view, zoom into a specific puzzle, etc.

Frightening Concepts

Since I have not yet tried creating objects that are clickable, I believe that this aspect, along with switching views, will be the most challenging for me. In order to overcome this, I will try to research methods of how I can implement this. I have a few ideas about how I can create clickable objects, and I will create a tester p5 sketch, where I will try to implement all of these concepts, which are complex to me. After I am able to make sure that they work well, I can then confidently add them to my midterm project.

Concept and Design

For the midterm project, I’d like to design a Peking Opera experience. In this experience, the user will be able to have a look at the traditional setting of a Peking Opera theater, listen to a clip of a famous Peking Opera, and interact with some of the objects within the scene. This idea comes from the cafe experience the professor showed us in class, and I decided to make some similar experience which is practical but also related to my own culture.

The general style is cartoonish and the general interaction will be: 1. On the start page, the user will click somewhere on the canvas and will enter the theater; 2. within the theater, the user will be able to click on several objects and the object clicked on will zoom in and the user can have a closer look at it.

The most frightening part

I think the most frightening part for this project is sketching the entire setting. I was worried how I could build out the desired setting for a traditional Peking Opera theater and characters. And technically, I was concerned of using transformation function of p5js.

To tackle the setting of the theater, I first looked up some pictures online to decide the color and design of a Peking Opera theater. And I found something like this:

In this picture, I identified some theme colors, including Chinese red, yellow, and brown, and some patterns such as Chinese dragon and phoenix. Therefore, I decided to make a simplified version of this using these colors and patterns, with a character downloaded online that has similar appearances.

For the technical difficulty, I look at some online tutorials on how transformation, specifically scale(), works and started trying with some simple images and shapes to understand this function. With the scale(), I am able to make the character turn around when she’s moving backward.

Next steps

The next steps will be adding the start page, instructions for the suer, and the function of returning to the previou scene as well as refining the entire setting.

Concept

Initially, I had planned on making a project with gravity manipulation as a core mechanic. However, I did not particularly like the ideas that I came up with. One particular idea was to create a voxel-based game where players could create objects that would fall to the ground. Upon impact, the land would be destroyed based on the momentum of impact. However, this proved to be difficult. I might attempt this for a future project, but the idea I settled on took its roots in this idea of a voxel-based landscape, where a voxel is a 3-dimensional pixel.

My idea was to implement a landscape constructed with voxels, and the player could play around with. Additionally, I wanted to give players the ability to change the view from 3D to 2D and vice versa. What I have so far is the project below:

I really enjoy pixel art, which is why I wanted my landscape to be pixel-based instead of being a continuous plane. Some of my previous projects have had the same style, so I wanted to stick to something that I knew design-wise.

I particularly like the way I transition from the 2-D view to the 3-D view. The 2-D plane rotating as it grows and morphs into a 3-D landscape gives a sleek look to the experience.

 

Golan Levin’s “Computer Vision for Artists and Designers” addresses a pivotal intersection in the contemporary world: the confluence of technology and art. By targeting novice programmers, particularly those with roots in the arts, Levin emphasizes the transformative potential of computer vision. This isn’t merely about enhancing artistic endeavors with technology; it’s about a fundamental shift in the capabilities and representations of art in the digital realm. One significant aspect of Levin’s work, inferred from the search results, is the “demystification” of computer vision. This underscores two primary challenges: the intrinsic complexity of the discipline and potential reservations artists might feel towards such a technical domain. By striving for accessibility, Levin advocates for a broader, more inclusive landscape where technology is not a barrier but an enabler.

The exploration of new applications of computer vision in the arts suggests a forward-thinking and innovative approach. This showcases not only the diverse ways artists are leveraging technology but also presents technology as an active collaborator in the artistic process. Such insights could serve as an inspiration, illuminating the expansive possibilities when art and technology seamlessly intertwine. While the search results don’t explicitly indicate Levin’s biases, his emphasis on interdisciplinary collaboration suggests a strong belief in the synergy of art and technology, potentially challenging traditional notions of their separateness.

Reflecting on this raises intriguing questions like: How might computer vision redefine traditional artistic mediums? As tech becomes an intrinsic part of art, how does the role of the artist evolve? Are there ethical ramifications, especially concerning surveillance and privacy, when integrating computer vision into art? Which highlight the evolving dynamics of art in the age of rapid technological advancements and the imperative to navigate this landscape thoughtfully.

“The Psychopathology of Everyday Things” from “The Design of Everyday Things” by Don Norman delves into the intricacies of design in everyday objects. Reflecting upon the reading, one can undoubtedly identify with Norman’s perspective, especially when considering our daily interactions with objects around us. For instance, modern technology and gadgets, despite their advances, can often be counterintuitive in their designs. I’ve personally faced challenges with certain digital interfaces, which, instead of simplifying processes, make them more convoluted, aligning with Norman’s argument about poor design leading to user errors.

However, one might argue that as technologies and needs evolve, there’s a learning curve associated with new designs, making it imperative for users to adapt. This suggests that not all design ‘errors’ are solely the fault of the designer. In terms of bias, Norman does seem to lean towards placing the onus on the design rather than the user. While his points are valid, it’s essential to consider that user education and adaptation play roles in how effectively one interacts with a design.

This reading has indeed made me more observant and critical of the designs I encounter daily. Instead of solely blaming myself for misusing an object, I now consider whether its design might be contributing to the confusion. Which leads to the following questions, how can modern designers better incorporate user feedback? To what extent should designs cater to established user habits versus introducing new paradigms?