Category: F2024 – Mang

Reflecting on the reading “Computer Vision for Artists and Designers,” it’s clear that computer vision techniques have found a meaningful place in the world of art. The text highlights a variety of artistic applications, showcasing everything from abstract creations to works addressing sociopolitical themes. Golan Levin does a great job of illustrating this diversity by mentioning numerous artists and their projects that incorporate computer vision, which really underscores the potential of this medium.

When it comes to how computer vision differs from human vision, the distinctions are pretty significant. While our visual system is adaptable and can interpret a wide range of visual stimuli effortlessly, computer vision is limited by its programming and the specific conditions it can handle. Unlike us, computers can’t easily generalize or grasp context, which are essential parts of how we see. Lastly, the potential for tracking and surveillance through computer vision has a mixed impact on interactive art. On one hand, it opens up new avenues for artists to engage audiences and create immersive experiences. On the other hand, it raises important privacy concerns and the risk of misuse in monitoring behavior. In the end, artists need to find a careful balance, using computer vision to explore their creativity while also being aware of ethical concerns.

Concept: For my midterm project, I created an interactive zoo where users can virtually visit different animal enclosures and experience their sounds in a visually engaging way. The idea behind the project is to give users a simple, intuitive interface to explore various animal exhibits within a zoo environment. The user can click on an enclosure to hear the animal and see a more detailed visual of it.

Frightening Aspect: One of the key challenges I faced was managing sound playback. If I don’t handle this properly, sounds could overlap or continue playing when I switch enclosures which would make the experience messy.

Risk Management:

• I added stopSound to stop the sound when I leave an enclosure. This makes sure that as soon as I press ‘R’, the sound for the current enclosure stops, avoiding any overlap with future sounds.
• When I press ‘R’ to return to the zoo home screen, I use a loop to stop all the sounds from the enclosures. This guarantees that no sound continues playing when I leave an enclosure.

Reading through the passage by Golan Levin and others, I was re-introduced into the world of computer vision without being overwhelmed by technical jargon. As someone who used to get in touch with CV only on a library-resorting basic, aka. using OpenCV in Python, rediscovering some rudimentary algorithms behind it from a perspective of technology in arts brought me some really fascinating ideas compared to the old-school path-finding tasks.

The paper acknowledges how CV has traditionally been the realm of expert researchers and applications limited to military and law enforcement (and yes, of course, teenagers’ robotics tournaments).  With advancements in software development tools, the rise of open-source communities, and the affordability of digital video hardware, CV is no longer an exclusive playground. Artists and designers are now empowered to experiment and innovate, integrating vision-based interactions into their creative projects with relative ease.

A particularly insightful section for me is to emphasize the importance of the physical environment in enhancing the performance of CV systems. The authors argue that optimizing lighting conditions, using retroreflective materials, and selecting appropriate camera lenses can significantly improve the accuracy and reliability of vision algorithms. In my own projects that utilized webcam and process on the video pixels, although the ‘algorithm’ is simply resorting to the grayscale conversion, it is already noticeable to see how the physical environment affects the outcome. The collaboration between software and the physical setup underscores a holistic approach to designing interactive media, where both technological and artistic considerations play crucial roles.

Reflecting on the questions posed by the paper, it becomes clear how CV fundamentally differs from human vision. While humans effortlessly interpret context, recognize patterns, and infuse semantic meaning into visual data, CV relies on meticulously crafted algorithms to process pixel information without inherent understanding, creating both opportunities and challenges for us. By all means, the opportunities lie in the possibility of fine-tuning, altering, and utilizing what the computer has ‘seen’ with algorithms (in other words, to ‘think’ after the ‘listening’ and before ‘speaking’) while it is hard to change the optical structure of human vision. On the other hand, this ‘thought-precedes-realization’ approach in harnessing CV could result in hindering the imagination of artists as the fine line between envisioning and then realizing and the more intuitive manner of mixing the creation and designing at the same time can be easily blurred by lines of codes.

Besides, this powerful capability also introduces ethical considerations, especially when used in interactive art. The capacity for tracking and surveillance can enhance the immersive quality of art installations, making them more responsive and engaging. Yet, it also raises concerns about privacy and consent. For web-based instances like the games I’m currently working with, it is easy and necessary to ask for permission from the user, while installations and broader observational systems could skip the users’ first-hand consent. How do we balance the creative potential of CV with the need to respect individual autonomy and privacy? These questions are crucial as artists and technologists continue to push the boundaries of what interactive art can achieve.

 

Norman’s exploration of everyday objects like doors, switches, and scissors turns the mundane into a reflection of hidden complexities and subtle interactions. As I dove into his ideas about affordances and signifiers, I began seeing my surroundings in a different light. The handle of my coffee mug, the curve of my desk lamp, and even the layout of my keyboard buttons weren’t just functional anymore; they became silent storytellers, each with their own design narrative.

As a left-handed person, Norman’s ideas on affordances and signifiers took on an even deeper meaning for me. I’ve often struggled with objects clearly not designed with me in mind can openers, scissors, and the constant smudging when I write. These frustrations were aptly highlighted in Norman’s push for more inclusive, human-centered design. His arguments about making design accessible to everyone, not just the majority, struck me on a personal level.

The concept of discoverability really stood out during the time I spent reading the chapter. I’ve often found myself baffled by unfamiliar appliances, caught between curiosity and confusion. Norman helped explain why some objects feel intuitive and welcoming, while others seem to thrive on leaving us puzzled. I think back to my experiences with the university printers, with poorly labelled buttons, and an digital interface buried in endless menus. Something as simple as scanning a document became a frustrating guessing game, all because there were no clear signifiers to guide me. In contrast, a well-designed printer would have had intuitive, clearly marked buttons or icons, making the process straightforward and stress free. It was a perfect illustration of Norman’s design principles or lack thereof.

Norman’s “paradox of technology” also resonated with me. As our devices become more advanced, they often become harder to understand. It’s ironic how progress can make us feel more disconnected. I think about my grandmother struggling with her new smartphone, a device that’s supposed to make life easier but instead leaves her feeling lost. It’s a clear example of how our pursuit of innovation can create barriers instead of breaking them down.

The emphasis on feedback also struck me as important. In our world of touchscreens and voice commands, clear feedback is more necessary than ever. I find myself missing the click of a button – the small but essential signal that something has actually happened. Without feedback, it’s easy to feel disconnected from the technology I am using.

Norman’s idea of conceptual models made me reflect on my own assumptions. How often do I approach a new device or system with a flawed understanding of how it works? It’s humbling to realize that many of my mental models are incomplete, and that much of what I think I know is just a simplified version of reality.

After finishing the chapter, I felt both inspired and responsible. Norman reveals how design influences every interaction we have, and as someone studying interactive media, I realize I might one day contribute to shaping these interactions. Norman’s call for human-centered design isn’t just about convenience – it’s about honoring the complexity and diversity of the human experience.

In the end, his message reminds us that between people and objects lies a world of potential – and that good design is about more than just solving problems. It’s about understanding what it means to be human.

MIDTERM IDEA

My idea midterm is to create a short game. One of my favorite (unfinished) games is It Takes Two and it involves the cooperation of two players in order to move on to the next level/ chapter of the game. Though my game will likely not be as intricate as the storyline in It Takes Two, I wanted to draw upon my experiences and create a simply two player corporation minigame.

For my project, I want to have a pattern game with one user using the WASD, one using the arrow keys, and communal SPACE key.  On the screen there will be a combination of the WASD, arrow, and space keys which the two players must work together to enter in the right order to defeat some boss. I definitely still need to work on the storyline/ goal of the game, but I wanted to figure out the basic mechanism and gameplay for my project as a starting point for my midterm project.

GREATEST FEARS 

The biggest obstacles I will face in this project is making the project simple, yet complex at the same time. I believe I will have an issue of doing more than I can in the time I am allotted along with adding an unnecessary amount of detail to my project. For example, with the graphics of the game, I want to include frame-by-frame animations and movements of two characters that represents each player. However realistically, animation takes forever, and I do not unfortunately do not have the free time to draw/ create characters for the complex movements I wish.

While I do fear the images/ shapes would not satisfy what I am looking for, I hope to minimize my fears by designing and animating simple character designs that are already available in the p5 library. I feel if I could make a self-portrait of myself, I hope I would be able to small version of that assignment. In addition, I hope to create a lot of test cases and examine online examples to gauge how others were able to create their games.

MY PROGRESS

I feel when I start projects, I always dive in without much of a plan and either end up scraping the idea because I forgot what the objectives were or scraping up ideas as I work. As such, I decided to make a mood/ idea board that laid out almost a step-by-step process for what I want to implement into my midterm project, so I don’t get overwhelmed or forget to include something I wanted. Below I have included my vision for my game project and list out the components I want (as time permits).

In the sketch, you can see there’s two players that are dancing, but the controls are bounded to the player’s control either the arrow keys or the WASD keys. Together, they’ll have to work together and coordinate which creature they are looking to target first and eliminate. Otherwise, if the creatures reach them within a certain distance a health point will be lost. I do feel I haven’t fully decided on the setup of the players, but I think once it is tested, I can then also decide if I need to make any changes.

Vision Board For My Project

In addition, I have also included the external influences that are inspiring my game. Namely again, the co-op game It Takes Two, arcade games like Space Invader, and character designs of cartoons/ Google Doodles.

External Influences/ References

Thinking about my midterm idea last weekend, I realized that I wanted to create something that was related to Christmas. Being far away from home, I often think about my childhood and those cozy winter holidays that I used to spend with my family and friends, so my midterm is a representation of one of my favorite times of the year.

When deciding on the style of my project, at first I wanted to create some sort of game. However, I decided to try something new – different from the typical collision-detection or platform games that I did all the way through my Introduction to Computer Science class. I decided for it to be an interactive game that would incorporate different elements of my memories about Christmas – music, visual elements, and the overall atmosphere of the celebration because it would be both a pleasant experience for me and something more unusual. As I found out later during the week, when the professor was showing us the previous midterms, my idea about the interactive game was not so unique. The Cafe game is a great example of such, and in some way, I got some additional inspiration by looking at it, but, of course, I will do my best to make my game even better.

Designing the Idea

As I did not have much time this week, my goal was to simply outline the idea and start working on the basic fundamentals of my game. First of all, the opening screen will be full of Christmas celebration elements. One of the first concerns that I faced was to find good high-quality textures for my game, because usually, the free game images I see on the internet are not that good-looking, and for the good ones you need to pay. However, after I spent some time looking for a free PNG no-background images website, I came across a couple where I found exactly what I needed for my game. This is one of the good websites I used – CleanPNG. I also decided to add the possibility of entering the fullscreen – when you press ‘f’ inside the p5.js editor. I remember the technique the professor showed us to make the canvas and all the elements automatically adjusted based on the size of the window, so I even tried to implement it using windowResized() function, but I decided that it would add unnecessary complications further in the game in case I needed to hardcode something, so I abandoned the idea and decided to stick to the size of the canvas 1800×900.

The second stage of my game will take place inside the house. Once the user clicks the door, the inner side of the house will appear on the screen. I took some inspiration from the game called ‘Fortnite’ which I used to play in my middle school years. It had a very nice Christmas event called Winterfest, and that is what it looked like:

The idea was that you log in to the game every day and you can open one gift per day. It had some in-game outfits, weapon skins, and other customizable items, but the main reason I still remember this game is the memories of how I used to play it with my friends during the school winter break.

For my midterm project, I want to implement the idea of opening gifts. I think that it will be the biggest challenge – to think of how the animation of opening the gifts will work and what I want to put inside. I really want to make it a good-quality interaction, so I will do my best to find all the necessary resources to learn the code elements I need to implement for that. As of now, I have not chosen any specific solution, but I am sure I will use a lot of OOP in my code, as well as a lot of functions. In fact, I already started creating functions for each of the elements that are shown on the screen because I can potentially add some animations or sounds for them next week.

Conclusion

So far I really like how I am working on developing my idea. I really want to use my imagination and try to create something that I would be proud of. As I have mentioned before, this game is a good representation of my memories, and I will try to make the most out of my time to make it great.

 

I read about Myron Kruger’s “Videoplace” from another article and I think Myron W Krueger’s philosophical approach to make an interaction between Arts and Computer Science is a pioneering idea in the 60’s that made him a first generation virtual reality and augmented reality researcher. His examples of creating art on the hands of his friend and visual aesthetics reminded me of the times when I used to draw just lines and patterns on the “Paint” software of windows computer and sometimes it would come out pretty aesthetically, but he took this simple idea to generate something out of the box. From this paper, as I read more about the VIDEOPLACE, I felt it was more advanced than the technologies of that time and surely created a path for today’s AR and VR tech. I felt this paper works as a stepping stone for students interested in Computer Vision and Artificial Intelligence.

I also felt very curious to know more about Messa di Voce, created by Golan Levin in collaboration with Zachary Lieberman. The concept of what can be done in the field of Human-Computer Interaction expanded more through this project. Previously, image or pattern recognition would come to my mind while talking about HCI, but now I see how our voice, can be an excellent element for interacting with audiovisual installations. I agree on the fact that algorithms play a key role in this field and I wonder do we all need to keep rebuilding new algorithms to explore more opportunities in HCI? The idea of building machine vision techniques “from first principles” resonates with my desire to understand how these systems work under the hood. While many environments provide off-the-shelf solutions, the challenge of implementing vision algorithms from scratch seems like a rewarding process that deepens one’s understanding of the mechanics of vision systems.

After reading this paper, I want to explore how vision-based systems can interact with audio, visuals, and physical devices to create immersive, interactive experiences. Tools like Myron, Rozin’s TrackThemColors, and especially Max/MSP/Jitter with CV.Jit seem like powerful platforms for experimenting with art and performance which I haven’t used till now.

Concept
I decided to implement a clicker/incremental game with a twist. Instead of just clicking buttons, you’ll be hunting down flying shapes, upgrade your abilities and much more. Cursor will have a box (scope/aim) attached to it, and when an enemy shape is inside this box, clicking will damage it. As you defeat enemies, you’ll earn different types of resources to upgrade your skills in a skill tree. The game will also have prestige rounds (levels), letting you to progress over the game with some bonuses. This idea didn’t come out of nowhere – it’s inspired by a bunch of incremental and clicker games I’ve played, like To The Core, Cookie Clicker, Adventure Capitalist, and others. I’m trying to take the parts I liked best from these games and mix them up into something new and fun.

Design
I’ve been sketching out ideas in Figma. So far, I’ve got designs for the skill tree and the main page. I’m still working on the game screen, but I’ve got the enemy shapes and cursor box figured out.

Main page

Skill tree and upgrades

Enemies and shooting box

I decided to keep the design simple and stick to a pixel art style. This choice not only fits the retro vibe of many clicker games but also helps keep the visuals clean and easy to understand. Plus, pixel art is pretty forgiving when you’re not an art pro, which is perfect for me. The simple style should also help with performance, which is a bonus given my worries about how smoothly everything will run.

Challenges
One big worry is how well the game will run. P5.js is great, but I’m not sure if it can handle lots of moving shapes and calculations without slowing down. I might need to cut back on some features or find ways to make the code run faster. Especially when there are many enemies on screen, or when the skill tree gets complex, the game might start to lag.

Another tricky part is balancing the difficulty. I want the game to get harder as you play, but not so hard that it’s frustrating. Finding that sweet spot will take some trial and error. It’s not just about making enemies tougher – I need to balance how fast you earn resources, how much upgrades cost, and how much stronger they make you. If I mess this up, the game could end up boring or impossible.

Designing the skill tree is also proving to be a challenge. I want it to be interesting and give players meaningful choices, but not so complicated that it’s overwhelming. Each skill needs to feel useful, and there should be different strategies players can try. But I also don’t want to end up with a situation where there’s one “best” path that makes all other choices pointless.

Risk Prevention

• To tackle the performance issue, I’m going to start by making a simple version with just a few moving shapes. I’ll test how many I can add before the game starts to lag. This will help me figure out if I need to change my plans or look for ways to optimize the code. I’ve also started looking into ways to optimize p5.js, like using object pooling for enemies instead of creating new ones all the time.
• For the difficulty balance, I’m planning to create a basic difficulty curve and then play-test it a bunch. I’ll probably need to tweak the numbers a lot, but starting with something I can actually play will help me see what works and what doesn’t.
  
• To handle the skill tree challenge, I am planning to implement main skill class. This basic structure will let me experiment with different skills and costs without committing to a full implementation. I’ll gradually expand this as I figure out what works and what doesn’t.
• Also I want to implement progress save functionality, I’m going to start by just saving the bare minimum – maybe just the resources and which skills are unlocked. I will probably use browser’s local storage to store this data, which should be simple to implement.

By tackling these challenges one by one, I’m hoping to reduce the risks and make steady progress on the game. It might not be perfect right away, but having something playable will make it much easier to improve over time.

This article made me think about how computer vision is different from human vision in some important ways. Humans can easily understand the meaning and context of what we see, but computers need very specific instructions to make sense of images. The article talks about how even simple things like detecting motion or finding objects are challenging for computers. We take for granted how our brains can instantly recognize people, objects, and what’s happening in a scene. But for a computer, each of those tasks requires complex algorithms and programming.

I found it interesting how the article emphasized the importance of setting up the physical environment to help computer vision systems work better. Things like using special lighting, high-contrast backgrounds, or reflective markers can make a big difference. This shows that computer vision isn’t just about fancy software – the hardware and physical setup matter a lot too. It made me realize that artists and designers using computer vision need to think carefully about the whole system, not just the code.

The use of computer vision for tracking and surveillance in art projects raises some tricky issues. On one hand, it allows for really cool interactive experiences where the artwork can respond to people’s movements and actions. But it also means constantly watching and recording what people are doing, which could feel invasive. I wonder how artists think about the ethics of using these technologies. The Suicide Box project mentioned in the article is a good example of how computer vision can be used to gather data on sensitive topics in ways that make some people uncomfortable. Overall, I think computer vision creates exciting new possibilities for interactive art, but artists need to be thoughtful about privacy concerns and potential negative impacts of surveillance.

Concept:

The game is a simplified and modified version of the mini game snowy spin in the steam game Pummal Party. The game is an arena game in which each player controls a snowball. To win the game, the players will try to knock other players’ snowballs out of the arena or collect coins. Each game will contain 3 rounds. The player who wins two rounds out of three wins the game. There will be a scoreboard displaying the current score and current coin number of each player. 

User Interaction :

1. The game starts with an instruction screen, waiting for user input to begin.
2. Each player controls their snowball using keyboard inputs:
   • Player 1: WASD keys
   • Player 2: Arrow keys
3. Players navigate the arena, collecting items and attempting to knock opponents out.
4. The game consists of 3 rounds, with the winner being the first to win 2 rounds.
5. After game completion, players can choose to start a new session.

Features:

• A snowball class:

Pressing the keyboard applies a force to the ball in the direction of the keyboard control. The ball will accelerate until it reaches maximum speed. When collision between the balls is detected, each ball will move according to the physics of the elastic collision. The ball can collect the items along the path of its movement. The ball will “fall” if the ball is out of the arena.

• A game class:

That will display the arena, the balls, and the items. The arena will be a perfect circle.

• A item class: 

Items are randomly generated around the arena every (possibly) 10 seconds. The players can collect items by moving their balls to the item. 

• Players win a round by knocking the other player out of the arena or collecting 3 gold coins. Players win a game by winning 2 rounds out of 3.

• Background music and/or sound effects of ball colliding and players collecting items 
• Power ups : 
• Increase / Decrease size of snowball 

  Increase / Decrease speed of snowball

  • Coins

Code Design:

1. Snowball Class:
   • Properties: position, velocity, size, player ID
   • Methods: move(), accelerate(), checkCollision(), collectItem()
2. Game Class:
   • Properties: arena, players, items, current round, scores
   • Methods: startGame(), updateGame(), checkWinCondition(), nextRound()
3. Item Class:
   • Properties: type, position, effect
   • Methods: spawn(), applyEffect()
4. Arena Class:
   • Properties: size, shape
   • Methods: draw(), checkBoundaries()
5. Main game loop:
   • handleInput()
   • updateGameState()
   • checkCollisions()
   • drawScreen()
   • playAudio()

Potenial Obstacles:

1. Collision detection and physics
2. Implementing smooth player controls