The Concept

For my midterm project, I want to make a walking game controlled by sounds the player records. Instead of assigning fixed voice commands, the game will let each player choose a short sound for Up, Down, Left, and Right. A player could use spoken words, clicks, whistles, or other distinct sounds. I like the idea that the controls become personal to the player, and that learning to move is part of the experience.

User Interaction

The player will first enable their microphone and record one sound for each direction. Once all four sounds are saved, they can press “Start listening” and play using the microphone. Each recognized sound moves the character for a chosen number of seconds. The player can adjust that duration with a slider and re-record a sound if it is difficult to recognize.

Movement on the two axes works independently. For example, if the player makes the Up sound and then makes the Left sound while Up is still active, the character moves diagonally. The arrow keys also work, which gives me a way to test movement separately from the microphone.

Code and Design Progress

I started with a p5.js prototype of a walking character. It loads a sprite sheet, separates it into animation frames, and displays the correct row for each direction. The character animates while moving and stays within the canvas.

I then added the microphone setup, recording buttons, sound recognition, and timed movement. The code samples a sound’s frequency profile, divides it into three parts over time, and compares it with the four recorded signatures. It rejects sounds when the match is too uncertain. I also changed the movement code to keep separate horizontal and vertical timers, which allows two directions to remain active together.

Most Frightening Part

The most uncertain part is sound recognition. A person will not make a sound exactly the same way twice, and background noise could cause a false command. Rapid commands are another challenge: the program needs to notice where one sound ends and the next begins.

To reduce this risk, I built the recognition system before adding more game features. The prototype normalizes the frequency profile so a louder or quieter version of a sound can still match. It waits for a short silence to finish recording a command and rejects ambiguous matches. This gives me a working system to test with real sounds, although I still need to find out how reliably it performs in different rooms and with different players.

Next Steps

My next step is to test several sets of four sounds and record which commands are missed or confused. I can use those results to adjust the detection thresholds and the pause between commands. After the controls feel reliable, I want to design a space for the character to navigate so that the audio movement has a clear goal.

The artwork I kept thinking about after reading Levin’s essay was Cheese. The performers had to hold a smile while a computer measured it, and an alarm went off when their smiles dropped below a certain level. I can imagine laughing at that at first, then becoming very aware of my own face. It made me wonder when an artwork that responds to you starts to feel like it is judging you.

That example also made the difference between human and computer vision clearer to me. I might look at someone’s smile and wonder whether they are tired or joking. The computer needs a way to turn what its camera captures into something it can measure. Even finding a person in the image takes work. Frame differencing looks for changes between video frames, so someone standing still could disappear from its view. Background subtraction finds what differs from an empty scene, while brightness thresholding works when a person or object is clearly lighter or darker than its surroundings. I was surprised by how much the artist can help simply by changing the lighting or background.

I like Krueger’s idea that “the entire human body ought to have a role in our interactions with computers.” It opens up more playful possibilities than clicking a mouse. I would be curious to make something that reacts when a visitor turns toward it or moves away, without trying to decide what their expression means. Tracking can make a person feel involved in an artwork, but Cheese reminded me how easily that same attention can feel like surveillance. I think visitors should be able to tell when an artwork is watching them and choose whether to take part.

The Concept
I want to create a rhythm platform game where the user has to hit a button for each beat of the song. If the user fails to do so, they will lose and will have to repeat the level.

The game will be something like this, where players must hit a button and it’ll perform an action, maybe the action is to jump, hit, duck, etc.

The player needs to be listening to the song in order to know what to do. But the game will also have visual cues so the player isn’t entirely reliant on just guessing the beat.

Design
I want the game to have a bit of a retro feel, almost like a musical Mario game. The game will consist of pixel art and will feel almost nostalgic. I want to highlight the style of gaming that popularized platform type games.

The Most Frightening Part
I’m mostly worried about making the key the player presses in line with each beat of the song. It being a platformer minimizes the risk, but how big is the margin of error for the user? Additionally, how can I make it so that the player needs to do an input at specific beats of the song.

Minimizing the Risk
I did my best to work with sound in p5js and tried to make it so that the background color changed during the beat of the song.

song.addCue(2, changeBackground, 'palevioletred');
song.addCue(3, changeBackground, 'mediumvioletred');
song.addCue(4.2, changeBackground, 'pink');
song.addCue(6, changeBackground, 'orchid');
song.addCue(8, changeBackground, 'skyblue');

I used addCue to change the background during certain parts of the song. I probably could have put it into the loop but I first wanted to just highlight parts where I heard a beat and put the changeBackground function there. I also think it’ll be funner if the beats aren’t entirely consistent, as that would mean the levels will be harder.

 

Computer vision is different from human vision because computers do not actually “see” images the way humans do. A computer sees an image as pixels and needs algorithms to figure out what those pixels mean. Humans can understand objects, movement, and situations much more naturally, while computer vision usually needs specific conditions and instructions to recognize something.

There are different techniques we can use to help a computer see or track what we are interested in. Some examples are frame differencing, background subtraction, and brightness thresholding. Frame differencing can detect movement by comparing two frames, while background subtraction can separate people or objects from the background. Brightness thresholding can also help detect objects when there is a strong difference in brightness between them and their surroundings.

Computer vision’s ability to track people also connects to surveillance, which can make interactive art more complicated. I think it can be interesting because artists can use tracking to make an artwork react to people, but it also raises questions about privacy and how much people are being watched without realizing it. One artwork I found interesting was Sorting Daemon by David Rokeby. I like it because there is a lot happening in the composite image. The system watches people outside, detects them, separates their images from the background and organizes parts of them by color and size. I think it is interesting because it turns something that feels like surveillance into an artistic image while also making us think about how technology can automatically analyze and categorize people.

After reading and understanding “Computer Vision for Artists and Designers” I found it particularly interesting how computer vision is able to turn an individual’s physical movements and environment into a method of interactive art through digital media. It was definitely enjoyable to be able to see examples of interactive artworks that were shown in the article as they revealed that a camera does not simply have to be used to record someone or something, instead it can be used as a meaningful tool to make an artwork responsive. The LimboTime project also stood out to me because it showed me that even beginner programmers can create an interactive game using simple computer vision techniques. A key topic that was mentioned about the physical environment was also interesting as I was previously unaware of how elements such as lighting, contrast, and background can influence a computer’s ability to detect a person. 

Computer vision differs from human vision as humans are able to naturally understand what they are seeing in front of them, whereas computers can only see an image as a collection of pixels. Therefore, a computer tends to require several specific instructions and codes to recognize peculiar things in images.Computer vision also relies heavily on the conditions of the surrounding environment, whereas humans are generally able to distinguish an object from its surroundings more easily.

There are particular techniques that have been mentioned in the text which are used to help the computer in tracking and seeing what we are interested in. Some examples that are mentioned include frame differencing which compares two video frames in order to detect movement. Background subtraction which is also similar but compares the current image to an image of the empty background in order to identify the difference. Brightness thresholding is another technique which is mentioned which allows the computer to identify objects based on whether they are lighter or darker than the surrounding threshold. Techniques such as finding the lightest or darkest pixel can also be used to identify specific objects. 

I think computer vision’s tracking and surveillance approach can have both a positive and negative effect on interactive art. When looking at the positives, this approach invites artists to create meaningful experiences where their artworks can respond directly to the viewer’s body, movement, or facial expressions. This therefore invites the audience to make their own interpretations of the artwork through personal experience and involvement. However, one of the disadvantages that particularly stood out to me when reading the text is the concerns regarding privacy and surveillance as this technology can be used to monitor individuals without their knowledge or consent. This therefore reveals that computer vision can be both a useful tool for creating interaction and also invites artists to question how technology can observe and collect information about us.

The Concept:

For my midterm project, I want to create a scooter game inspired by something I have noticed a lot since starting university, which is how chaotic scooters can be around many university campuses. There are always people riding around, almost crashing into each other, going into bushes, or trying to avoid people walking in front of them. I thought this could be turned into a simple and funny game based around trying to survive riding a scooter through campus.

The player will control a student riding a scooter and use the left and right arrow keys to move around different obstacles. The main goal is to survive for as long as possible and get a high score. The player will have three lives, and crashing into things like bushes, students, or other scooters will make them lose a life.

However, I also wanted there to be something unexpected instead of the game only being about avoiding everything. One idea professor Mang suggested was making professors worth bonus points. If a professor appears, instead of avoiding them, the player can purposely crash into them to gain extra points. I think this makes the game a little more funny and gives the player a reason to pay attention to what is coming towards them rather than treating every object in the same way.

Start Screen and Instructions:

The game will begin with a start screen that explains the controls and objective. The player will be told to use the left and right arrow keys to steer the scooter, avoid obstacles, and look out for professors to gain bonus points. The game will only begin once the player presses a key, such as SPACE.

Main Game:

The game will be from a top to down perspective, with the player and scooter towards the bottom of the screen. Different objects will move down the screen towards the player to create the feeling that they are riding forward through campus. These could include students, bushes, other scooters, and professors.

As the player continues, I want the game to gradually become more chaotic, with obstacles appearing more often or moving faster. The score will increase as the player survives, while hitting a professor will give additional points. The player will have three lives, and once all three are gone, the game will end.

Game Over and Restart:

Once the player loses all three lives, they will reach a game over screen showing their final score and possibly other statistics such as how many professors they managed to hit. There will also be an option to play again without having to restart the entire sketch.

Code and Design:

For the code, I am planning to use a class for the different obstacles that appear throughout the game. The objects can have different types, such as a student, professor, bush, or scooter, while sharing things like their position and movement. I also want to use an array to keep track of the different obstacles on the screen.

Visually, I want the game to look like a university campus from above. For now, I will probably use basic shapes while testing the game, and later I want to replace some of them with images and add more details to make the campus environment feel more complete. I also want to add sounds such as a crash sound, scooter sounds, and possibly a different sound when the player gets bonus points.

Most Frightening / Uncertain Part:

The part I am most uncertain about right now is collision detection. Since most of the game depends on the scooter either avoiding or hitting different objects, I need the code to recognize when the player actually touches something. I also need different collisions to have different results. For example, hitting a bush or student should cause the player to lose a life, while hitting a professor should give them points instead.

Another thing I am slightly worried about is having multiple objects moving on the screen at the same time without making the code too complicated, especially since they will have different effects when the player interacts with them.

What I Will Do to Reduce the Risk:

To reduce this risk, I want to start with a very basic prototype before working on the final images and design. I will first create a simple player using a shape that can move left and right using the arrow keys. I will then make one obstacle move down the screen and test whether the program can recognize when the two shapes collide.

Once I get this basic collision working, I can start using the same idea with different types of obstacles and eventually replace the basic shapes with the actual images I want to use. This way, I can make sure the main mechanic of the game works before focusing too much on how everything looks.

// check if player hits obstacle
  if (
    playerX < obstacleX + 50 &&
    playerX + 40 > obstacleX &&
    500 < obstacleY + 50 &&
    570 > obstacleY
  ) {
    textSize(40);
    text("CRASH!", 220, 300);
  }
 }

What I found most interesting from the reading was realizing that computer vision is less about teaching a computer to see like we do, and more about finding ways to make visual information understandable to it. As humans, we can look at a space and recognize a person or object almost immediately, even if the environment changes. A computer does not automatically have that understanding because a video is initially just made up of pixels. This also explains why computer vision can be more dependent on its environment. Changes in lighting, contrast, or the background that might not affect our ability to see something can make it much harder for a computer to detect.

Because of this, I thought it was interesting that improving computer vision does not always mean making the technology or code more complicated. Techniques like frame differencing can detect movement by comparing one frame to another, while background subtraction can identify something that is different from an already known background. Brightness thresholding can separate objects based on how light or dark they are. However, we can also make these techniques work better by changing the physical environment itself. The LimboTime example especially stood out to me because the creators used a large white background to make the players easier to detect. I would normally assume that improving tracking would require better code, so I liked the idea that sometimes designing the environment around what the computer is capable of seeing can be just as important.

At the same time, I think this ability to track people creates an interesting problem when computer vision is used in interactive art. Tracking someone’s body or movements can make an artwork feel much more responsive because the person can become part of the interaction without needing something like a mouse or keyboard. However, there is a very small difference between a system responding to someone and watching them. I found standards and double standards interesting because the belts physically follow people as they move through the space, making that feeling of being watched part of the experience itself. It made me think that the feeling of being watched doesnt necessarily have to be hidden in interactive art. Artists can use the feeling of being tracked with intent, not only to create an interaction but also to make the person more aware of what it feels like to constantly be observed by technology.

Reading Reflection

The Minsky story at the start is funny, but I think it explains why computer vision is so hard to appreciate. Seeing feels effortless to us, so it seems like it should be simple, when really it’s one of the things we understand least about our own minds. Levin points out that text already comes as symbols, while video is just rectangles of pixels with nothing telling the computer what any of them are. What I kept thinking about is that human vision isn’t assumption free either. We fill in gaps, fall for optical illusions, see faces in random objects. The difference is that our assumptions are invisible to us, while a computer’s have to be written down by someone. In LimboTime, the program treats the limboer’s head as the topmost point of the middle blob of black pixels. The computer isn’t seeing a head there. A few workshop participants decided what a head is for one afternoon, and the computer can only ever find what they told it to look for.

Looked at that way, each technique in the reading is a different guess about what matters in a scene. Frame differencing guesses that what matters is whatever moves, which is why it works on the Tour de France and fails in an office waiting room. Background subtraction guesses it’s whatever wasn’t there before. Thresholding guesses it’s whatever is darker or lighter than everything else, and brightest pixel tracking is basically a max search over an array. None of this is hard to code. The harder part, and what Levin spends most of his time on, is making the world fit the guess with backlighting, infrared, polarizing filters, or retroreflective material (the same stuff 3M makes for safety uniforms). His conclusion says a scene’s “quality” is defined by the algorithm analyzing it, which means there’s no such thing as a good image in general, only an image that’s good for one particular piece of code. The idea I found most useful is at the end of section three, where he says to choose features that are easy to detect and also carry meaning. The mouth is an easy dark shape to find, and its roundness happens to track vowel sounds. Pupils reflect infrared and also show where someone is looking. Power of real design skill is embodied by the difference of what’s computable and what’s meaningful.

That same idea is also what makes surveillance work, and Suicide Box shows it most clearly. The camera never knows anyone is jumping. It picks up vertical motion through simple frame differencing, the same technique a beginner would use for a game, and the location of the camera turns that motion into a death. Levin mentions in the abstract that computer vision was mostly funded for military and law enforcement, and then his conclusion lists everything it can now track, including identities, facial expressions, gait and gaze direction. In 2006 that probably read as a list of creative possibilities. Reading it now, it sounds a lot like a description of the systems that watch us in airports and on streets.

Since the technology is identical, I think the line between interactive art and surveillance comes down to whether the person being tracked can see what the system sees and gets something back. Videoplace shows you your own silhouette, so you’re playing with the system instead of being processed by it. Sorting Daemon turns people on the street into color swatches, and most of them probably never find out. Cheese sits somewhere in between. The actresses knew they were being watched, and an alarm pushed them to make their smiles look more sincere to a machine. Every interactive piece with a camera does a softer version of this, getting you to move in ways the camera can read. Usually that’s the fun part, but Cheese shows how quickly responding to someone turns into making them perform. If I were to determine for what counts as a head or a smile, I would let people see what’s being tracked instead of hiding the camera and letting it feel like magic.

Midterm Report

I was thinking on notes of what is culturally important to me and values wise. Being from Pakistan, hospitality is a big part of the culture, and I believe I have embodied it in a nuance way. Whenever, my friends and I go out I want to pay for us. There is an element of looking after people that brings me a lot of joy, and I wanted to embody that in the game.

So, I will be making a game where the faster you approach the card reader you win. It will mainly be pressing a key to get to the card reader faster, while the other 2 players reach the machine.

I have prepared the 4 frames that will be needed for the game, and I have made a pixelated version of the frames. I wanted to have the first 3 frames just like appear and kind of build a storyline, and the fourth frame has the game in it.

Here is a wire-frame of the game:

Then, here are the 4 frames of the storyline. 

Parts I am worried about

When the user starts playing the game, I am concerned about how the hand would elongate. I will have to parse the image and maybe separate the hands from the arms like save it as a separate asset too. Then, as the user plays the hand extends towards the card machine, so i am concerned about the game mechanics too.

I will be doing something like following to move the separate assets:

progress = 0                        // 0 = start, 1 = touching the card machine

on key press:
    progress = progress + 0.05      // each press moves the card a bit closer

every frame:
    progress = progress - 0.002     // slides back if you stop pressing
    handX = startX + (machineX - startX) * progress
    handY = startY + (machineY - startY) * progress
    draw hand at (handX, handY)

    if progress >= 1: you pay

Progress so far

Concept

My project is inspired by the Chinese film A Letter to Grandma. The film deeply touched me because of the way it portrays love, memories, and connection through letters.

Before the age of phones and instant messaging, letters were one of the most important ways for people to stay connected. A letter was not only a way to communicate information, but also a way to express emotions that might be difficult to say in person. This made me think about how we communicate with people today. Although we can send messages instantly, we often do not take the time to carefully write down how we feel. I therefore want to create an interactive experience where people can write, save, and keep their own letters, especially letters expressing their thoughts and feelings towards family and friends.

I also wanted to incorporate some important visual elements from the film. Flowers and birds are meaningful symbols throughout the movie, so I decided to bring these elements into my own work. The flower represents memory, love, and something that continues to grow, while the bird represents the movement of a message from one person to another.

Current progress

At the moment, I have completed the first major part of the project.

The papers are currently hanging and gently moving, creating the feeling that they are suspended in the air.

I am happy with this stage because the basic visual language of the project is already beginning to communicate the idea of letters, memories, and connection.

Next step

My next major goal is to make each piece of paper interactive.When the user clicks on a letter, I want the screen to transition into a writing interface.

The idea for this part actually came from something very simple. Over the weekend, I was bored and found a website where I could just sit down and type. This made me think about how writing itself could become part of the artwork.

The Writing Interface

I also want the writing interface itself to feel artistic rather than looking like a normal text editor.

My idea is to design a visual keyboard on the screen.

When the user presses a key:the corresponding key on the visual keyboard will react，the typed letter will appear on the screen，the text will gradually form into a letter

I also want to introduce a small bird into this interaction.

As the user types, a little bird could fly from the keyboard toward the words on the screen.

The bird would act almost like a messenger, connecting the act of typing with the idea of sending a letter.

This connects back to the original inspiration from the film: a letter is not just text; it is something that carries emotion from one person to another.

Author jz8020Posted on October 5, 2026October 5, 2026Categories F2026 - Mang