Category: S2026 – Mang

Concept & User Interaction

I was very hesitant on an idea from my midterm. I really wanted to incorporate something that was personal to me and create a game that I could imagine my younger siblings being able to play. The idea I ended up with is: a puzzle game! I love solving jigsaw puzzles, so when I thought of doing something that would digitalize that, I was immediately excited about the idea. In addition to that, I decided to give it an Ancient Egypt theme! The last puzzle I solved consisted of landmarks in Egypt (image below), so inspired by that, the images that the user can solve as a puzzle in my game will be Egyptian landmarks (but more cartoon-like).

The first thing I started working on was the design. I feel even though we spend a lot of time on p5.js in class, the design and aesthetics of our projects really matter as it is what users will perceive first. So, I went to Canva and started working on what I want to my game to look like. I also started looking for fonts that would match my theme. However, since I don’t have Canva Pro, I wasn’t able to incorporate the fonts I found online into my designs, but I will definitely be using them in my code. Something that is missing in my design but I definitely plan to add into my final implementation is a reference image for the user to refer to on the game screens. It was a bit difficult to get everything to fit into the dimensions provided by Canva, however, I will try to work more on the layout in the coding stage.

The font I’m planning to use would look something like this if used for the title:
I’m still debating if its a bit illegible or if it fits the theme nicely. For now, I’ll just have it as a possible option, because I feel the regular serif fonts can be a bit boring.

After design, the next question is obviously user interaction and the actual code. The user interaction in my program is pretty obvious, the user selects their puzzle image, their difficulty, and then they solve the puzzle by moving the pieces from the right side (consisting of the jumbled up pieces) to the left side. Initially, when I was thinking about my idea, the picture I had in my head was jigsaw-like shaped pieces, however, I felt that actually implementing that in p5.js might be a bit too troublesome so I settled for simply splitting the image into smaller squares. Then, thinking about how the pieces would go onto the solved board, I thought that it would make sense if the user moves the piece close to where it actually is in the image, the piece would snap into place (paired with appropriate sound effects). I think that is the most satisfying way to gamify this interaction, and I can already think about what the code for that could like. In addition to sound effects, I’m planning to add some background music to each screen, probably relating back to the image selected for the puzzle. Finally, on the ending screen, I will probably match that with a celebratory sound effect, and confetti. For the confetti, I’m imagining Egyptian hieroglyphs raining down, as that would link back to theme I’m going for and it would be a unique idea. However, I still have to think more about the implementation of that.

In terms of code, I haven’t been able to make much progress, however, I’ve started to watch a video Coding Train and Patt Vira videos to understand how to code the mouse interactions I’m thinking of.

Most frightening or uncertain part of my midterm project

The most frightening part of my midterm project so far is how to really implement my ideas into code in an efficient, simple way.  There are two main interactions I’m intimidated by, 1 – Splitting the images into smaller squares, especially maintaining the aspect ratio and quality of the images and 2- Snapping of the pieces into place when solving the puzzle. For the former, I believe The Coding Train has done a video on a similar concept so I hope I can refer to his tutorials to understand the code, and the latter, I believe will become more familiar through trial and error. Another thing I’m concerned about that’s less technical and more about aesthetics, is finding appropriate assets (i.e. images, background music, sound effects) as that is what most non-technical users will notice, rather than, the code. To overcome this, I’ve already started searching for ideas and looking at past projects to see what resources they used to find their assets. Luckily, we also live in the age of AI, so if it comes down to it, I may have to use AI-generated images to find exactly what I’m looking for, however, my preference is that the images are more realistic.

In “Computer Vision for Artists and Designers,” Golan Levin explains that computers do not see like humans. I realized that computer vision is not really vision. It is measurement.

When I look at a room, I see people, emotions, and context. A computer sees pixels. It compares brightness values. It detects motion between frames. It subtracts a stored background. Techniques like frame differencing and thresholding reduce the world to contrast and change. The computer does not understand meaning. It only processes differences in light and color.

This difference forces artists to redesign the environment. If the lighting changes, the system fails. If a person stands still, motion detection fails. So artists add backlights. They use white walls. They create strong contrast. In Videoplace by Myron Krueger, participants stand in front of a bright background so their silhouettes are easy to track. The interaction feels magical. But it depends on careful staging for the machine.

I was also struck by works about surveillance. In Sorting Daemon, David Rokeby reduces people to color patches. The system does not know who they are. It only sorts hues and shapes. This makes me uncomfortable. It shows how easily people can be turned into data.

For me, the most important insight is this: computer vision changes not only how machines see us, but how we design spaces for machines to see.

Concept
My midterm project is a simple car race game called “Mirror Lane Racer.” The player drives a car on a three-lane road and tries to survive by avoiding traffic cars. The main “new element” is Mirror Mode: sometimes a mirror shard pickup appears, and if the player collects it, the controls flip for a few seconds (left becomes right, right becomes left). This is meant to create a quick moment of confusion and force the player to adapt.
My reference of this work is the idea of subway surfers.

The experience starts on an instruction screen and waits for the player to press SPACE (or click) before the game begins. If the player crashes, the game switches to an end screen showing the score and a clear restart option (press R) so a new run can start without refreshing the page.

Code plan
I’m planning to structure the sketch with three game states: Start, Play, and End. In the Play state, the game will repeatedly (1) spawn traffic cars and mirror shards, (2) update their positions as they move down the screen, (3) draw everything, and (4) check for collisions.

To meet the OOP requirement, I’ll use classes like: 1) PlayerCar: stores lane position and has methods to move left/right and display itself. 2) ObstacleCar: spawns in a lane, moves downward, and gets removed when off-screen. 3) PickupShard: spawns sometimes; if collected, it triggers Mirror Mode with a timer.

Traffic cars and pickups will be stored in arrays so multiple objects can exist at once. I’ll also include a `resetGame()` function that clears arrays, resets score/timers, turns off Mirror Mode, and puts the player back in the center lane for restarts.

Sound plan
The game will include at least one sound, and I’ll likely use a few short sounds: 1)An engine loop 2)A crash sound when the player hits traffic  3)A pickup sound when collecting a mirror shard

Because browsers usually require a user gesture before playing audio, I will start audio only after the player presses SPACE/clicks to begin. After that, I can trigger sounds during gameplay events (pickup/crash). I already tested that a keypress can reliably start audio, so I won’t risk the sound requirement failing at the end.

Reading this made me think about how differently computers and humans actually process vision. When I look at a painting I immediately read context, emotion, and narrative without even thinking about it. A computer just sees raw pixel data and well, to me that’s like doing pure formal description with zero interpretation, listing visual elements without ever asking what they mean. I also had no idea how much the physical environment matters. I always assumed it was mostly about the software but things like backlighting, infrared lighting, and high contrast surfaces can make or break whether an algorithm even works. It’s almost like you’re setting up the scene for a machine to read rather than a person. The surveillance aspect makes me both uncomfortable and kind of fascinated honestly. The fact that this technology came out of military and law enforcement doesn’t just go away because it’s in a gallery. I think that’s why I find something like Rokeby’s Sorting Daemon so much more compelling than installations that just use body tracking to make something pretty because it actually wrestles with that baggage. When I know something is tracking me I feel it, and I think that feeling is worth exploring more than something with purely just a cool visual. The work that sits in that discomfort feels a lot more honest to me.

Computer vision and human vision both “look,” but they don’t work the same way. Humans are good at understanding meaning. If you see a person in a messy room, you instantly know what’s important, even with bad lighting or weird angles. Computers usually don’t understand meaning by default. They see the world as numbers (pixels), and they often need clear conditions or extra rules to decide what something “is.” Humans also handle depth, shadows, and changes in lighting very naturally. A camera image is flat, and small changes (a shadow, glare, low light) can confuse the computer a lot more than a person.

To help the computer “see” or track what we care about, we usually make the situation simpler. One technique is controlling the environment: use bright, even lighting and a clean background so the subject stands out. Another is increasing contrast (like wearing a bright color against a dark wall). In code, you can use methods like motion detection (compare frames to find movement), background subtraction (separate the person from the background), thresholding (turn the image into black/white based on brightness), and color tracking (follow a specific color range). You can also add markers on purpose—like a QR-code-style tag, a colored sticker, or a high-contrast shape—so the computer can track that instead of trying to recognize a whole body or face. The general strategy is: don’t make the computer solve a hard problem if you can redesign the setup to make it an easy one.

Computer vision is strongly connected to tracking and surveillance, and that affects interactive art in a few ways. On the positive side, tracking can create powerful experiences: the artwork can “respond” to your body, movement, or presence in a way that feels alive. But the same tracking can also feel invasive, because people know cameras are used to monitor and identify them in real life. That can create discomfort or distrust, especially if the artwork is not clear about what it records. Artists using computer vision should think about consent and transparency: tell the audience what is being tracked, avoid collecting personal data when it isn’t needed, and design interactions that don’t require identifying someone (tracking motion or silhouettes instead of faces). In interactive art, the best use of vision often focuses on participation and expression, not surveillance.

Reading Levin’s “Computer Vision for Artists and Designers” made me realize how much I take my own sight for granted and got me questioning what it means for a computer to “see.” I think I always imagined it as something similar to human vision? just more technical with lots of codes and numbers and letters. This reading though, made it clear that computer vision is not actually vision in the way we experience it. While we humans automatically recognize faces, objects, details, and context without consciously thinking about it, a computer just receives pixel values without understanding the meaning since it only processes numbers and the differences between frames. That really stood out to me because it shows how much interpretation we naturally do as humans without noticing it.

Another important difference is that human vision is flexible while computer vision is mostly dependent on certain conditions, for example if the lighting or background changes a little, the computer vision system can fail completely. The reading emphasized how algorithms like frame differencing or background subtraction rely on stable environments and clear contrasts which made me realize that computer vision is not just about writing code but it’s also about carefully designing the physical environment so that the system can succeed. There are things like using a strong contrast or controlled lighting or also reflective materials can improve the tracking of the computer vision. This also allows the artist to be responsible for preparing the place or area so the computer can see clearly.

Overall, this reading made me see computer vision through a more structured system that has its own limitations. It also made me think about how these tracking systems can feel like surveillance, which gives the artist power to make the audience feel observed or even part of the piece in a more intense way. As someone working in interactive media, this makes me more aware that designing with computer vision is not just about making something work, but about understanding what the system can actually perceive and what it cannot.

Concept:
Instead of making a competitive game that includes winning or losing, I wanted to create something calm and creative while still connecting it personally. I decided to create “Morocco’s Door Studio” for my Midterm project. Initially, I got inspired by traditional Moroccan doors because of the detailed arches, colorful zellige tiles, and the decorative handles that make each door unique. Whenever I come across these doors in Morocco, each door is different, although the pattern and color can be similar, the entire final piece is different. The experience is similar to a digital sticker book because the user starts with a blank Moroccan door frame and can build their own entrance by selecting different decorative elements such as tiles, arches, and knockers while adding their own colors to it. The goal is for the user to experiment freely and design something that feels visually pleasing to them. Traditional Gnawa music will play in the background to enhance the atmosphere and make the experience feel immersive and culturally rooted.

Design:

The design will focus on Moroccan tones like gold, blue, red, and nude colors. The door frame will remain centered on the canvas as the main focus, while decorative options will appear as selectable buttons on the side of the screen (for example “Tiles” “Knockers” “Arches”). When the user selects a category and clicks on the door area, a decoration will appear at the mouse position. Each decoration will be represented as an object and added to an array so multiple elements can be placed on the screen. The experience will begin with an instruction screen that explains how to play. The user must press a key or click a button to begin designing. After decorating, there will be a “Reset” or “Start Again” button that clears the canvas so users are able to experiment again. Additionally, to make the interaction more engaging a soft sound will play when a decoration is placed.

Images: (inspo/planning for my game/experience)

Challenging Aspects:

One of the most challenging aspects of this project I think will be managing the different decoration objects and making sure they are stored and displayed properly. Since every click creates a new object, I need to make sure they are correctly added to an array and drawn continuously in the draw() function. Another challenging aspect is organizing the different design categories. I need to make sure that when the user selects “Tiles” only tile decorations are placed, and not arches or knockers. Another issue might be finding transparent PNG images for the decorations which might take a while, but I can try creating them on my own or maybe using AI for this part.

Risk Prevention:

Although I am still working on this project, and I won’t know what issues I will run into, to reduce the risk of the objects becoming confusing, I will first create a simple test version where clicking the screen generates basic colored shapes using a class that I recently found and learned about: Decoration. This will help me confirm that the objects are properly created and stored in the array.

Before adding detailed images, I will test everything using simple shapes like rectangles and circles. Once the system works, I will replace those shapes with the actual decorative images I have saved.

For the category selections, I will use a variable maybe naming it currentCategory that will update whenever a button is clicked and this way the program always knows which type of decoration to place on the screen.

I think some of the ways that computer vision differs from human vision is that while we see “meaning”, computer sees only the math behind it. Because human vision is very semantic, for example when we look at a video we instantly see a person, a mood or a story. But for a computer that same video is just a massive dump of pixel data and color coordinates. Because the computer has zero clue if it’s looking at a human or something else until we write an algorithm to prove it. To help the computer see what we’re interested in we can use cheats like  background subtraction, where the computer compares a live shot to a blank photo of the room to spot what’s new or frame differencing to track motion by subtracting one frame from the next.

The fact that computer vision is rooted in military surveillance really colors how it’s used in interactive art. I think because computer vision was born in military and law enforcement labs, they are designed to track and monitor. That’s why they bring a sense of control to interactive arts. For example, in works like Myron Krueger’s Videoplace the tracking is used for play. It turns your body into a paintbrush, giving you a role in the digital world. Also, projects like Suicide Box show that surveillance can be used to track the tragic social phenomena that the government might ignore.

The Concept 

I’ve decided to create an interactive game called “Star Catcher.” The core idea is a fast paced arcade game where the player controls a collector at the bottom of the screen to catch falling stars. Since I wanted the interaction with the game to feel smooth, I have decided it to be mouse driven. The user starts at a menu screen and has to click to enter the game. My goal is to at the end add more layers but for now, I’m focused on the core loop: fall, catch, score, repeat.

Designing the Architecture

I’ve already started laying the foundation of the code using object oriented programming. 

• The player class which handles the paddle at the bottom.
• The star class which manages the falling behavior, the random speeds, and resetting once a star is caught or hits the ground.
• State management where I implemented a gameState variable to handle the transition from the Start Screen to the Active Game and finally to the Game Over screen.

The “Frightening” Part 

The most scary part for me was asset management and collision logic. I was really worried about the game crashing because of broken image links or the sound not playing.

In order to minimize the risk I wrote a “fail safe” version of the code. Instead of relying on external .png or .mp3 files that might not load, I used: createGraphics() to generate my own star “images” directly in the sketch and p5.Oscillator to implement a synthesized beep instead of an audio file. This helped me to test the collision detection algorithm (using the dist() function) and get immediate audio feedback without worrying about any file paths.

 

 

Midterm Idea:

I was inspired by Portal 2 by Valve in this endeavour, with my own twist on the gameplay and concept.

While I haven’t settled on a fitting name thus far for the project, the concept entails a “multi-faceted” reality, with a ray-gun that fires a bullet. When this bullet is incident on specific “panels” in the game, it may reflect off the surface of the panel. The objective is to reflect the bullet such that it reaches a designated mechanism to progress to the next level or finish the game.

The Twist:

This gets interesting in such that there are two realities that this special ray-gun, bullet and panels engage with. That is, the ray-gun may fire bullets in either one of Dimension ‘A’ or Dimension ‘B’. If the panel is of the same ‘dimension’ as the bullet, it will hit the panel and shatter it (this may unlock new areas or cause a loss if the wrong panel is destroyed). Alternatively, if the panel and the bullet differ in their ‘dimension’, then the bullet reflects off the panel and this can then be further reflected off other panel as the concept graphics below show:

 

The game will feature inspiration from very simple brutalist architecture, with 1K PBR Textures because I like a degree of realism in my games. Maps will generally be dimly lit, and there will be cool snippets of lore that players can find as they explore different parts of the map or progress though the game.

User Interaction:

Other than the aforementioned ray-gun, there will be WASD keys for forward-backward and left-right movement. Additionally, to look around them, users can drag their mouse towards the sides of the screen to pan their camera. Users can walk to all accessible parts of the map, which will be made sufficiently obvious to avoid confusion, as I am intending on using a darker theme for this game.

Users may use E to fire a bullet, and Q to switch the dimension setting of the ray-gun, id est the dimension of the bullet it fires.

Progression:

There are a total of two levels I plan to implement, and users are limited to three shots from their ray-gun before they have to restart the ENTIRE game because I want to convey a feeling of greater loss, not just that they spawn back at their current level.

Development & Implementation:

I have thus far coded in a very simple game engine to handle all objects. I am going to implement physics and collisions soon, along with more textures and the map.

There are many classes in this, but the two I shall focus on here are ‘BasePart’, which handles a simple 3D box or part of the map in the game, and ‘Workspace’, which is a class that contains all BasePart and other extended instances in the game, as well as contains the camera that displays the map to the user.

Major Concerns and What I’m Dreading:

This projects may seem overly ambitious, but the initial game engine setup has come through nicely. I am not looking forward to collision handling since that always is a nightmare.

Code and Credits:

Find here a link to my code: https://editor.p5js.org/rk5260/sketches/b6B9BVyoE

All code is written by myself (exception below), with many elements I first learnt how to use from the p5.js documentation, after which I implemented them myself.

Exception: The code for the PBR shaders was composed and modified from multiple existing shaders on github and p5.js documentation itself.

All PBR textures sources are credited in a credits.txt file in the PBR Texture’s directory. (Thus far its only GravelConcrete).