Computer Vision is full of bits and bytes. There is no way perfect algorithm to obtain the computer vision of the background. The computer vision seems to only see the whole back as a series of pixels. This means that without human to define what these pixels are, the computer does not know what to do with the image in their vision. In other words, the image data is a stream of nonsense to the computer.
The way the computer analyze the changes in its vision is similar to how the sensors work, I think. For example, in one of the techniques mentioned in the reading, “detecting motion” is done by comparing the pixels of the first frame (F1) and the subsequent frame (F2). Any changes in pixels detected by the processor can be concluded that was a movement in the vision. However, it is clear that this technique is unable to recognize what type of shape is it as well as the color of the moving objects. Similarly, since this algorithm is created to only detect motion, it does not understand the general image that it is capturing.
Therefore, computer vision can only do good at the jobs that t is assigned.
For this project, I’ve drawn inspiration from a childhood game my brother used to play, called the “Fishing game.” It involved catching fish with a fishing hook within a time limit. I’ve adapted this concept into my own version of the game, adding some twists and modifications. Instead of a traditional fishing game with a box of fish, my game features fish scattered randomly across the canvas. The objective remains the same: capture each fish individually using a fishing hook (the mouse) and deposit them into a fish bowl before time runs out. This adaptation allows for a unique gameplay experience while still capturing the essence of the original game.
Design
1. The game begins with an instructions page featuring a button labeled “Click here to continue” at the bottom.
2. Upon clicking, the player is directed to a page where they can select the number of fish to be displayed on the canvas, adjusting the game’s difficulty.
3. After choosing the level of difficulty, the player clicks a button labeled “Click here to start the game.”
4. Upon starting the game, a countdown timer begins, indicating the time limit for completing the game.
5. Background music plays throughout the game, creating an immersive experience.
6. If the player successfully captures all the fish before the timer runs out, they win the game.
7. Upon winning, a new page displays congratulating the player on their victory and showing their completion time.
8. A victory music plays to celebrate the player’s success.
9. If the player fails to capture all the fish before the timer expires, they lose the game.
10. Upon losing, a new page shows that the player has lost the game.
11. An end game music plays to signal the conclusion of the game.
Challenges with the project
1. Implementing the functionality to drag and drop each fish individually to the fish bowl has posed a significant challenge.
2. Creating an on-screen countdown timer that accurately tracks the remaining time and triggers the end game page when it reaches zero has been another obstacle.
3. Providing users with the ability to select their preferred difficulty level by choosing the number of fishes in the game initially proved to be problematic, requires for code refining.
4. Design challenge: Positioning the fish bowl on a table above water and making the users to drag the fish from the water surface into the bowl, which would make it aesthetically pleasing.
Code
let fishes = [];
let fishBowl;
let fishingHook;
let timer;
let gameStarted = false;
let gameFinished = false;
let numFishes = 5; // default number of fishes
function preload() {
fishBowl = loadImage('bowl.png'); // Load image for fish bowl
fishingHook = loadImage('hook.png'); // Load image for fishing hook
gameMusic = loadSound('gamemusic.mp3'); // Load background music
victoryMusic = loadSound('victorymusic.mp3'); // Load victory music
losingMusic = loadSound('losingmusic.mp3'); // Load losing music
}
function setup() {
createCanvas(800, 600);
timer = new Timer(numFishes); // timer object with the number of fishes
}
function draw() {
background("#2196F3");
if (!gameStarted) {
displayInstructions(); // Display instructions if game hasn't started
}
else {
timer.update(); // Update timer
if (!gameFinished) {
// Draw fish bowl
image(fishBowl, width / 2 - 50, height / 2 - 50, 150, 150);
// Draw fishing hook
image(fishingHook, mouseX - 25, mouseY - 25, 50, 50);
// Display and update fishes
for (let fish of fishes) { //checks each elements of the "fishes" array
fish.display();
//hooking the fishes to the hook
if (dist(mouseX, mouseY, fish.x, fish.y) < 25) {
fish.hooked = true;
}
if (fish.hooked) {
fish.move(mouseX, mouseY); //the fish is hooked to the hook
}
}
// Check if all fishes are inside fish bowl
let allFishesInside = true;
for (let fish of fishes) {
if (!fish.insideBowl) {
allFishesInside = false;
break;
}
}
if (allFishesInside) {
gameFinished = true;
timer.stop();
//play music
if (timer.timeLeft > 0) {
victoryMusic.play(); // Play victory music if game finished before countdown ends
gameMusic.stop(); // Stop background music
}
else {
losingMusic.play(); // Play losing music if countdown ends before game finished
gameMusic.stop(); // Stop background music
}
}
}
else {
fill(255)
textSize(40);
textAlign(CENTER, CENTER);
text("Game Over!", width / 2, height / 2);
text("Time left: " + timer.getTime() + " seconds", width / 2, height / 2 + 40);
}
}
}
function displayInstructions() {
// Display instructions
fill('rgb(0,0,134)'); //color of ellipse
ellipse(width/2, height/2,650,150); //ellispe in the center behind the circle
fill(255); //color of the texts
textSize(40); //size of texts
textAlign(CENTER, CENTER); //texts in the center of the canvas
text("Click to start the game", width / 2, height / 2);
}
function mousePressed() {
if (!gameStarted) { //if the game is not started
gameStarted = true; //the variable is set to true and it resets the fishes array
fishes = []; // Reset fishes array
//adding fishes to the canvas
for (let i = 0; i < numFishes; i++) {
fishes.push(new Fish(random(width), random(height)));
}
timer.start(); //countdown starts
gameMusic.loop(); // Play background music on loop
}
}
class Fish {
constructor(x, y) {
this.x = x;
this.y = y;
this.hooked = false;
this.insideBowl = false;
this.fishImage = loadImage('fish.png'); // Load fish image
}
display() {
image(this.fishImage, this.x - 25, this.y - 25, 50, 50); // Draw fish image
}
move(x, y) {
this.x = x;
this.y = y;
// Checking if the fish is inside the fish bowl
if (dist(x, y, width / 2, height / 2) < 50) {
this.insideBowl = true; // If the distance is less than 50, then the insideBowl property is set to true
}
}
}
class Timer {
constructor(numFishes) {
this.totalTime = numFishes * 1; // Adjust the total time based on the number of fishes
this.timeLeft = this.totalTime;
this.running = false;
}
start() {
this.startTime = millis();
this.running = true;
}
stop() {
this.endTime = millis();
this.running = false;
}
update() {
if (this.running) { //if the timer is running
let timePassed = millis() - this.startTime; //calculates the time passed since the timer started
this.timeLeft = max(0, this.totalTime - floor(timePassed / 1000)); // Calculate remaining time
if (this.timeLeft === 0) {
this.stop(); // Stop the timer when time runs out
}
}
}
getTime() {
return this.timeLeft;
}
}
Future Improvements and plans
1. Add more colors and deigns to the overall design of the game to make it more attractive, I want to make the design more aesthetically eye pleasing.
2. Add an instructions page to guide users on how to play the game.
3. Enable users to drag the fishes individually to the fish bowl for a more interactive experience.
4. I want to add more music to the game like subtle sounds and musics when the user interacts with the game.
5. Refine all aspects of the code to align with the initial plans and ensure smooth functionality.
The reading on “Computer Vision for Artists and Designers” discusses how computer vision is becoming more accessible to students/artists due to easier-to-use software and open-source communities. Out of the many projects showcased, I was really impressed (and slightly creeped out) by Rafael Lozano-Hemmer’s installation Standards and Double Standards (2004), where belts were controlled by a computer vision-based tracking system, causing the buckles to rotate automatically to follow the public. It was really interesting to see this type of interaction that, in a way, isn’t intentional, direct, or digital. However, when it came to the project Suicide Box by the Bureau of Inverse Technology (1996), where a motion-detection video system was utilized to record real data of suicides, I found that ethically concerning. You don’t need to record such events to store data; yet, on the other side, it might serve as a security measure for people who have presumably been missing. It is a pretty controversial project, to say the least.
Furthermore, the reading discussed the different kinds of problems that vision algorithms have been developed to address, and their basic mechanisms of operation, such as detecting motion, detecting presence, object tracking, and basic interactions. All of which the designers of C2’s doors should have taken into account. Moreover, something new I have come across is the term “Telecentric lenses,” which are lenses used to improve object recognition by maintaining constant magnification regardless of distance. Yet, I came to find out that it is high in cost, large in size, and heavy in weight, in addition to causing some distortion issues. So, I wonder when it is appropriate to use it or if it is smart to do so to begin with. All in all, this was a very interesting read that showed me that interaction can be more than just key or computer-based; rather, it’s more about innovative ways to bridge the two different worlds we live in! Last but not least, I wonder where the line is drawn when it comes to privacy and motion/facial detection. Have we as a society come to accept that we are being watched & listened to all the time, whether it’s your phone’s facial recognition or the immediate response after “Hey Siri!” ?
The main point that stood out to me from this week’s reading was the wide prospects of the use cases surrounding computer vision.
To begin with, two artworks stood out to me for two varying reasons. Both artworks, however, expanded the scope of possibilities for me concerning the applications of computer vision within the context of art.
The first of these artworks is Rafael Lorenzo-Hemmer’s Standards and Double Standards (2004). This work piqued my interest due to its incorporation of space and inanimate objects which are activated through the help of computer vision. Personally, I find the overlap between the digital and the tangible to be an interesting area of focus so this work immediately caught my attention for its symbolic repurposing of an everyday object which is then given a sense of agency through programming that is supported by computer vision. Moreover, this work allowed me to consider the potential of using computer vision without requiring a visual output based on the data that the program is using. For example, in Krueger’s Videoplace, the user can see a visualisation of the input that the computer vision system is receiving (their silhouette) and it becomes central to the work. Conversely, Standards and Double Standards makes use of the input internally in order to trigger an another action. Finally, I definitely appreciated that this work does not feature a screen (!) as I feel that it has become an overly predictable method of presenting interactive art.
That being said, the next work that I have identified is Christopher Moller’s Cheese (2003) – an installation which solely presents screen-based work. While I do feel that this installation is an exception to the statement above (due to its bold imagery and simple presentation (and the fact that the work itself is not interactive)), what stood out to me was not the effectiveness of the work itself but the technical implications concerning the computer vision system that made the work possible. Considering the exponential development of technology, and the fact that the work was produced over two decades ago, one can’t help but wonder what can be done with facial recognition technology today. The reading mentioned how sophisticated the computer vision system needed to be in order to recognise slight changes in emotion and provide a response (albeit a simple one).
This has lead me to ponder what is possible with facial recognition technology (and computer vision as a whole) within the artistic space today. I was reminded of an installation produced in 2019 which I had looked at for another class entitled Presence and Erasure by Random International. As part of my presentation on this work I discussed the concept of consent within interactive art and, as an Arab and a Muslim, I immediately recognised that such a work would may not be able to exist in certain parts of the world (such as this one) as a result of social and cultural beliefs. Ultimately, going down this rabbit hole as led me to consider the endless possibilities we have with today’s technology but it has also helped me understand that just because you can pursue an idea, does not always mean that you should.
Concept:
I’m trying to make sort of a version of “Super Mario” game using one player who walks along the path collecting gold coins and the game ends when it collides with the enemy. The player have to jump (using UP key) to collect coin, and there’ll be sound incorporated with each jump and coin collection. When the game is over there’ll be a screen saying “Game Over” and if the player wants to play again they can press SHIFT key and the game will start again. There’ll be a scoring system displayed on the screen too.
Difficulty/Challenges:
Challenge would be to move the sprite smoothly along the background. I want to design the background myself and make the player and enemy designs too – to make it unique. I guess collaborating all the elements of the game together to respond to the user interaction would be a challenge so that the game runs properly.
Visualization:
I want my game screen to look like this (primarily, this is not a definite design, just drawn into Procreate as I’m currently using dummy sprites only to run the game code):
Coding:
There’ll be a Player class, an Enemy class, and a Coin class. I’ve designed the basic code for collision etc. Here are some highlighted code snippets:
Diving into the article about computer vision for artists and designers felt like opening a door to a room where art meets science in the most fascinating way. I’ve always thought of computer vision as something you’d find in sci-fi movies or high-tech security systems, not something that could be part of creating art. The idea that artists are using this technology to make interactive pieces where the artwork changes based on how people move around it is pretty cool. It’s like the art is alive, reacting to us just as much as we react to it.
Reading about projects like Videoplace and Sorting Daemon really got me thinking. It’s one thing to look at a painting on a wall, but it’s something entirely different to be part of the art itself. The thought that my movements could influence an art piece, creating something unique each time, is both exciting and a bit mind-blowing. It’s not just about observing; it’s about participating, and that changes the whole experience of what art can be.
The technical side of things, like how computer vision can track movements and interpret them into visual changes in an art piece, was a bit complex but also intriguing. I didn’t realize so much went into making these installations work. It’s not just about having a good idea for an art piece; it’s also about solving problems, like making sure the lighting is right so the cameras can see what they need to. This made me appreciate the art even more, knowing the blend of creativity and technical skill that goes into it.
However, what stuck with me the most was thinking about the bigger picture—like what it means for a machine to watch us and then create art based on that. It’s a bit like the machine is an artist too, interpreting our actions in its own digital way. And while it’s amazing, it also raises questions about privacy and how much we’re comfortable with machines “seeing.”
Overall, the article was a peek into a future where art and technology blend in ways that make the audience a part of the artwork. It’s a reminder that creativity knows no bounds, especially when artists start playing around with tech to see what new forms of expression they can discover. It makes me wonder what other surprises the art world has in store as technology keeps advancing.
I grew up in a village without a park, a place where families, friends, and pets gather to enjoy nature and each other’s company. To experience the joy of a park, I had to travel to nearby villages or cities. This sparked my love for parks and inspired my midterm project: creating a virtual park. My goal is to recreate the sense of community and fun found in a park, from children playing on swings to people walking their dogs and old friends catching up. This project is my way of bringing the park experience to those who, like me, have always admired it from afar.
Concept
Building on the foundational inspiration, the core idea of my midterm project is to craft an immersive user experience that mimics the tranquility and community feel of a real park. To achieve this, I’ve begun by setting up a serene background image that serves as the canvas for the park’s life. Central to the scene are two swings, each with a figure that moves in harmony with the swing’s motion. To ensure these movements are as natural as possible, I’ve implemented the lerp() function, which allows for smooth transitions and adds a lifelike quality to the animations.
Adding another layer of interaction to the park, I’ve introduced a jukebox feature. This allows users to engage with the environment actively by selecting and controlling the music to suit their mood, enhancing the personal experience of the park. While the music feature is currently in development, with plans to enable song changes and stops, it promises to be a significant aspect of user engagement.
Looking ahead, I plan to enrich the park’s atmosphere with spritesheets for various people movements, creating a dynamic and bustling park scene. This will include groups of people sitting on the grass or on carpets, adding to the communal vibe. Additionally, to further the immersion, I intend to incorporate ambient sounds, such as birds chirping, which will be a constant backdrop to the park experience, not subject to user control. This sound layer aims to deepen the user’s connection with the virtual environment, making it feel alive and vibrant. Moreover, diversifying the playground equipment, especially with different swings, will offer a variety of interactions for the user, simulating the choice and freedom one finds in a real park. This expansion not only adds visual appeal but also invites users to explore and find their favorite corner of the virtual park.
Challenging Aspects
Drawing Complexities: The first significant challenge is the detailed drawing of park structures and objects using p5.js. Every element, from swings to human-structure, requires time-consuming manual drawing. This process demands a high level of precision and artistic skill to ensure the park’s visual appeal and thematic consistency.
Interactive Features: Introducing interactivity, such as clickable buttons and objects that respond to mouse hover, adds another layer of complexity. For example, making the jukebox highlight when hovered over involves sophisticated input detection and dynamic response coding, enhancing user engagement but also increasing development complexity.
Animating Characters with Spritesheet: Utilizing spritesheets for character movement presents a formidable challenge. Ensuring these animated figures navigate the park without colliding with objects or wandering off the grass involves intricate collision detection and boundary management.
The article provides a comprehensive overview of the history, techniques, and applications of computer vision in the interactive media arts. It is interesting to see how artists have used computer vision to create novel and expressive experiences, such as Myron Krueger’s Videoplace, Rafael Lozano-Hemmer’s Standards and Double Standards, and David Rokeby’s Sorting Daemon. The article also explains some of the basic algorithms and tools for implementing computer vision, such as frame differencing, background subtraction, brightness thresholding, and object tracking. I find these algorithms useful and inspiring for my own projects, as they enable me to detect and analyze the movements and gestures of people and objects in real time.
Messa di Voce by Golan Levin and Zachary Lieberman was very interesting to me. This is an audiovisual performance that uses computer vision to track the locations of the performers’ heads and analyze the sounds they produce. The system then generates graphics that are projected from the performers’ mouths, creating a dynamic and expressive dialogue between sound and image. I think this artwork is a creative and engaging example of how computer vision can augment and transform human communication and expression.
I found Cheese by Christian Möller pretty funny. This is an installation that uses computer vision to measure the sincerity of the smiles of six actresses. The system uses an emotion recognition algorithm to detect the changes in the facial expressions of the actresses, and alerts them when their smiles fall below a certain threshold. I think this artwork is a humorous and critical commentary on the social and cultural expectations of happiness and beauty, as well as the limitations and biases of computer vision.
I was interested in the Detecting Motion 9code listing 1) concept in this week’s reading. Frame differencing is a simple technique that may be used to detect and quantify motions of humans (or other objects) within a video frame.
The reading emphasizes how computer vision algorithms are not naturally capable of comprehending the subtleties of various physical surroundings, despite their advanced capacity to process and analyze visual input. The physical environments in which these algorithms function can greatly increase or decrease their effectiveness. The use of surface treatments like high contrast paints or controlled illumination like backlighting, for instance, is discussed in the reading as ways to enhance algorithmic resilience and performance. This suggests that the software and the real environment need to have a symbiotic connection in which they are both designed to enhance one another.
This concept reminded me of The Rain Room. In my opinion, this installation has motion sensors that act like computer vision that allow people to move through a space where it is raining everywhere but where they are standing. The computer vision system’s exact calibration with the actual environment is crucial to the immersive experience’s success because it allows the sensors to recognize human movement and stop the rain from falling on the people.
I am super jealous 😤 of people who are extremely good at one thing. Hayao Miyazaki doesn’t have to think that much, but just keep making animated movies.. he’s the greatest artist I look up to. Slash and Jimi Hendrix does not get confused, they just play guitar all time, because they are the greatest musicians. Joseph Fourier does just Mathematics and Physics, with a little history on the side… but he’s still a mathematician.
My lifelong problem is that I specialize in everything, in extreme depths. When you are a competent artist, engineer, musician, mathematician, roboticist, researcher, poet, game developer, filmmaker and a storyteller all at once, it’s really really hard to focus on one thing….
which is a problem that can be solved by doing EVERYTHING.
Hence, for my midterm, I am fully utilizing all a fraction of my skills to create the most beautiful interactive narrative game ever executed in the history of p5js editor, where I can control the game by improvising on my guitar in real time. The story will be told in the form of a poem I wrote.
Ladies and gents I present you G-POET – the Guitar-Powered Operatic Epic Tale. 🎸 the live performance with your host Pi.
This is not a show off. This is Devotion to prove my eternal loyalty and love of arts! I don’t even care about the grades. For me, arts is a matter of life or death. The beauty and the story arc of this narrative should reflect my overflowing and exploding emotions and feelings I have for arts.
Also, despite it being a super short game made using JavaScript, I want it on the same level, if not better than the most stunning cinematic 2D games ever made in history – the titles like Ori and the Blind Forest , Forgotton Anne, Hollow Knight. They are produced by studios to get that quality, I want to show what a one man army can achieve in two weeks.
Design
I am saving up the story for hype, so below are sneak peaks of the bare minimum. It’s an open world. No more spoilers.
(If the p5 sketch below loads, then use Arrows to move left and right, and space to jump )
And below, is my demonstration of controlling my game character with the guitar. If you think I can’t play the guitar… no no no Pi plays the guitar and narrate you the story, you interact with him, and tell him, oh I wanna go to that shop. And Pi will go “Sure, so I eventually went to that shop… improvise some tunes on the spot to accompany his narrations, and the game character will do that thing in real time”.
See? There’s a human storyteller in the loop, if this is not interactive, I don’t know what is.
People play games alone… This is pathetic.
~ Pi
💡Because let’s face it. People play games alone… This is pathetic. My live performance using G-POET system will bring back the vibes of a community hanging out around a bonfire listening to the stories by a storyteller…same experience but on steroids, cranked up to 11.
No, such guitar assisted interactive performance in real-time Ghibli style game has not been done to my knowledge.
(In the video, you might notice that low pitch notes causes player to go right, and high pitch notes is for going left. There are some noises, I will filter it later.)
To plug my guitar to my p5js editor, I wrote a C++ native app to calculate the frequency of my guitar notes through Fast Fourier Transform and map particular ranges of frequencies to some key press events, which is propagated to the p5js browser tab. A fraction of the C++ code to simulate key presses is
Of course, I need the characters. Why do I need to browse the web for low quality graphics (or ones which does not meet my art standards), while I can create my own graphics tailored to this game specifically?
So I created a sprite sheet of myself as a game character, in my leather jacket, with my red hair tie, leather boots and sunglasses.
But is it not time consuming? Not if you are lazy and automated it 🫵. You just model Unreal metahuman yourself, plug the fbx model into mixamo to rig, plug into Unity, do wind and clothes simulation and animate. Then, apply non-photorealistic cel shading to give a hand-drawn feel, and utilize Unity Recorder to capture each animation frame, do a bit more clean up of the images through ffmpeg, then assemble the spritesheet in TexturePacker and voilà … quality sprite sheet of “your own” in half an hour.
Also when improvising the guitar during the storytelling performance, I need the game background music to (1) specifically tailored to my game and (2) follow a particular key and chord progression so that I can improvise on the spot in real time without messing up. Hence, I am composing the background , below is a backing track from the game.
In terms of code, there are a lot a lot a lot of refactored classes I am implementing, including the Data Loader, player state machine, Animation Controllers, Weather System, NPC system, Parallax scrolling, UI system, Dialgoue System, and the Cinematic and Cutscene systems, and Post Processing Systems and Shader loaders. I will elaborate more on actual report, but for now, I will show an example of my Sprite Sheet loader class.
And I am also writing some of the GLSL fragment shaders myself, so that the looks can be enhanced to match the studio quality games. An example of the in game shaders is given below (this creates plasma texture overlay on the entire screen).
Yes, there were a lot of frightening aspects. I frightened my computer by forcing it to do exactly what I want.
Challenges? Well, I just imagine what I want. In the name of my true and genuine love for arts, God revealed all the codes and skills required to me through the angels to make my thoughts into reality.
Hence, the implementation of this project is like Ariana Grande’s 7 Rings lyrics.
I see it, I like it, I want it, I got it (Yep)
Risk Prevention
Nope, no risk. The project is completed so I know there is no risks to be prevented, I am just showing a fraction of it because this is the midterm “progress” report.
