Category: S2025 – Shiloh

Link to sketch: https://editor.p5js.org/bobbybobbb/sketches/7XBZCIX_C

My project is a calm flower shop experience where users can purchase flowers and make bouquets. Users have 100 dollars to spend and create a bouquet. Once they’ve spent $100 and checked out, they can restart the experience by leaving the store. You can also just leave the store whenever, but until you’ve checked out and spent $100, the experience will not reset.

Implementation

I drew all the elements (from the store front to the flowers) by hand to give it this homey and soft aesthetic. I also chose lo-fi cafe music to set this tone of relaxation. Feedback for actions users make was very important to me when designing this; I wanted them to know exactly how their actions affect the experience. For example, for the clickable objects in the store, hovering over them will create a white mask over them. Another feedback I implemented was the mouse click sound that occurs after the user clicks on something clickable. I also wanted the outside of the store to be inviting and encourage users to click on the screen; every time a user hovers over the screen, the doors to the shop will open, encouraging them to come in and use their mouse. Otherwise, it remains closed.

When users go to check out with a cart full of flowers, I display a bouquet full of the flowers they have. I had to think about how flowers are arranged in angles and randomly translated and rotated each flower to make it seem like they’re situated in a bouquet:

push();
// random positions and orientation of flowers
translate(random(170,240),random(140,210));
rotate(random(-PI/6,PI/6));
// display each flower inside the cart
image(flowerOptionsBouquet[cart[i]],0,0,200,200);
pop();

One thing I had to think about was how to make images clickable because all my elements are displayed using images. Instead of creating hit boxes and defining boundaries, I wanted to just let the images themselves be tracked as clickable things. That’s why I settled on using createImg() instead of the loadImage() and image() functions. createImg() does the same thing, except the image has .show() and .hide() functions for easily turning them on and off. They also have .mouseClicked(), which allows me to call a function once the images are clicked. The downfall of this system is that it doesn’t rely on draw; the images are constantly being displayed unless you hide them. Even if you call background() in draw, the canvas doesn’t reset. That’s why the bulk of my work is done in setup(), but still works and responds to mouse clicks from the user. This method also requires booleans to keep track of which scenes are being displayed so I can turn things on and off. 

doorOpen = createImg("door_open.png","alt");

versus:

redFlowerBouquet = loadImage("red_flower.png");

It was very important to me that I got all the mechanisms, clicking, switching between scenes before creating something beautiful, so here are some examples of what the early prototypes looked like before:I  didn’t  even  spell  flowers  right:

Improvements

One thing I noticed was just how long my file was; I feel like there’s a lot of lines for a fairly simple project. It might be due to the fact that I have to turn on and off image so often. Next time, I’ll try implementing hit boxes instead of hiding/showing images and see if that lends to simpler code. I’d also like to implement a feature where you can buy multiple bouquets at a time given the money you have and be able to display all the bouquets in your collection. As of right now, you can only buy one $100 bouquet, but users might want to split up that $100 and buy 2 or more bouquets.

Images

Inside the store:
Hovering over objects:
Example bouquets:

Concept

For my midterm project, i created an interactive experience at the airport that is a bit different from the usual way we think about travel.  Instead of conventional destinations, the airport’s departure board offers emotional states that users might be seeking: peace, guidance, restart, sign, purpose, direction, and inspiration.

The idea behind this project came from this simple question:
“What if people travel not to visit a place, but to experience a certain emotion?”

Reflecting on my own travels, I realized that my strongest memories aren’t necessarily about locations themselves, but about how those places made me feel. So, once a user chooses their destination, they receive a quote/message they might need to hear.

Project features

In my project, there are three main pages:

1. Welcome Page:

   • The user is greeted with a sky background and a welcoming message, all while having airport sounds in the background.
   • A “Start” button starts the experience.

2. Airport Scene:

   • Once the user clicks “Start,” they are transported to an airport background with lofi sounds.
   • They need to click on the departure board to proceed.

3. Departure Board Page:

   • This page focuses on the departure board, listing different emotional “destinations.”
   • Lofi music plays softly in the background to enhance the experience
   • Then, clicking on a specific “destination” triggers a pop-up message with a quote related to that emotion or the desired state a person wants to achieve.

Problems I ran into

Some ideas I initially planned like zoom-in features and extra interactive elements were difficult to make in the given timeframe. So i had to make the most essential features first.

So, the biggest challenge was finding suitable images online that matched the vision I had in my mind. So, i had to create some elements myself and manually edit the departure board to fit my concept and include the emotional destinations. I also combined multiple images to achieve the desired design for each stage of the experience.

Initially, I tried drawing the entire page from scratch, but I later realized that wouldn’t work well with the multiple pages interactive format I wanted.

Future improvements

If I were to expand this project, I would like to add animations to create smoother transitions between pages and implement a customized quote generator where i take input like the name of the user to make it feels personal.

Despite the challenges, I’m proud of how this project turned out. I put in a lot of time and effort to make it look as close as possible to what I imagined. By making this project, i could represent a concept I resonate with: travel as an emotional journey rather than just a physical one.

Link: https://editor.p5js.org/Ellina_Chermit/full/i1vywV_MT

Link to fullscreen sketch

Concept

The idea behind my game came from a personal incident, and is intended to be funny and inspired by our very own NYUAD campus. The game setting is such that the player is working under the palm trees outside C2 competing with time to complete their assignment due in 5 minutes. Birds above have started attacking the player, who now needs to dodge the droppings falling from above while at the same time finish their assignment as soon as possible. The game is an adaptation of the classic dodge-style game with an added game element. This added element is a progress bar that represents the player’s assignment progress and it gets filled upon the repeated pressing of the space bar. The main goal is to fill up the progress bar (the win condition), while dodging the droppings falling from above without getting hit by any (getting hit is the lose condition).

Game components

The sketch starts by displaying a home page with instructions on how to play the game. Upon pressing the “Enter” button, the game starts. The images of the keyboard buttons on the instructions are taken from flaticon.com.

The main game screen consists of four elements:

1. bird
2. dropping
3. computer
4. progress bar (yellow long rectangle on the right).

Bird poop (dropping) falls from where the birds are located at the top of the screen, at regular intervals across random columns in different speeds. The computer is situated at the bottom of the screen and can be moved horizontally to different columns using the left and right keyboard buttons. When the space bar is pressed, the progress bar fills up (red indicates the level of assignment progress). At the top right, there is a timer that shows how much time has elapsed in seconds, and the best time  – which is the lowest time taken to complete the assignment, since the goal is to finish the assignment as quickly as possible without getting hit by a dropping – will be tracked across game plays and displayed on the home page.

I drew the images used for the bird, dropping and computer on Procreate and used a background remover application to make the background transparent. The image in the game background with the palm trees is taken from NYUAD’s website.

Implementation

Each of the four main elements (bird, dropping, computer, progress bar) are implemented as a class and have their respective functions, including display, fall, move, fill etc. There is also a Game class that manages game state variables, starts/resets the game and checks for collision between computer and the droppings.

One of the design decisions I made was to divide the canvas into (invisible) columns and have the computer move one column at a time instead of moving continuously, and droppings fall randomly from one of the columns instead of from any random pixel. This way, the user can’t “cheat” by placing the computer at a location where droppings rarely fall and it has to constantly dodge droppings, which I personally felt made the game a bit more challenging and fun.

class Computer {
  constructor() {
    this.x = 0;
    this.y = 480;
    this.alive = true;
    // initially static
    this.direction = 0;
    // place the computer at a random column
    this.col = int(random(0, numCols));
  }
  
  move(dir) {
    // if the right arrow is pressed
    if (dir === 1) {
      // numCols - 1 so that the last column is not accessible
      // last column space is reserved for placing the progress bar
      if (computer.col < numCols - 1) {
        computer.col++;
      }
    // left arrow is pressed
    } else {
      if (computer.col > 0) {
        computer.col--;
      }
    }
  }
  
  display() {
    this.x = colWidth * this.col + leftMargin;
    image(computerImg, this.x, this.y, computerWidth, computerHeight);
  }
}

A new dropping is created for every 1/3 second and its column and speed is randomly chosen. I settled on 1/3 second (frameCount % 20 === 0) among other values to make the game just challenging enough without frustrating the user by bombarding them with too many obstacles.

// create a new dropping 
if (frameCount % 20 === 0) {
  droppings.push(new Dropping());
}

for (let i = droppings.length - 1; i >= 0; i--) {
  let dropping = droppings[i];
  
  // check if any dropping has collided with the computer
  if (this.checkCollision(dropping, computer)) {
    // play sound effect
    blopSound.play();
    // remove the dropping from the array
    droppings.splice(i, 1);
    gameOverFail = true;
    break;
  } else {
    dropping.fall();
    dropping.display();
  }
  
  // remove droppings that have went beyond canvas
  if (dropping.y > height) {
    droppings.splice(i, 1);
  }
}

Problems I ran into

One of the biggest challenges was managing the state of the game such that the game can be restarted and the correct screens based on whether the game is in process, lost or won is shown without fail. I was able to do this by keeping track of three boolean variables:

let gameStarted = false;
let gameOverSuccess = false;
let gameOverFail = false;

...

// switch screens
if (!gameStarted) {
  instructions.display();
} else {
  game.run();
}

if (gameOverSuccess) {
  ...
} else if (gameOverFail) {
  ...
}

I track these variables in the draw() function and change what is shown on screen should any of these variables are modified.

Checking for collision between the computer and droppings was not very challenging as I implemented it as a simple function that checked whether any dropping overlaps in any direction with the computer, but it required configuring of values to ensure that collision was detected when the two appear to touch each other to the visible eye (instead of being triggered as soon as they barely touch) . As I had already tested creating a progress bar that fills upon a key press in the previous week as part of my midterm progress, incorporating it into the game took little time.

Future improvements

I quite like how the game turned out, both in terms of visuals and playing experience. However, one thought that I kept having was how nice it would be to be able to choose the difficulty of the game. I didn’t want to make the game too challenging so I settled on values for the speed and frequency of obstacles that would cater to the general audience, but this means that for certain groups of people who enjoy playing games, my game may appear very trivial. Therefore, one of the main improvements I would be interested in making is creating multiple degrees of difficulty (easy, medium, hard) that the user can choose from. I think this would make the game more engaging and potentially have users come back to the game to succeed in each round, instead of just attempting once.

Initial concept

I initially wanted to make a top-down shooter game, I designed a lot of stuff, a collision algorithm for the bullets, obstacles, and enemies, a multitude of weapons with special traits, power ups, an infinitely-expanding map, and a bunch of other interactive elements. I got really frustrated with the implementation and gave up on the idea. I came back a day later and didn’t know whether I should abandon the game or not, so I changed to code to make it a horizontal shooter game,and  below is where i reached, before completetly abandoning the idea, even though I had almost all the logic and algorithms implemented, I just couldn’t work with a project a didn’t love.

Hence the delay of submitting this assignment. 

Current concept

I woke up 2 days after, I saw a spider on top in some corner no one goes to in my house with a small net. I decided it to befriend it and I called it Webster.

This sparked an idea in me, which is to make a game of a spider swinging around the world with a web. Then I drew my initial concept.

The most frightening part

This would probably be the implementation of gravity and the physics of the web/rope. I already implemented them though.

class Spider {
  constructor(x, y) {
    // store position & velocity in p5 vectors
    this.pos = createVector(x, y);
    this.vel = createVector(0, 0);
    // spider radius = 15 is used for collisions
    this.radius = 15;
    // track if spider is attached to rope or not
    this.attached = false;
  }

  update() {
    // apply gravity each frame
    this.vel.add(gravity);

    // if spider is attached, we do some rope physics
    if (this.attached && ropeAnchor) {
      // figure out how far spider is from anchor
      let ropeVec = p5.Vector.sub(ropeAnchor, this.pos);
      let distance = ropeVec.mag();

      // only if rope is stretched beyond rest length do we apply the spring force
      if (distance > ropeRestLength) {
        let stretch = distance - ropeRestLength;
        // hooke's law, f = k * x
        let force = ropeVec.normalize().mult(stretch * ropeK);
        // add that force to our velocity
        this.vel.add(force);
      }
    }

    // apply damping (which is basically air resistance)
    this.vel.mult(damping);

    // move spider according to velocity
    this.pos.add(this.vel);
  }

  show() {
    // draw the spider sprite instead of a circle
    push();
    imageMode(CENTER);
    image(spiderImg, this.pos.x, this.pos.y, this.radius * 2, this.radius * 2);
    pop();
  }
}

What are some of the ways that computer vision differs from human vision?

While humans tend to rely on context, experience, and intuition to recognize objects and interpret scenes in front of them, computers process raw pixel data and require algorithms for example to to make sense of whatever visual input they are “seeing”. In addition to that, human vision can naturally adapt to different lighting and lighting changes, whereas computer vision can struggle with color perception under different illumination conditions. Similarly, motion recognition when it comes to humans is more intuitive and predictive, whereas in computers, it depends on frame differencing or object tracking for example to detect movement.

What are some techniques we can use to help the computer see / track what we’re interested in?

According to the paper, to help computers see and track objects we’re interested in, frame differencing can be used. When using this technique, frames are compared, and if pixels change between these frames, the computer sees it as movement. Another technique is brightness thresholding, which separates objects from the background based on their brightness levels. In simple terms, the process involves setting a specific brightness value (aka the threshold), and any pixel brighter or darker than that value is considered part of the object or background. For example, in an image, if the threshold is set to a certain brightness level, pixels brighter than that will be identified as the object, and those darker will be treated as the background.

How do you think computer vision’s capacity for tracking and surveillance affects its use in interactive art?

I think computer vision’s capacity for tracking and surveillance has really  expanded the possibilities of interactive art by allowing artist to create art that allows real time audience engagement and way more personalized experiences. Installations can now respond dynamically to movement and different  gestures, creating really immersive environments that evolve based on the viewer’s presence. By using computer vision, interactive art becomes more fluid and responsive, transforming the more traditional passive viewing of art, into something more active and engaging. I think as a whole, this technology not only enhances and improves  the storytelling and emotional impact of art, but also opens new doors in terms of  large scale public art and immersive installations that really blur the line between the digital and physical worlds.

Concept

“Stock Picker Fun” is a fast-paced, simplified stock market simulation game. The player’s goal is to quickly decide whether to buy or sell stocks based on their recent price trends. The game features:

• Simplified Stocks: Three fictional stocks (AAPL, GOOGL, MSFT) with fluctuating prices.
• Quick Decisions: Players must make rapid buy/sell decisions based on visual cues.
• Visual History: Mini-graphs display each stock’s recent price history, aiding in decision-making.
• Clear UI: A clean and intuitive user interface with color-coded indicators.
• Progressive Difficulty: The speed of stock price changes increases over time, adding challenge.
• Profit/Loss Tracking: A simple display of the player’s money and score.

A Highlight of Some Code That You’re Particularly Proud Of

I’m particularly proud of the drawGraph() function:

function drawGraph(data, x, y) {
    stroke('#fff');
    noFill();
    beginShape();
    for (let i = 0; i < data.length; i++) {
        vertex(x + i * 5, y + 40 - (data[i] - data[0]) * 0.5);
    }
    endShape();
    noStroke();

Embedded Sketch

Reflection and Ideas for Future Work or Improvements

Reflection:

This game successfully simplifies the stock market experience, making it accessible and engaging for a wide audience. The visual history and clear UI provide valuable feedback, allowing players to quickly grasp the mechanics and make informed decisions. The progressive speed adds a layer of challenge, keeping the gameplay dynamic.

Ideas for Future Work or Improvements:

1. More Data Visualization:
   • Add candlestick charts or other advanced visualizations to provide more detailed stock information.
   • Implement real-time data streaming from an API to simulate live market conditions.
2. Advanced Trading Features:
   • Introduce different order types (limit orders, stop-loss orders).
   • Add the ability to short stocks (bet on price declines).
   • Include options trading.
3. Dynamic News Events:
   • Generate random news events that impact stock prices, adding an element of unpredictability.
   • Use visual cues or animations to indicate the impact of news.
4. User Profiles and Persistence:
   • Implement user profiles to save game progress and track performance over time.
   • Use local storage or a database to persist data.
5. Sound Effects and Animations:
   • Add sound effects for buy/sell actions, price changes, and game events.
   • Incorporate more animations to enhance the visual feedback and create a more immersive experience.
6. More Stock types:
   • Add more stock types, that have different volatilities.
7. Game over conditions:
   • Add game over conditions, such as running out of money.
8. Add a pause feature:
   • Add a pause feature to the game.
9. Mobile optimization:
   • Optimize the game for mobile devices, using touch controls and responsive design.

By implementing these improvements, the game can be transformed into a more comprehensive and engaging stock market simulation.

Introduction

Computer Vision is the amalgamation of various mathematical formulae and computational algorithms accompanied by several computational tools – capable of carrying out the procedure. What was once deemed to expensive and high level (limited to experts such as on AI and signal processing), computer vision now has become readily available. Various software libraries and suites provide student programmers with the ability to run and execute those algorithms required for the object detection to work. The cherry on top, with mass refinement and larger availability of computer hardware, at a fraction of cost of what would have been in the early 1990’s,  now anyone, and by anyone I mean all of the institutions can access it and tinker around with it.

Difference between computer and human vision:

Computer Vision has a designated perimeter, where it scans for array of objects vertically and horizontally. Upon detecting a change in shade of pixel, it infers detection. By using complex algorithmic thinking, which is applied in the back-end, it is able to analyze and detect movement among various other traits such as character recognition etc. Various techniques like “Detection through brightness thresholding” are implemented. Alongs the similar lines happens to be the human vision. Our retinas capture the light reflecting from various surfaces, using which our brain translates the upside down projection into a comprehensible code. Our brain is trained to interpret objects, while computer vision requires algorithmic understanding and aid of Artificial intelligence. With AI, amongst a data set, training is done be it supervised or not, to teach the computer how to react to a certain matrix of pixels i.e image scanned.

Ways to make computer vision efficient:

As mentioned in the reading and the paper,  one of the things that I love is ‘background subtraction’. The capability to isolate the desired object. In my opinion, tracking several objects using this technique, and having a variety on the trained data set helps with more accurate and precise judgment. Especially if many objects are present at the same time. However, other techniques such as ‘frame differencing’ and ‘brightness thresholding’ exist as well. Also, from other readings, the larger the data set, and training time, the more the accuracy. However, to acquire image data, it comes with ethical dilemmas and added cost.

Computer Vision’s surveillance and tracking capability, and its implementation in interactive Media:

Works like Videoplace and Messa di Voce are example of earlier demonstration of interactive media and computer vision’s combination. Installations can track and respond to human input. This ‘feedback loop’ triggers a sense of immersion and responsiveness. In my humble opinion, the use of computer graphics takes away the user from traditional input techniques and gives them freedom to act as they will. Though it is also true, that the computer will make the sense out of the input, adjacent to the trained data set, and a totally random input might lead the system to fail. This is where the idea of ‘degree of control’ comes into play. Personally, I believe, as long as we have a combination of interactive components, user will never get tired of running inside the same tiring maze, but the use of computer vision definitely makes it seem less tangible and more user centered. Hence, I decided to use it for my midterm project as well!

1. Concept

I was inspired by the ‘coffee shop expo’ project, where the user can explore a place and click on different buttons freely; but I wished for more room for exploration and especially more freedom, in the way that a user can control a sprite  using keys to move around. Then, if this sprite lands on a specific place, a new place of discovery opens up.

I spent a considerable amount of time to develop the concept: a 2D RPG adventure, casual game in a pixelated world with a medieval setting, with story snapshots (as clear, not-so-pixelated images) appearing from time to time as a quest is completed. The user takes on a role as a character (represented with a sprite) that has migrated to a new city and wants a new job to get some money. There is a task board in a tavern villagers come to post various miscellaneous tasks. The user can pick a job from it, some examples below though my goal is to incorporate at least two from the following:

• • fetch water from the well,
  • harvest a number of carrots,
  • doctor’s mission – fetch herbs for medicinal purposes,
  • help me find my dog,
  • blacksmithing job,
  • help deliver letter.

I was wondering whether it would be strange to haveboth pixelated and non-pixelated graphics. After explaining my concept idea to my friend, my friend thought of a game that existed like that: “Omori.” Omori has pixelated sprites but clear CG scene snapshots, as well as iconic music – things to get inspired by – as well as a fan wiki for sprites which I could try to make good use of.

Interactive elements are to be present, such as a mouse click for opening a door and revealing a character’s speech. Audio is also heard for different story scenes, which should be chosen appropriately based on the atmosphere – eg. music with a sense of urgency on the task  board, relaxing music in the village, music with a sense of triumph and excitement when the quest is completed, etc. On-screen text could be used to beckon and inform the user of narration, instructions and dialogue.

After the experience is completed, there must be a way to restart the experience again (without restarting the sketch).

2. Code Highlights

I think the main challenge would be in the OOP, specifically making the character classes. I found a useful resource for collecting sprites: Spriter’s Resource. In particular, I would like to use village sprites from Professor Layton and the Curious Village. Here are the Character Profiles. I selected the following from the various characters in Professor Layton and the Curious Village:

• Luke (user’s character)
• Franco (farmer who needs help with harvesting carrots),
• Ingrid  (neighbour grandma who needs help with delivering a letter),
• Dahlia (noblewoman with a lost cat),
• Claudia (Dahlia’s cat),
• Lucy (young girl who needs help with getting herbs for her sick mother),
• Flora (mysterious herb seller).

Since the challenge lies in OOP, I would like to practise making an object, namely the user’s character “Luke.”

In the development of the code, I found it challenging to adapt the walk animation code (discussed in class with Professor Mang) it in two ways: (1) into an OOP format and (2) to adapt it to my spritesheet, which does not have different frames of walking and does not have different different directions that the sprite faces. With (1):

• I decided to have variables from the walk animation placed into the constructor as the class’s attributes.
• Instead of keyPressed() as in the walk animation, I used move() and display() since keyPressed() is not an allowed function within a constructor (probably it’s an issue because of it being local when it should be a global function)

class User_Luke {
  constructor() {
    this.sprites = [];
    this.direction = 1;  // 1 = right, -1 = left
    this.step = 0;
    this.x = width/20;
    this.y = height/15;
    this.walkSpeed = 3;
    this.scaleFactor = 0.2; // Scaling factor
    
    // 12 images across, 4 down, in the spritesheet

    let w = int(luke_spritesheet.width / 6);
    let h = int(luke_spritesheet.height / 2);

    for (let y = 0; y < 2; y++) {
      this.sprites[y] = [];
      for (let x = 0; x < 6; x++) {
        this.sprites[y][x] =
          luke_spritesheet.get(x * w, y * h, w, h);
      } // iterate over rows
    } // iterate over columns

    this.x = width / 2;
    this.y = height / 2;

    imageMode(CENTER);

    // Display first sprite
    image(this.sprites[0][0], this.x, this.y);
  }
  move() {
    ...
  }
  display() {
    ...
}

With (2), I set conditions if direction is 1 (representing right direction) or -1 (representing left direction). Since my spritesheet only shows the sprites in one direction, I used an image transformation:

class User_Luke {
  constructor() {
    ...
  }
  move() {
    ...
  }
  display() {
    let spriteWidth = this.sprites[0][0].width * this.scaleFactor;
    let spriteHeight = this.sprites[0][0].height * this.scaleFactor;
    
    // Finally draw the sprite
    // The transparent areas in the png are not
    // drawn over the background
    if(this.direction === -1) {
      image(this.sprites[0][0],this.x,this.y, spriteWidth, spriteHeight)
    } 
    else if(this.direction === 1) {
      // We will use the scale() transformation to reverse the x-axis.
      // The push and pop functions save and reset the previous transformation.
      push();
      // Scale -1, 1 means reverse the x axis, keep y the same.
      scale(-1, 1);
      // Because the x-axis is reversed, we need to draw at different x position.
      image(this.sprites[0][0], -this.x, this.y, spriteWidth, spriteHeight);
      pop();
    }
  }
}

I also noticed my sprite appear at a huge size – to deal with this, I used a scale factor in the spriteWidth and spriteHeight (already shown in above code).

3. Embedded Sketch

4. Reflection and Next Steps

I experienced multiple challenges along the way, but I gained valuable experience with OOP. I feel that making the next sprites won’t be so challenging, since I can use Luke’s code as reference and adapt it. I think it will be important for me to plan deadlines for me since there are big subtasks for this midterm project including:

• Finding background(s)
• Finding snapshots
• Coding all the sprites
• Interactive elements – door open animation, ‘choose a job’ buttons

• What are some of the ways that computer vision differs from human vision?

No computer vision algorithm is universally able to perform its intended function (eg. recognizing humans vs. background) provided any kind of input video, unlike the human eye and brain which can work together in general to perform its intended function (eg. recognizing humans vs. background). Instead, the object detection algorithm or tracking algorithm crucially relies on distinctive assumptions about the real-world scene that it is to study. If the algorithms’ assumptions are not met, then it could perform poorly, producing not very valuable results, or completely fail in its function. Take a first example: frame differencing is a computer vision technique that detects objects by detecting object movements. This is achieved by comparing the corresponding pixels of two frames by finding the difference in color and/or brightness between all corresponding pixels. Thus, the frame differencing algorithm can perform accurately on “relatively stable environmental lighting,” and “having a stationary camera (unless it is the motion of the camera which is being measured).” Hence, providing videos with much active movement, like in the NBA games, would be much more suitable than providing videos of focused people in the office. In addition to frame differencing, background subtraction and brightness thresholdings are more examples where having some presumptions are important for computer vision tasks. Background subtraction “locates visitor pixels according to their difference from a known background scene” while brightness thresholding uses “hoped-for differences in luminosity between foreground people and their background environment.” Thus, considerable contrast in color or luminosity between foreground and background is important for an accurate recognition of objects; otherwise, in nighttime scenes, the algorithm may detect objects in the video scene incorrectly as background. On the other hand, I personally feel that the human eye remarkably uses a combination of these three, and perhaps more, algorithms to detect objects, which allows it to perform extraordinarily well compared to current computer vision.

• What are some techniques we can use to help the computer see / track what we’re interested in?

It is of great importance to design a physical environment with the conditions best suited for the computer vision algorithm, and, in the other way, select software techniques that are best with the physical conditions at hand. There are interesting examples that stood out to me for enhancing the suitability and quality of the video input provided to the computer vision algorithm. I believe that infrared (as used in night vision goggles) should complement conventional black-and-white security cameras, which can massively boost signal-to-noise ratio of video taken in low-light conditions. Polarizing filters are useful to handle glare from reflective surfaces, especially in celebrity shows. Of course, there are lots of cameras to consider as well, optimized for “conditions like high-resolution capture, high-frame-rate capture, short exposure times, dim light, ultraviolet light, or thermal imaging.”

• How do you think computer vision’s capacity for tracking and surveillance affects its use in interactive art?

Computer vision’s capacity for tracking and surveillance opens doors for interactivity between the computer and the human body, gestures, facial expressions and dialogue/conversations. Already some difficult algorithms can correctly identify facial expressions, which could be used to detect someone’s emotional levels, and can be used in mental health initiatives to help people suffering emotionally. This might relate to Stills from Cheese, an installation by Christian Möller. Additionally, like in Videoplace, participants could create shapes using gestures, and their silhouettes in different postures can be used to form different compositions. If computer vision were combined with audio and language, then systems could be even more interactive with the increase in affordances.

Week 5 – Reading Response

->  Computer Vision for Artists and Designers

I believe that computer vision differs from human vision in several ways. First, unlike human vision, which is connected to cognition, experience, and contextual understanding, computer vision processes images as raw pixel data. This means that it does not know what it is looking at unless it is programmed to recognise specific patterns. Second, humans are able to recognise objects in various lighting conditions and angles, while computer vision usually has a hard time with these unless trained on extensive datasets. Additionally, human vision uses sensory data in real time with prior knowledge, while computer vision relies on predefined algorithms to take out meaningful information. Lastly, on instinct, humans are able to infer meaning from abstract images or recognise emotions in facial expressions, but computer vision needs complex models to achieve basic levels of recognition.

Due to the fact that computer vision lacks human intuition, we can improve its ability to recognise and track objects. This may be done by using lighting conditions to ensure the subject is distinguishable from the background (Myron Krueger’s Videoplace). Also, you could enhance visibility in low-light conditions by using infrared light, which is not visible to humans but detectable by cameras. Computer vision’s ability to track people and objects has a significant impact on interactive art. This means that artists can now use vision-based systems to create interactive installations that respond to body movement, voice, or gestures. It can also be used to highlight issues of privacy, surveillance, and control (Sorting Daemon). Overall, computer vision can reshape interactive art by allowing new forms of engagement, however, its surveillance capabilities also raise questions about privacy and ethics. This means that while this technology enables creative expression, it can also be a tool for control, highlighting the importance for artists/designers to handle these implications thoughtfully.