Month: February 2024

My midterm project will center around an idea of a digital drum kit. As a drummer, I sometimes want to test new beats and patterns when I am not around a physical kit, and in such times I like to use the “musicca” website(https://www.musicca.com/drums), and this is the website that I got the inspiration for my project from. Each sound on the drum kit is tied to a specific key and the digital drum kit is played just like a physical one would, by combining sequences of keys corresponding to the notes. Accurate and user-friendly implementation of the sounds and keys will be a major part of the project. I will use the image below for a drum kit and I will label each drum/cymbal with the corresponding key so that new users can easily navigate and learn in a matter of seconds. I have taken this image from “musicca”, only editing out the drums sticks that are present on the website as I think my implementation does not require drum sticks, because of a different visual feedback that is discussed below.
Additionally, I have found all the essential sounds for each key from the website “freesound” (https://freesound.org/).
Next, I wish to make the project more visual so that besides the audio feedback, users can also get a visual feedback corresponding to the sound that has been played. For that I will implement circle waves that arise from the center of the drum/cymbal that has been played and will propagate across the screen. This is going to be the most challenging part of the project, and I am thinking about implementing collision effect for those waves, but I will decide on it later depending on the visual effects created. I am still searching for ideas to implement besides the circle waves that will make appealing visuals, but so far I sympathize with this implementation the most.
There are 2 classes for the project so far. A class for drums, each object being a different drum or cymbal, and a class for the waves, each object being the wave tied to a certain drum/cymbal.
There will be a button at the bottom of the drum kit image, suggesting the users to start a new session if the previous user is done, and the functionality of button is simply to remove the existing waves from the screen, everything else does leave a residual effect from the previous to the new user.

Concept, Thoughts and Research

For the midterm project ,I had initially thought of making a game but decided to challenge myself with something different.  I thought of making interesting generative art with user interaction and customization. I came across this kaleidoscope piece by Sheenest that uses Perlin noise to automatically generate random kaleidoscopes (without user input) – https://github.com/sheenest/Perlin-Kaleidoscope-Project . This idea of using symmetry to create generative art has always intrigued me . Kaleidoscopes have inspired a lot of art and any way of automating them + making them interactable is something I want to work on . Some cool kaleidoscope art that I found :

Something like this is inspiration for what I want to do. For sound effects, I plan on using my favorite Youtuber Melodysheep’s sound library –

Music Library — Melodysheep

This sound library was originally designed for space /astronomy videos but I believe  alternating the tracks will create a very immersive experience for the user .

The user interaction that I want to add to this project would be :

1. Some way to draw something on the screen
2.  Then play the program, the program uses Perlin noise to change the points and colors of what was drawn to create beautiful patterns
3. Customizable bars for the user , I found a library on GitHub that automatically generates a GUI for the variables in the program . Here’s the link – https://bitcraftlab.github.io/p5.gui/.
4. A way to save the image
5. An option to add your own image instead of drawing something
6. (maybe) use computer vision in some way to use audio to influence the pattern

Identifying Challenging aspects of the project 

Some key challenges that I expect to overcome are :

1. Adding user interactivity AND applying Perlin noise is something that hasn’t been done before in such projects. Enabling the user to draw something with the mouse(on a graphics buffer that can be hidden later) is going to be challenging to implement .
2. Thinking about how noise can be used to vary the user input so that it appears different but not too different . The user input should still significantly affect the final result.
3. Sound effects associated with the project – when to play and pause sounds and if computer vision ends up being a part of it , how should sound influence the noise factor ?

Understanding Code for Kaleidoscope 

I started by looking at some classes in Sheenest’s code that I could recycle and modify to integrate user input into. One class that I plan to use after modification is the lin class (which represents a randomly drawn line) :

class lin {

  constructor( maxCurl , maxSpeed , size , sides , strokeMax , strokeMin , steps , fluct = 5 , order = 0.6) {

    // this.keyVel = createVector( 0 , 0 );

    // maxSpeed = 30; 
    this.maxCurl = maxCurl;
    this.maxSpeed = maxSpeed ;
    this.size = size ;
    this.sides = sides; 

    this.strokeMax = strokeMax ;
    this.strokeMin = strokeMin ;

    this.steps = steps ;

    this.trail = [] ; 
    this.vel = new polar ( 0 , 0 );

    this.noiseRad = new perlin( 
      'noiseRad',
      random(100),
      random(10),
      0.2 ,
      this.steps , 
      fluct , // more octaves/fluct , more fluctuation in velcities, default value is 5
      0.5 // 
    );

    this.noiseTheta = new perlin( 
       'noiseTheta', //name
       random(100), //seed 
       random(10), //step 
       0.2 , // counts 
       this.steps , 
       4 , // lod
       order //  order referring to the falloff level of the curves
       // more order/falloff , more persistence in the curves (more circles)
    );


    this.cHSB = []; 

    this.noiseHue = new perlin(
      'noiseHue',
      random( 100 ),
      random( 10 )
    );

    this.noiseSat = new perlin(
      'noiseSat',
      random( 100 ),
      random( 10 )
    );

    this.noiseBright = new perlin(
      'noiseBright',
      random( 100 ),
      random( 10 )
    );
    
    let bound = 1/sqrt(2) * boundary/2 ; 
    this.o_pos = createVector( random( - bound , bound ) , random( -bound, bound ) ); //random start position
    //this.pos is the independent position value that syncs with trail[0] after each iteration
    this.pos = this.o_pos.copy(); 

    //this.hueVal and this.cVal is the independent color value that syncs with cHSB[0] after each iteration
    this.hueVal = random(360) ;//random hue value at the start
    this.cVal = color ( this.hueVal , 0 , 0 );

    this.trailMemory = [] ; 
    this.cHSBMemory = [];
    this.memorySize = 1000;
    this.colormemorySize = this.memorySize * 4 ; //coz max inverse_speed = 4, each iteration of update() adds a max of 4 color values into cHSBMemory

    this.reverseCount = 0 ;
    this.reverseColorCount = 0 ;

    this.inverse_speed = [] ; // first value is for syncing of colors when this.vel is low
    // array size is equal to memmory size, to save past inver_speed values to tally with inverseColorCount

  }

This class creates a line object . I plan on adding simple user interactivity with something like this by creating a line object from user input and making an instance of this class -then manipulating the lines smoothly .

A very simple and primitive  example of the kind of code I can use for user interactivity is:

function draw() {
  if (mouseIsPressed) { // check if the mouse button is pressed (preferably the left click button) 
    stroke(0); // set the stroke color 
    strokeWeight(5); // set the stroke weight
    line(pmouseX, pmouseY, mouseX, mouseY); // draw a line from the previous mouse position to the current mouse position
  }

Instead of drawing a line using the line function, I will use the constructor for the lin class to create an instance of the line / multiple lines drawn by the user.

I have spent much time understanding the code generally used to create kaleidoscopes and thinking about design and user interactivity  .I look forward to  integrating user interactivity to create an intriguing generative art experience.

 

 

 

 

This weeks reading was about Computer Vision, its types and implementation. One thing that stuck out for me was the accessibility of simple computer vision techniques and the emphasis the author put on how even undergraduate or high school students can use some of the techniques described in the text. The author even provided source code for four of the techniques mentioned in the paper, which is another indicator of accessibility of the subject. Presenting the subject of computer vision to a widespread audience with different interests and proficiency in the field can result in a broader reach of computer vision into different disciplines and we can see applications of a vast variety and creativity.
However, it is not be mistaken as an easy subject, as it has been regarded in the past, but rather rather a subject with low barrier of entry and high ceiling, as the opportunities and possibilities with computer vision keep increasing indefinitely, especially with novel applications using AI and ChatGPT, self-driving cars, AI image generators, etc. This seems like the tip of the iceberg as we enter into an era of advanced AI and VR, hence this reading can be an excellent starting point for anybody who wants to interact with the roots of the tree that might shape the future.

Concept and User Interaction

One of the games that went viral in South Korea is the Ideal Type World Cup. It’s a simple game where several rounds are played, each presenting two options of celebrities, and you have to choose one out of the two. The selected option advances to the next round, and you continue choosing the one you like more until there is only one winner left.

Recently, while I was studying, I wanted to take a break, so I decided to play this game with my friends. I found one on the internet about the ideal Korean male actor and then played another one about food. Then, it occurred to me that it would be very interesting to try making this World Cup with the food we have on campus. The final idea for my midterm project is to create a Campus Food World Cup, where there will be photos of food from places on campus (Marketplace, D2, D1, Mysk, Blacksmith, etc.). Users will click on the photos of the food they prefer to find out what their top campus food is.

Design

Some elements that I would like to add to my project, as part of the design, include a beautiful interface and fonts. Additionally, there will be background music playing during the game and another piece of music at the end of the game. When the user selects the winner, I would like to add a sound effect. The overall interface will feature colors related to our campus, using NYU colors, etc., to establish a connection with our campus. Also, on the starting page, I plan to add some text about our campus and the dining systems we have.

Code

The biggest challenge I faced initially when I started to conceptualize the logic for my project was figuring out how to make my code remember the winners I had chosen from each round and continue the game to the next round until there was only one winner. To understand how the logic should work, I decided to experiment with numbers before loading all the pictures and designing the project. Therefore, I created an array of 32 numbers.

To facilitate several rounds of the game, I introduced a ‘winner’ variable, which would be determined by the mouse location.

if (currentPair.length === 2) {
    let winner = (mouseX < width / 2) ? currentPair[0] : currentPair[1];
    nextRoundCandidates.push(winner);

If the mouse location is on the left half of the canvas, the ‘winner’ variable would be the element on the left side, and the same logic applies to the right side.

Then, I created an array called ‘nextRoundCandidates’ so that the array would contain all the elements that should pass to the next round.

if (currentRoundCandidates.length === 0) {
    if (nextRoundCandidates.length === 1) { // If only one candidate remains, the game is over
      gameIsOver = true;
      return;

If there are no more candidates left for the next round, the program recognizes that the game is over.

if (gameIsOver) { // Restart the game if it's over
  gameSetup();
  return;
}

The program then executes the gameSetup() function.

function gameSetup() {
  // Reset and reshuffle the game for a new start
  candidates = Array.from({length: 32}, (_, i) => i + 1);
  currentRoundCandidates = shuffle(candidates);
  
  nextRoundCandidates = [];
  currentPair = [];
  updateCurrentPair();
  roundNumber = 1;
  gameIsOver = false;
  loop(); // Restart the drawing loop
  
  // Show a restart message
  showMessage = true;
  messageStartFrame = frameCount;
}

This code resets the game to its starting conditions, reshuffling the candidates and resetting all relevant variables for a new game.

Although there are still many aspects to add to this project, I am pleased that I was able to write the code for the main logic of the game. Now, I will focus on the details and design part of the project to make it visually appealing.

This week’s reading on computer vision was very interesting. The key idea presented was how computer vision is becoming more and more accessible to ordinary people and how we can use its power to create better and more interactive art.

The various uses of computer vision that the author gives in the beginning of the article illustrate how simple concepts can be so effective at generating something intriguing. The author’s discussion of some simple algorithms that enable object tracking was particularly interesting to me. I looked deeper into the code listings and found the algorithms very useful. I hope to include them in some way for my midterm project.

Additionally, the author’s emphasis on cleverly designing the physical environment stood out for me. His demonstration of how code can be simplified by creating the right physical environment was interesting to think about . It reminded me of how sometimes we focus on the wrong problem and looking at it from a broader perspective may greatly help . 

Overall, the idea of computer vision becoming more and more accessible to the general public excites me . With the advent of AI tools like Sora and Dall-E ,it seems like it’s only going to be a matter of time until creativity and design takes precedence over raw technical skill. These are the skills that I look forward to developing .

 

Project Concept
The core idea behind my midterm project is to develop an interactive game controlled entirely through hand gestures. The game will leverage a hand-tracking library to interpret the player’s hand movements as input commands. I aim to create an engaging and intuitive user experience that does not rely on traditional input devices like keyboards, mice, or game controllers.

Design & User Interaction
Players will interact with the game through various hand gestures. For instance, an open hand gesture will start the game, while making a fist and holding it for a brief moment will pause or resume the game. The game’s mechanics and objectives will be designed around these gestures, ensuring that the player’s physical movements are seamlessly translated into in-game actions.

To detect and interpret these gestures, I will use a hand-tracking library that provides real-time hand position and gesture recognition. The player’s hand movements will be captured through a webcam and processed to identify specific gestures. Based on the detected gestures, the game will execute corresponding actions, such as starting, pausing, or resuming gameplay.

 

Code Design
Gesture Detection Functions: I have implemented functions like detectOpenHandToStart() and detectHands() to detect specific hand gestures. These functions use the hand-tracking library’s predictions to analyze the hand’s position and orientation.

Hand Highlighting: The highlightHand() function visualizes the player’s hand position on the screen, enhancing user feedback and interaction.

Gesture Recognition Algorithms: Functions are OpenHand () and ClosedFist (), which distinguish between different hand gestures by analyzing the distances between hand landmarks. These algorithms are crucial for converting physical gestures into game commands.

let video;
let handpose;
let predictions = [];
let isGamePaused = false;
let fistDetectedTime = 0;
const fistToggleDelay = 2000;

function setup() {
  createCanvas(640, 480);
  video = createCapture(VIDEO);
  video.hide();
  handpose = ml5.handpose(video, modelReady);
  handpose.on('predict', results => {
    predictions = results;
  });
}

function modelReady() {
  console.log("Handpose model ready!");
}

function draw() {
  background(255);
  image(video, 0, 0, width, height);
  detectHands();
}

function detectHands() {
  if (predictions.length > 0) {
    const landmarks = predictions[0].landmarks;
    highlightHand(landmarks, 'green');

    if (isClosedFist(landmarks)) {
      let currentTime = millis();
      if (currentTime - fistDetectedTime > fistToggleDelay) {
        isGamePaused = !isGamePaused;
        console.log(isGamePaused ? "Game Paused" : "Game Resumed");
        fistDetectedTime = currentTime;
      }
    }
  }
}

function highlightHand(landmarks, color) {
  fill(color);
  landmarks.forEach(point => {
    ellipse(point[0], point[1], 10, 10);
  });
}

function isOpenHand(landmarks) {
  let minDist = Infinity;
  for (let i = 4; i <= 20; i += 4) {
    for (let j = i + 4; j <= 20; j += 4) {
      let dist = distanceBetweenPoints(landmarks[i], landmarks[j]);
      if (dist < minDist) {
        minDist = dist;
      }
    }
  }
  return minDist > 50;
}

function isClosedFist(landmarks) {
    let maxDist = 0;
    for (let i = 4; i < landmarks.length - 4; i += 4) {
        let dist = distanceBetweenPoints(landmarks[i], landmarks[i + 4]);
        if (dist > maxDist) {
            maxDist = dist;
        }
    }
    return maxDist < 40; 
}

function distanceBetweenPoints(point1, point2) {
  return Math.sqrt(Math.pow(point2[0] - point1[0], 2) + Math.pow(point2[1] - point1[1], 2) + Math.pow(point2[2] - point1[2], 2));
}

Challenges & Risk Mitigation
The most challenging aspect of this project was developing reliable gesture recognition algorithms that can accurately interpret the player’s intentions from the hand’s position and movement. Misinterpretation of gestures could lead to a frustrating user experience.

To address this challenge, I focused on refining our gesture recognition algorithms (isOpenHand() and isClosedFist()) to improve their accuracy and robustness. I conducted testing with different hand sizes and lighting conditions to ensure the algorithms’ reliability across a wide range of scenarios. Additionally, I implemented visual feedback mechanisms (via highlightHand()) to help players adjust their gestures for better recognition.

Next Steps
In conclusion, this project represents a significant step towards creating more natural and immersive gaming experiences. I aim to explore new possibilities in game design and interaction by leveraging hand gestures as input.

Reflecting on the insights gained from “Computer Vision for Artists and Designers: Pedagogic Tools and Techniques for Novice Programmers” offers a perspective on the intersection between technology, art, and education. This journey through the text has broadened my understanding of computer vision as a technical field and deepened my appreciation for its artistic and pedagogical applications. It’s fascinating to see how computer vision can bridge the tangible world and digital expression, enabling artists and designers to explore new dimensions of creativity. The examples of interactive installations and projects highlighted in the text, such as the LimboTime game or the innovative use of surveillance for artistic expression, showcase the power of computer vision to create engaging and thought-provoking experiences. These projects underscore the potential of computer vision to transform our interaction with the digital world, making technology an integral part of artistic exploration and expression.

Moreover, the text’s emphasis on making computer vision accessible to novice programmers through pedagogic tools and techniques is particularly inspiring. It demystifies a complex field, making it approachable for artists and designers who may not have a strong background in programming. This approach democratizes the field of computer vision and also encourages a more inclusive community of creators who can bring diverse perspectives to the intersection of art and technology. The discussion around optimizing physical conditions to enhance the performance of computer vision algorithms was enlightening, highlighting the intricate dance between the digital and physical realms. As I reflect on the reading, I am left with a profound excitement about the possibilities at the confluence of computer vision, art, and design. This text educates and inspires, pushing the boundaries of what we perceive as possible in digital art and design.

Concept:

Growing up, I used to spend my days playing Nintendo games with my siblings, whether it was a game like Mario Sunshine or even Mario Party. The nostalgia of the games and characters’ associated with Nintendo made me want to create something similar. Therefore, for my midterm project, I decided to create a pac-man like game, in which Mario and Luigi chase after stars. This will need two players to take the role of Mario and Luigi. My idea for this is that it would be some sort of competition to see who can collect the most stars. The whole concept behind this idea is to let users interact with the art directly, which is why I will try to emphasize user-experience and user-centered design as I start to build this project. 

Design:

For this project, I have decided to keep the design simple and clean. The color scheme will be soft but vibrant against the dark background. I am leaning towards going for a “retro” color scheme. I want my game to make users excited to play it, thus, the colors are significant.  For user-interaction, I want the two players to use keys on the keyboard, such as WASD and the arrows, to move the characters up, down, left and right. This will be the main form of interaction for my project. In terms of sound, I want each character to have a distinct sound for when they collect their stars. I also want to potentially have a sound for when the game is over. This will create an immersive and memorable experience for users, enhancing their engagement throughout the game. I might incorporate subtle background music that complements the pace of the game in order to further enrich the overall auditory experience of the users. However, I want to prioritize getting the sound of the characters catching the stars first. Carefully considering both the visual and auditory aspects of the game will allow me to conceptualize the experience I want to create for users, one that is captivating and memorable. 

In order to visualize the concept and design of my game, I drew basic wireframes. These drawings are very basic and overly simplistic but they give me a rough idea of what design and feel I want to showcase to users. 

Sketch of Game Layout: 

Fears and Minimizing Risk:

Reflecting on this project, I think I am most frightened about two things: how I will be able to implement the keyPressed function properly to ensure that each character moves according to their keys and how I would be able to indicate that the game has ended. To tackle these two issues, I am practicing using both sound and the keyPressed functions in different ways to ensure that I understand how to use them in order to be able to manipulate them the way I want for this project. For the keys, I have been studying from slides and practicing using videos from Youtube. By doing this, I will get a better understanding of the function, so I will be able to implement it successfully in my project. In regards to the sound, I am still figuring out how to make it so that a noise indicates the end of the game. I am thinking of potentially putting a timer that would stop all activity once it reaches the end. However, I am still figuring this out but with time and trial and error, I will be able to implement such a feature in my project.

This reading on computer vision was very intriguing and made me want to create interactive artworks as such. In particular, I liked the “Stills from Cheese” installation as it has a humourous approach to it. I also enjoyed reading about the “LimboTime” game as this was a game that I used to always play with friends in which it was more of a physical game, where an actual stick was used as the “horizontal line”, rather than an interactive virtual one.

Moreover, many of the artists and artworks in this reading sounded familiar from a class I previously took (Understanding IM), in which we learned about David Rokeby, Rafael Lozano-Hemmer, Camille Utterback, Romy Achituv, etc. However, what stood out differently this time is the idea of computer vision in specific and how there are two main characters, human and computer. For computer vision, those two players have to exist so that the artwork/installation would be interactive and to really be successful.