Week 7 – Midterm Project

Concept:

Ever wondered what it would feel like to direct a show? Well, you would be glad someone (me!) sweated and cried over code to give you this chance.

For my midterm project, I wanted to create a ‘theater/stage management experience’. This idea stems from our assigned Crawford reading on interactivity. He argued that activities like reading or watching a performance are not actually interactive since a person can only react to the experience and not influence the outcome or the final showcase. I created this experience not to counter his point, but to instead give the user a playful way to interact with the theatrical world (or at least influence how a show turns out) by stepping into the shoes of a stressed director. The experience is made in a game format (mini-games), and I decided to focus on three problems a real director might face backstage: broken spotlights, mixed-up scripts, and a panicked actor. I believe three problems were the optimal choice since it allowed me to work on the user experience/aesthetics while still keeping it challenging (but not too much) for the user.

Sketch:

Link to the experience: https://editor.p5js.org/Leyanah/full/27UhALCdQ

Process:

To make this, I first started with a game plan (as described in my midterm progress post) and worked on the game cover and each individual mini games separately. This helped me work on each section and notice errors early on. It also gave me a sense of confidence to know that at least some parts of my project were up and working. Then, I merged everything into one p5 console. This was the hardest part since I had a lot of debugging to do on so many lines of code (especially removing redundant lines that were crashing the game). In general, I really like my game. I am especially proud of having been able to create different states, which allow the user to navigate through the game, from the backstage to the mini games and eventually to the show. I have also used ‘class’ to make the tears fall randomly at different speeds in my tear mini game and to make my actor move (using a sprite sheet) at the end for the show. Here are the codes:

class Tear {
  constructor(x, y) {
    this.x = x;
    this.y = y;
    this.speed = random(4, 7);
  }
  update() {
    this.y = this.y + this.speed;
  }
  display() {
    fill(120, 190, 255);
    ellipse(this.x, this.y, 18, 26);
    triangle(this.x, this.y - 22, this.x - 8, this.y - 6, this.x + 8, this.y - 6);
  }
}
class Character {
  constructor(img, x, y, h, frames) {
    this.img = img;
    this.x = x;
    this.y = y; //y is where the feet touch the floor
    this.h = h;
    this.frameW = img.width / frames; //width of one picture
    this.frame = 0; //which picture of the sprite sheet is shown
    this.w = this.frameW * h / img.height; //keeps the picture's proportions
  }
  display(brightness) {
    tint(brightness); //used to change the brightness of the picture
    image(this.img, this.x - this.w / 2, this.y - this.h, this.w, this.h, this.frame * this.frameW, 0, this.frameW, this.img.height);
    noTint();
  }
}

One part of my code that I think makes the whole difference is this:

function drawShow() {
  ShowTime = ShowTime + deltaTime / 1000;
  let Results = [LightsFixed, ActorCalm, ScriptReady];

  //the actor on stage - final performance
  if (ActorCalm) {
    ShowActor.frame = floor(frameCount / 8) % 8; //the next picture of the sprite sheet
    ShowActor.display(255);
  }
  else {
    CryingActor.display(255);
  }

  //the stage is dark without working lights
  if (LightsFixed) {
    Darken(60);
    //spotlight beam for the show
    blendMode(ADD);
    fill(60, 48, 18);
    triangle(Width / 2, -20, Width / 2 - 150, LightY, Width / 2 + 150, LightY);
    blendMode(BLEND);
  }
  else Darken(200);

This part of the code is what makes my experience interactive since the show will be affected based on the success or failure of each mini game. For instance, if the ‘tear’ game fails, the actor would appear crying on stage; otherwise, the code displays his spritesheet. Below are images of these two different outcomes.

Similarly, the ‘wires’ game determines whether the show will have good lighting or we’ll just have a dark stage. As for the audio, I decided to have looping audio throughout the game (triggered by the start button and stopped when we reach the ‘show’ stage) and winning/losing soundtracks for the mini games. I used more pronounced winning/ losing soundtracks for the final show to mark a difference from the mini games and give the user a better ‘sense of accomplishment’ when they finish the game (I noted this when I made my friends test the game).

References:

I opted for generated characters and backgrounds to focus on improving the game itself rather than creating everything from scratch, given our two weeks. To generate my characters and props, I used “Pixler.dev” (Link) and Claude for my backgrounds. As for my audio, I sourced all my sound effects and music from “Pixabay” (Link). To help me with coding, I referred to the p5 directory and “The Coding Train” videos on OOP (Link and Link) and back buttons (Link). I also had trouble with the timer bar, which was not matching the time limit. I asked Claude, and I understood that p5 gives a variable called deltaTime, the milliseconds since the last frame. Adding it up counts real time so the bar matches the seconds in each mini game.

Reflection:

In terms of further improvements, I would like to work on making the script game’s influence slightly more consequential. What I mean is that right now, the tears change which actor appears and the wires change how dark the stage is, but the script game only affects the review bubbles. If I had more time, I would give it a visible effect on stage too. For instance, if the script is ready, the actor could deliver his lines with speech bubbles showing the script text. If it isn’t, he could freeze and show question marks or stammer.

Finally, I am grateful to my friends who tested my game. Their feedback helped me judge the ‘flow and feel’ of the game. For instance, I increased the overall timer (especially the tear game timer). I also made the scripts shorter so that the lines can be read faster. Now that I have a finished project, I can admit that I really enjoyed the process. I had lots of fun coming up with the script and the reviews (especially the bad ones) and seeing my ideas come together into one game. I hope people enjoy the game.

Midterm Project

Concept

The idea for this project actually came from a quiet weekend when I watched the Chinese movie Letters to Grandma(《给阿嬷的情书》). The story really hit home for me. It’s about a time before smartphones and social media, when people had to write real letters across oceans, waiting endlessly just to share their feelings with family and loved ones. That deep sense of yearning and patience resonated with me so much that I was like, “I really, really want to build something that brings that letter-writing feeling back.”

In the film, flowers and ocean waves are huge symbols of distance and memory, and I was so set on bringing those exact visuals into my project, along with some traditional Chinese aesthetic touches. I spent a ton of time tweaking the visuals—making the falling petals and moving waves look just right—and honestly, seeing it all come together gave me such a huge sense of achievement!

At first, I just had on-screen typing, but then I realized just typing wasn’t quite fun or meaningful enough. I really wanted it to feel real. After getting some great advice from my professor, I added a feature where you can actually export what you write into a real Word document (.docx). That way, it’s not just a digital demo—you get to save your words as a real letter, just like people used to do back in the day.

How this was made

The Home Page, Layered Aesthetics & Swaying Physics

For the home page, Instead of relying on a flat, static layout, I spent a lot of time crafting a rich 3D depth by splicing three separate flower images (flower.png, flower2.png, etc.) together so that the hanging stationery looks like it’s literally growing out from deep inside the blossoms. However, the most challenging technical hurdle here was simulating realistic pendulum motion for the hanging ropes and letter cards. I solved this by using trigonometric sine waves (sin()) tied to frameCount,(learn from YouTube video)generating independent speeds, phase offsets, and curved paths for each individual rope:

function createRopes() {
  let lengths = [250, 340, 270, 395, 305, 370, 260, 420, 285, 355];
  let positions = [-115, -86, -57, -29, -10, 10, 29, 57, 86, 115];
  ropes = [];
  for (let i = 0; i < 10; i++) {
    ropes.push({ 
      offsetX: positions[i], length: lengths[i], 
      swingSpeed: random(0.008, 0.02), phase: random(TWO_PI), 
      swingAmount: random(8, 14), rotationAmount: random(0.04, 0.08) 
    });
  }
}

function drawRopes() {
  let ropeStartY = getRopeStartY();
  for (let rope of ropes) {
    let swing = sin(frameCount * rope.swingSpeed + rope.phase);
    let sway = swing * rope.swingAmount;
    beginShape();
    for (let j = 0; j <= 24; j++) {
      let t = j / 24;
      let y = ropeStartY + rope.length * t;
      let curve = (t + sin(t * PI) * 0.6) * sway;
      vertex(baseX + curve, y);
    }
    endShape();
  }
}

Interactive Writing, Text Scrolling

When users click a letter to transition into the writing view, I built a virtual keyboard to increase interactivity .Every time a key is pressed.it triggers a audio feedback using singleKey.wav and lights up a temporary press highlight (flashKey) to mimic a real  keyboard:

function drawKeyboard() {
  let rows = ["QWERTYUIOP", "ASDFGHJKL", "ZXCVBNM"];
  let keyW = 32, keyH = 34, gap = 4, startY = 475;
  for (let r = 0; r < rows.length; r++) {
    let letters = rows[r];
    let totalWidth = letters.length * keyW + (letters.length - 1) * gap;
    let startX = (width - totalWidth) / 2;
    for (let i = 0; i < letters.length; i++) {
      let letterKey = letters[i];
      let x = startX + i * (keyW + gap), y = startY + r * (keyH + gap);
      let isPressed = flashKey === letterKey && frameCount < flashUntilFrame;
      if (isPressed) { fill(0); stroke(0); rect(x, y, keyW, keyH, 4); fill(255); }
      else { fill(255, 160); stroke(0, 120); rect(x, y, keyW, keyH, 4); fill(0, 200); }
      noStroke(); text(letterKey, x + keyW / 2, y + keyH / 2);
    }
  }
}
function drawTypedText() {
  push();
  textFont("cursive"); textSize(20); fill(50, 50, 50);
  let lines = wrapText(typedText, maxCharsPerLine).slice(-9);
  for (let i = 0; i < lines.length; i++) {
    text(lines[i], textStartX, textStartY + i * textLineHeight);
  }
  if (typingActive && frameCount % 60 < 30 && lines.length > 0) {
    let lastLine = lines[lines.length - 1];
    let cursorX = textStartX + textWidth(lastLine);
    let cursorY = textStartY + (lines.length - 1) * textLineHeight;
    line(cursorX + 2, cursorY + 2, cursorX + 2, cursorY + 20);
  }
  pop();
}

Ocean Waves & Practical Word Export

To transition out of the writing screen, I made a  design use multi-layered  ocean waves (drawWave()) . By stacking multiple wave layers with varying frequencies, semi-transparent water stream lines, and glowing foam highlights, the ocean rises to cover the canvas:

function drawWave() {
  push(); noStroke(); wavePhase += waveSpeed;
  let baseY = height - calmWaveHeight, riseHeight = 0;
  if (waveSending) {
    let elapsed = frameCount - waveSendStartFrame;
    let t = constrain(elapsed / waveSendDuration, 0, 1);
    let maxRise = height * 0.52; 
    riseHeight = t < 0.46 ? maxRise * easeWave(t / 0.46) : t < 0.64 ? maxRise : maxRise * (1 - easeWave((t - 0.64) / 0.36));
  }
  let backY = baseY - riseHeight;
  drawWaveLayer(backY + 32, 27, 0.012, 0, waveColors[0], 0.7);
  drawWaveLayer(backY, 8, 0.040, 6.0, waveColors[4], 1.0);
  drawInternalStreams(backY);
  pop();
}

Once the wave completely clears the screen, it triggers the document saving process. To be honest, I didn’t write the Word export logic entirely from scratch—I did some research online to figure out how web apps export files, discovered the docxlibrary, and then pieced together and adapted the code to fit my project’s data flow

async function downloadWordDocument(content) {
  const { Document, Paragraph, TextRun, Packer } = docx;
  const paragraphs = content.split(/\r?\n/).map(line => new Paragraph({
    children: [new TextRun({ text: line || " ", font: "Georgia", size: 24 })],
    spacing: { after: 120 }
  }));
  const wordDocument = new Document({ sections: [{ children: paragraphs }] });
  const blob = await Packer.toBlob(wordDocument);
  // Trigger browser download...
}

Improvement and problems

One of the biggest hurdles I ran into during development was handling mouse interactions on the home screen because the hanging letter cards are constantly swinging, rotating, I had to overcome this challenge by researching and implementing coordinate transformations to accurately detect mouse hovers on those moving cards.
However, I still have other unresolved issues, such as text layout : setting a fixed character limit per line continues to cause formatting bugs where wide letters like ‘W’ overflow the page while narrow characters wrap way too early, and I am still looking for a robust way to fix this line-wrapping problem.
I really wanted to add background music , but I struggled with it for an entire day—I kept running into errors every time I tried to upload my MP3 file, likely because the file size was too large, so I ultimately had to give up on the background music .

Reference

YouTube videos:

https://youtu.be/N633bLi_YCw?si=SZS-L-PLuXJI5_Ag https://youtu.be/o9sgjuh-CBM?si=0U7rIlb-7wCXG49J https://youtu.be/qRnUBiTJ66Y?si=zR0N2T6e4cUjKULZ

AI used for debugging errors, providing ideas for the text-wrapping logic, and helping implement the transformation math needed for mouse hit-testing on the letter.

 

 

Week 7 – MIDTERM Project

 

link: https://editor.p5js.org/am15643/full/W4lY8cfjO

Concept
For this project, I was really sure I wanted to make a generative art, but also a kind of generative art that is more personal. I am always told, that my room and my table really reflect my personality and that it looks like how people imagine it to look like, which is why I decided to make this about my table in the end! I just really wanted to challenge myself with what we have learned, like about the CSV files or OOP, and somehow include them all in a project that is fun but still doesn’t feel too much, and is not too confusing. I hope I managed to achieve that!

I had this concept since beginning actually, I think I have wanted to make this since we first started thinking about ideas, so I feel really proud that it managed to stay with me until the end. I wanted the code to have a start, a main, and tree subpages, although at the beginning I didn’t know exactly what I wanted these subpages to do, or what I would want to have in them, they just happened during the development of the whole project.

The Code
Overall, I had a lot of problems with the code, that it worked differently than I wanted it to, but in the end it got together. The way my project works, is that the user starts with  an instructions page, where I tried to create a flower from my culture, which is Hungarian, to put a little bit of myself even here. This wasn’T necessarily hard, it just took a very long time for me to do the measurements right.

Reference for the flower
//Hungarian-style flower
  //Red base
    fill('#FF2400');
    noStroke();
    circle(width / 2, 90, 130);
    circle(width / 2 - 70, 120, 70);
    circle(width / 2 + 70, 120, 70);
    circle(width / 2 - 50, 170, 50);
    circle(width / 2 + 50, 170, 50);
    circle(width / 2 - 20, 195, 40);
    circle(width / 2 + 20, 195, 40);
    rect(width / 2 - 50, 140, 95, 54);
  
  //Middle 'design'
    fill('#FFBF00');
    circle(width / 2, 70, 90);
    fill('#008000');
    ellipse(width / 2, 45, 20, 40);
  
  //Leaves outside, the green lines and blue dots
    stroke('#008000');
    strokeWeight(4);
    line(width / 2 - 70, 80, width / 2 - 150, 20);
    line(width / 2 + 70, 80, width / 2 + 150, 20);
  
    line(width / 2 - 70, 80, width / 2 - 130, 60);
    line(width / 2 + 70, 80, width / 2 + 130, 60);
    
    line(width / 2 - 70, 80, width / 2 - 100, 30);
    line(width / 2 + 70, 80, width / 2 + 100, 30);

    noStroke();
    fill('#0047AB');
    circle(width / 2 - 150, 20, 20);
    circle(width / 2 + 150, 20, 20);
  
    circle(width / 2 - 130, 60, 20);
    circle(width / 2 + 130, 60, 20);
  
    circle( width / 2 - 100, 30, 20);
    circle( width / 2 + 100, 30, 20);

Then the user gets to my main page where they can interact with my books, my headphones, my pictures and my lamp. I finally managed to change the AI background photo and I am really proud of it, as I made the end result in Canva (I took a ScreenShot of the process but Canva makes them blurry).

Then once the user is “at my desk” they have 4 interactable objects to choose from, where they are able to learn something about me through different things like what books I like, what music I listen to or photos.

The headphones and books pages I “designed” myself in p5, so they don’t look as aesthetic as the main desk page, but I had fun playing with the different shapes and colours. They are very minimalistic I think, but I really like it, especially the little game where you can guess what my favourite book is. I think this books part was the most fun to make out of all. The program just checks which book your mouse is on, and gives you back your result if you press on it. This is something I am really proud of.

//Guessing my favourite book game
    if(mouseIsPressed){
       if (mouseX > whiteBookX && mouseX < whiteBookX + whiteBookW && 
        mouseY > whiteBookY && mouseY < whiteBookY + whiteBookH) {
          push();
          textFont('Caveat');
          textStyle(NORMAL);
          textSize(30);
          text('❌ So close!', 120, 350);
          pop();
    } else if (mouseX > yellowBookX && mouseX < yellowBookX + yellowBookW && 
        mouseY > yellowBookY && mouseY < yellowBookY + yellowBookH) {
          push();
          textFont('Caveat');
          textStyle(NORMAL);
          textSize(30);
          text('❌ Almost!', 120, 350);
          pop();
    } else if (mouseX > redBookX && mouseX < redBookX + redBookW && 
        mouseY > redBookY && mouseY < redBookY + redBookH){
          push();
          textFont('Caveat');
          textStyle(NORMAL);
          textSize(50);
          text('🎉 👑', 120, 350);
          pop();
    } else if (mouseX > greenBookX && mouseX < greenBookX + greenBookW && 
        mouseY > greenBookY && mouseY < greenBookY + greenBookH) {
         push();
          textFont('Caveat');
          textStyle(NORMAL);
          textSize(30);
          text('❌ Close!', 120, 350);
          pop();
    } else if (mouseX > blueBookX && mouseX < blueBookX + blueBookW && 
        mouseY > blueBookY && mouseY < blueBookY + blueBookH) {
           push();
          textFont('Caveat');
          textStyle(NORMAL);
          textSize(30);
          text('❌ Try another!', 120, 350);
          pop();
        }
    }

Another part of my project that I am really proud of is weirdly the postcard, or the photos part of it. For the photos page I also used my actual photos: I laid out my polaroids on my floor and took a picture of them. Originally I really wanted the photos to be on a carousel, but my program was already full of images, and was hard for it to load, so in the end I decided upon one: of my brothers and I. I feel like having two brothers is a really defining and important part of my life, I really love them so so much, and that is why I feel so proud of this part, not only the code itself but also the idea.

The code was also something I had struggle with, but that’s what makes me feel so proud now. I tried the translate an rotate thing, and also for the image to “flip”. In reality it just disappears and a text appears when clicked on it, but I still am really proud of it:

//Design of subpage, what is on it
postcardsPage(){
    background(postBG);
  
  //White card behind photo
    push();
    drawingContext.shadowOffsetX = 4;
    drawingContext.shadowOffsetY = 4;
    drawingContext.shadowBlur = 4;
    drawingContext.shadowColor = color('black');
    noStroke();
    rectMode(CENTER);
    translate(width / 2, height /2 );
    rotate(-1.7);
    fill('white');
    rect(0, 0, 250, 360, 10);
    pop();

  //put both photo and text in the white box
  push();
  translate(width / 2, height / 2);
  rotate(-0.13);

  //"flip" image, make it disappear and show a text instead
  if (this.isFlipped) {
    textSize(24);
    textStyle(NORMAL);
    textAlign(CENTER);
    fill(50);
    textFont('Edu QLD Hand')
    noStroke();
    text('These are my brothers...', 0, -30);
    text('They are awesome', 0, 0);
    text('But do not tell them I said this', 0, 30);
    } else {
      //photo of me and my brothers
        imageMode(CENTER);
        image(siblingImg, 0, 0, 320, 220);
    }
    pop();

  //Click sign
    image(click, 350, 330);
  }

Speaking of which, a good design that I think I implemented was to call all the images, the background music and the csv file in a Promise rather than calling them all separately. The code still needs some time to start, but it definitely works a lot faster than it did when I just called them separately and not as a block:

async function setup() {
    createCanvas(500, 600);
  //loading all the images, music, strings
    [
      backgroundImg,
      lampImgOff,
      lampImgOn,
      headphonesImg,
      booksImg,
      postcardImg,
      backTrackButtonImg,
      postBG,
      playlistSongs,
      siblingImg,
      click,
      bgMusic,
    ] = await Promise.all([
        loadImage('desk.png'),
        loadImage('lampOff.png'),
        loadImage('lampOn.png'),
        loadImage('headphones.png'),
        loadImage('books.png'),
        loadImage('postcard.png'),
        loadImage('backtrack.png'),
        loadImage('postBg.jpeg'),
        loadStrings('playlist.csv'),
        loadImage('siblingImg.jpeg'),
        loadImage('click.png'),
        loadSound('bgMusic.mp3'),
      
    ]);

Future Improvements
One thing I could not fix, was the shuffle button in full screen mode. For some reason, no matter what I did the button just wouldn’t stay in the same place, and eventually I gave up on it. I really want to improve on that in the future.

Originally, I wanted the sound to only start when the user is in the MAIN page, and for it to stop at the START page. I tried multiple options on how to make that work but none of it did. One just completely viped away the music, the other did only start it once the user was in the main page, but then it didn’t stop when I went back., so I just decided to keep it throughout in a loop. I thing that could be something to work on.

I feel like I could definitely work on working faster. This is not something specific in my code, its just that this little interactive thing took me almost 2 weeks of heavy coding and a lot of hours of work. I think I need to work on not overcomplicating things, but also learning how to work faster, because I feel like I spent too much time on parts of my code that didn’t necessarily require that much attention.  This might get better as I get more and more familiar and confident with Javascript, but for now I had to google a lot of things, I took a long time to understand and then to implement them.

 

References

Basic design of multiple pages (inspiration, help)
https://editor.p5js.org/ltorrent/sketches/3Y31raQKU
https://www.youtube.com/watch?v=Yk18ZKvXBj4 (for dividing the scripts)
https://www.youtube.com/watch?v=Pn1g1wjxl_0&list=PLRqwX-V7Uu6aFcVjlDAkkGIixw70s7jpW (sound)

Resources for colour codes, emojies, fonts
https://htmlcolorcodes.com/
https://emojipedia.org
https://fonts.google.com/

Help with the backtrack button/understand the priciple
https://editor.p5js.org/Ramikd/sketches/8Czoi6JqB

Music from freesound, author doesn’t require credit, but I still want to give them:
VS Button Click 05.mp3 by Vilkas_Sound — https://freesound.org/s/707042/ — License: Attribution 4.0

Clicking sounf from Freesound
VS Button Click 04.mp3 by Vilkas_Sound — https://freesound.org/s/707041/ — License: Attribution 4.0

 

Midterm Project – Interactive Gallery

Concept

I got inspired by another interactive work that was designed as a cozy coffee shop, where the user could walk in and interact with a few things. I thought, why not do something similar using my own artwork? This is how I came up with the idea of creating an interactive art gallery. There would be multiple drawings and paintings of mine, each with its own interaction and movement controlled by the mouse. It’s like stepping into the story of each painting and showing how a simple interaction can change its mood. I also think this concept could draw more people toward art, especially on a larger scale, because it offers a more engaging experience than simply looking at static artwork.

A part of the project I’m really proud of is the flower and teddy bear paintings because I really like how the animations turned out, and they were also the most fun to work on! I wanted to add animations to the other paintings as well, but I realized that some ideas would be too time-consuming, so I settled for simpler methods. There were also some ideas I had to scrap because they didn’t work as I expected, even after multiple adjustments. I just couldn’t achieve the natural and realistic feeling I was aiming for.

To make the gallery feel less empty, I added paintings as decorations on the side walls (credits to my sister for one of the paintings). However, I couldn’t figure out how to tilt them to match the perspective of the walls, so I decided to edit them using another tool and turn the entire gallery into an image. I also chose to make the paintings look pixelated from a distance as an artistic choice. Since the background still felt a bit empty, I added a decorative pattern, using YouTube videos as references to help me create it.

To make the code more efficient, I created a class for the four paintings and gave each one its own coordinates and dimensions. This made it easier to detect where the mouse was clicked and change the game state to display the corresponding painting. Therefore, I didn’t have to repeat the if statement in mousePressed function.

function mousePressed() {
  // CHANGE STATE BASED ON WHERE THE MOUSE IS CLICKED //
  if (gameState == 'gallery') {
    for (let p of paintings) {
      if (mouseX > p.x && mouseX < p.x + p.w &&
          mouseY > p.y && mouseY < p.y + p.h) {
        gameState = p.name;
        break;
      }
    }

 

For the interactions, I settled on using the mouse. Some paintings respond to the mouse’s Y-axis, while others require the user to click on them. I chose clicking for certain interactions because I feel like people instinctively click on things to see if they do anything. I wanted the interactions to feel simple, so users could discover them without needing complicated instructions.

References and AI Usage

I mainly used references provided on the course website and the p5.js reference page to understand how to implement different features. I also used AI for some of the trickier parts of the project that I couldn’t find a solution to on my own, such as creating the flickering flame effect for the candle painting. AI helped me understand how to approach the perlin noise effect and implement it in my code, it suggested methods to use and explained each of them. It also helped me in the color transition in the sunset painting. Other than that, I relied on the knowledge I gained from previous assignments.

let darkness = map(sunY, 150, 450, 0, 80);
  let transition = map(sunY, 250, 450, 0, 1);
  
 // TRANSITION THE COLOR FROM ORANGE TO PURPLE DEPENDING ON THE SUN Y AXIS //
  let skyColor = lerpColor(orange, purple, transition);
  skyColor.setAlpha(darkness);
  fill(skyColor);
  rect(250, 25, 500, 650);

Reflection

I’m really proud of this project, especially because I got to showcase my own art. I think the idea has a lot of potential and could be executed in various ways. Perhaps I dreamed bigger than my current skills allowed, but I see that as an opportunity to learn. If I had more time, I would have put more effort into developing the interactions and refining the artwork. In the future, as I gain more coding experience, I’d like to explore this concept on a larger scale and make the gallery more detailed.

 

Week 5 – Midterm Progress (An Audio-Controlled Walking Game)

The Concept

For my midterm project, I want to make a walking game controlled by sounds the player records. Instead of assigning fixed voice commands, the game will let each player choose a short sound for Up, Down, Left, and Right. A player could use spoken words, clicks, whistles, or other distinct sounds. I like the idea that the controls become personal to the player, and that learning to move is part of the experience.

User Interaction

The player will first enable their microphone and record one sound for each direction. Once all four sounds are saved, they can press “Start listening” and play using the microphone. Each recognized sound moves the character for a chosen number of seconds. The player can adjust that duration with a slider and re-record a sound if it is difficult to recognize.

Movement on the two axes works independently. For example, if the player makes the Up sound and then makes the Left sound while Up is still active, the character moves diagonally. The arrow keys also work, which gives me a way to test movement separately from the microphone.

Code and Design Progress

I started with a p5.js prototype of a walking character. It loads a sprite sheet, separates it into animation frames, and displays the correct row for each direction. The character animates while moving and stays within the canvas.

I then added the microphone setup, recording buttons, sound recognition, and timed movement. The code samples a sound’s frequency profile, divides it into three parts over time, and compares it with the four recorded signatures. It rejects sounds when the match is too uncertain. I also changed the movement code to keep separate horizontal and vertical timers, which allows two directions to remain active together.

Most Frightening Part

The most uncertain part is sound recognition. A person will not make a sound exactly the same way twice, and background noise could cause a false command. Rapid commands are another challenge: the program needs to notice where one sound ends and the next begins.

To reduce this risk, I built the recognition system before adding more game features. The prototype normalizes the frequency profile so a louder or quieter version of a sound can still match. It waits for a short silence to finish recording a command and rejects ambiguous matches. This gives me a working system to test with real sounds, although I still need to find out how reliably it performs in different rooms and with different players.

Next Steps

My next step is to test several sets of four sounds and record which commands are missed or confused. I can use those results to adjust the detection thresholds and the pause between commands. After the controls feel reliable, I want to design a space for the character to navigate so that the audio movement has a clear goal.

Week 5 – Reading Response

The artwork I kept thinking about after reading Levin’s essay was Cheese. The performers had to hold a smile while a computer measured it, and an alarm went off when their smiles dropped below a certain level. I can imagine laughing at that at first, then becoming very aware of my own face. It made me wonder when an artwork that responds to you starts to feel like it is judging you.

That example also made the difference between human and computer vision clearer to me. I might look at someone’s smile and wonder whether they are tired or joking. The computer needs a way to turn what its camera captures into something it can measure. Even finding a person in the image takes work. Frame differencing looks for changes between video frames, so someone standing still could disappear from its view. Background subtraction finds what differs from an empty scene, while brightness thresholding works when a person or object is clearly lighter or darker than its surroundings. I was surprised by how much the artist can help simply by changing the lighting or background.

I like Krueger’s idea that “the entire human body ought to have a role in our interactions with computers.” It opens up more playful possibilities than clicking a mouse. I would be curious to make something that reacts when a visitor turns toward it or moves away, without trying to decide what their expression means. Tracking can make a person feel involved in an artwork, but Cheese reminded me how easily that same attention can feel like surveillance. I think visitors should be able to tell when an artwork is watching them and choose whether to take part.

Midterm Progress – Week 5

The Concept
I want to create a rhythm platform game where the user has to hit a button for each beat of the song. If the user fails to do so, they will lose and will have to repeat the level.

The game will be something like this, where players must hit a button and it’ll perform an action, maybe the action is to jump, hit, duck, etc.

The player needs to be listening to the song in order to know what to do. But the game will also have visual cues so the player isn’t entirely reliant on just guessing the beat.

Design
I want the game to have a bit of a retro feel, almost like a musical Mario game. The game will consist of pixel art and will feel almost nostalgic. I want to highlight the style of gaming that popularized platform type games.

The Most Frightening Part
I’m mostly worried about making the key the player presses in line with each beat of the song. It being a platformer minimizes the risk, but how big is the margin of error for the user? Additionally, how can I make it so that the player needs to do an input at specific beats of the song.

Minimizing the Risk
I did my best to work with sound in p5js and tried to make it so that the background color changed during the beat of the song.

song.addCue(2, changeBackground, 'palevioletred');
song.addCue(3, changeBackground, 'mediumvioletred');
song.addCue(4.2, changeBackground, 'pink');
song.addCue(6, changeBackground, 'orchid');
song.addCue(8, changeBackground, 'skyblue');

I used addCue to change the background during certain parts of the song. I probably could have put it into the loop but I first wanted to just highlight parts where I heard a beat and put the changeBackground function there. I also think it’ll be funner if the beats aren’t entirely consistent, as that would mean the levels will be harder.

 

Week 5 – Reading Response

Computer vision is different from human vision because computers do not actually “see” images the way humans do. A computer sees an image as pixels and needs algorithms to figure out what those pixels mean. Humans can understand objects, movement, and situations much more naturally, while computer vision usually needs specific conditions and instructions to recognize something.

There are different techniques we can use to help a computer see or track what we are interested in. Some examples are frame differencing, background subtraction, and brightness thresholding. Frame differencing can detect movement by comparing two frames, while background subtraction can separate people or objects from the background. Brightness thresholding can also help detect objects when there is a strong difference in brightness between them and their surroundings.

Computer vision’s ability to track people also connects to surveillance, which can make interactive art more complicated. I think it can be interesting because artists can use tracking to make an artwork react to people, but it also raises questions about privacy and how much people are being watched without realizing it. One artwork I found interesting was Sorting Daemon by David Rokeby. I like it because there is a lot happening in the composite image. The system watches people outside, detects them, separates their images from the background and organizes parts of them by color and size. I think it is interesting because it turns something that feels like surveillance into an artistic image while also making us think about how technology can automatically analyze and categorize people.

Week 5 Midterm progress

For my midterm, I am planning to create an interactive game called “LeBron James Simulator.” The idea is to develop a humorous basketball experience where the player uses their webcam to complete a series of basketball challenges while gaining points based on how their performance is similar to LeBron James. The game will take advantage of computer vision to track the player’s hand in order to complete the challenges.

The experience will start with a starting screen which will instruct the player to stand in front of the webcam to start the game. I am currently planning a few challenges which they will have to go through, such as trying to shoot a ball into a hoop and defense. The player will gain points on their performance, which will count toward a final ‘LeBron Rating’ shown once the game is completed. After players see their rating, they will get a chance to restart the game to try redeeming their legacy.

For the code, I plan to separate each step into states. The start screen, the basketball challenges, and finally the ending screen. I will also use object-oriented programming to create a basketball class that will store their properties such as shape, size and speed.with methods to check interactivity with the hoop and player.

The part of the project that i was most uncertain about was the computer vision and tracking. I needed to perfect this part as it is the main backbone of my experience. Fortunately i used the ML5.JS HandPose liberary to create a small prototype which suceesfuly worked in tracking my hand. The prototype was able to access my webcam and used the x and y to detect my fingertip to draw a small red circle.

Since my test was sucessful, my next step would now be figuring out how I can create the interaction between the player and the basketball.

Week 5 – Reading Response

After reading and understanding “Computer Vision for Artists and Designers” I found it particularly interesting how computer vision is able to turn an individual’s physical movements and environment into a method of interactive art through digital media. It was definitely enjoyable to be able to see examples of interactive artworks that were shown in the article as they revealed that a camera does not simply have to be used to record someone or something, instead it can be used as a meaningful tool to make an artwork responsive. The LimboTime project also stood out to me because it showed me that even beginner programmers can create an interactive game using simple computer vision techniques. A key topic that was mentioned about the physical environment was also interesting as I was previously unaware of how elements such as lighting, contrast, and background can influence a computer’s ability to detect a person. 

Computer vision differs from human vision as humans are able to naturally understand what they are seeing in front of them, whereas computers can only see an image as a collection of pixels. Therefore, a computer tends to require several specific instructions and codes to recognize peculiar things in images.Computer vision also relies heavily on the conditions of the surrounding environment, whereas humans are generally able to distinguish an object from its surroundings more easily.

There are particular techniques that have been mentioned in the text which are used to help the computer in tracking and seeing what we are interested in. Some examples that are mentioned include frame differencing which compares two video frames in order to detect movement. Background subtraction which is also similar but compares the current image to an image of the empty background in order to identify the difference. Brightness thresholding is another technique which is mentioned which allows the computer to identify objects based on whether they are lighter or darker than the surrounding threshold. Techniques such as finding the lightest or darkest pixel can also be used to identify specific objects. 

I think computer vision’s tracking and surveillance approach can have both a positive and negative effect on interactive art. When looking at the positives, this approach invites artists to create meaningful experiences where their artworks can respond directly to the viewer’s body, movement, or facial expressions. This therefore invites the audience to make their own interpretations of the artwork through personal experience and involvement. However, one of the disadvantages that particularly stood out to me when reading the text is the concerns regarding privacy and surveillance as this technology can be used to monitor individuals without their knowledge or consent. This therefore reveals that computer vision can be both a useful tool for creating interaction and also invites artists to question how technology can observe and collect information about us.