Category: F2023 – Shiloh

CONCEPT:

This midterm project was honed to be a simplistic learning experience, where users get to learn new words related to household, in the Moroccan dialect through sound and interactivity. It is a project that holds to my heart because it represents a part of my culture and my dialect. I can’t be happier to share it, now that it’s done!

The idea turned out exactly how it was imagined and planned which makes me very proud of the way I’ve come from the first assignments which were purely trial and error. After this assignment, I feel like I grasp more what should be done, how and when. From brainstorming to design to implementation, it went smoothly.

I present to you the outcome of my midterm: Merhba, Alf Merhba (Welcome a Thousand Welcome!):

Link to the full screen sketch : https://editor.p5js.org/aa8915/full/75GdtC-n2

HOW DOES IT WORK?

-Home Screen:  Click on audio graphic for the pronunciation of the title (this sentence is used to welcome people to your space). Click on PLAY to start the experience. -Instructions: You’ll be presented with a screen containing the idea of the experience, as well as how to proceed. Click on left arrow to go back to the home screen, or right arrow to proceed.

-Inside the house: Now you have six spaces to choose from (room 1, room 2, bathroom, kitchen, dining room and lounge). Click on any space to discover the pronunciation of Moroccan words for objects related to that space.

-In any space x: Click on any object and discover how it sounds in this dialect! Use the arrow (on top or in the bottom), to go back to the house.

PROCESS & DIFFICULTIES FACED:

1. Brainstorming: I didn’t spend a lot of time in this step because as soon as I read “midterm … experience”, I knew I wanted to do something related to something I identify with, but at the same time fun and useful for users to spend few minutes of their time interacting with. It took a few days to choose this concept among three other interesting concepts.
2. Design: I was slowed down a bit by this step because I had many ideas for my concept that I was struggling to imagine in a visual format. I knew from the beginning that I wanted the setting to be a house, but didn’t know if I should have a single space hand drawn with interactive elements or something else. Many thanks to my friend Nourhane for helping me decide on a vector based image with multiple spaces, to be made interactive. Then, I designed the home screen and instructions screen by myself to align with the color palette of the house.
3. Sound Recordings: My favorite part! I had so much fun recording the pronunciation of the words. I had to re-do many of the recordings multiple times because they either sounded weird, mispronounced or noisy. This part took about half a day.
4. Implementation: Implementing all the fun ideas was always my least fun part! Luckily enough this time, I enjoyed this part where I got to try different ways to make the project work. I was first confused with layering because I had 9 layers to include, but later on managed it by creating a SceneManager class.

The hardest part in the implementation of this project was the clickable object. Since my objects were a part of the image itself, I could not figure out any other way but using conditional statements for each and every object (about 60 objects). I measured the position of each object in order to make it clickable and add sound to it. The process to do this was not complicated, but very monotonous, but I’m glad it works. The only downside about this method is that only a specific part of each object is clickable. As a result, if a part of the sofa is hidden by a table, only the clear part is clickable. Depends on where the user decides to click, some parts may not work if clicked. Now that I think about it, I think it would have been a better idea to have vectors of the objects as separate images as well as a separate empty space. This way I could have created a Clicked Object class, have instances for each object image and make them clickable by using a conditional whenever  X object is selected.

HIGHLIGHT

 

There isn’t a part I’m specifically proud of this time, because I’m proud that the whole code worked without errors. I would say I appreciate all the efforts and time I put into measuring the position of each and every object.

if (sceneManager.onRoom2){
     //to go back to house
     if(mouseX>=0 && mouseX<=107){
       if (mouseY>=0 && mouseY<=49){
         clear();
    sceneManager.onWScreen = false;
    sceneManager.onHouse = true;
    sceneManager.onRoom = false;
    sceneManager.onRoom2 = false;
    sceneManager.onKitchen = false;
    sceneManager.onBathroom = false;
    sceneManager.onDiningRoom = false;
    sceneManager.onInstructions = false;
    sceneManager.onLounge = false;
    return;   
       }
     }
     //mirror object
     if (mouseX>=0 && mouseX<=206){
       if(mouseY>=61 && mouseY <= 274){
         mraya.play();
       }
     }
     //picture object
     if (mouseX>=259 && mouseX<= 368){
       if (mouseY>=128 && mouseY<=217){
         tswera.play();
       }
     }
     //painting object
     if (mouseX>=421 && mouseX<= 757){
     if (mouseY >= 87 && mouseY<= 251)
       {
         tablo.play();
       }
     }
     //bed object
     if(mouseX>=390 && mouseX<=795){
       if(mouseY>=341 && mouseY <= 500){
         namousiya.play();
       }
     }
     //desk object
     if(mouseX>=0 && mouseX<=205){
       if(mouseY>=390 && mouseY<=600){
         comoda.play();
       }
     }
     //lamp object
     if(mouseX>=273 && mouseX<=324){
       if(mouseY>=347 && mouseY<=465){
         bola.play();
       }
     }
     //lamp 2 object
     if(mouseX>=872 && mouseX<=921){
       if(mouseY>=345 && mouseY<=467){
         bola.play();
       }
     }
     //table object
     if(mouseX>=223 && mouseX<=377){
       if (mouseY >= 472 && mouseY <= 600){
         tabla.play();
       }
     }
     //table 2 object
     if(mouseX>=810 && mouseX<=945){
       if(mouseY>=493 && mouseY<=600){
         tabla.play();
       }
     }
     //plant object
     if (mouseX>=0 && mouseX<=86){
       if(mouseY>=287 && mouseY<=383){
         nebta.play();
       }
     }
     //vase object
     if(mouseX>=815 && mouseX<=861){
       if(mouseY>=336 && mouseY<=465){
         vase.play();
       }
     }
     //clock object
     if(mouseX>=833 && mouseX<=938){
       if(mouseY>=124 && mouseY<=224){
         magana.play();
       }
     }
   
   }

REFLECTION:

One element of the project that is missing is text. I could not figure out how to make the text for each object pop in an aesthetic way without creating a mess, so I decided to give up on it. However, I still want to improve the project even after turning it in, so I will work on finding an efficient and visually pleasing text to incorporate text.

Overall, I’m happy of the result of this project, I hope the users will also have a fun (and bug-free) experience with it!

Link to my sketch: https://editor.p5js.org/is2587/full/Bn466sW2p

Concept

My midterm project began as a whimsical idea: imagine a disco party where your favorite cartoon characters hit the dance floor. But this idea seemed too simple and I could not figure out how to make it more interactive and engaging for the user. As I began coding, I stumbled upon Dan Shiffman’s Teachable Machine playlist on YouTube. His playlist sparked a new inspiration – why not infuse ML into my project as well? Plus, a repeated suggestion I got from my friends regarding making the project more engaging was to somehow make the users mimic the dancing characters movement.

This fusion of disco vibes and machine learning transformed my simple “disco experience” into an engaging game. Now, users don’t just watch the dance moves; they become the stars of the show, mimicking poses detected by the webcam. To enhance the scene, I added dancing characters, some spotlights, and floor highlights.

Sketch

Problems/WHAT IM PROUD OF

Since working with a machine learning model was a new experience for me, I faced quite a lot of challenges throughout the coding process. One significant hurdle was integrating my trained machine learning model to recognize user poses. I had a hard time figuring out how I can increment the users score according to the poses detected by the model.

function classifyVideo() { 
classifier.classify(video, gotResults); 
}

function gotResults(error, results) {
  if (error) {
    console.error(error);
    return;
  }
  // Store the label and classify again
  label = results[0].label;
  classifyVideo();
}

function updateScore() {
  if (currentImage == clappingImage && label == "Clapping") {
    score += 1;
  } else if (currentImage == dabbingImage && label == "Dabbing") {
    score += 1;
  } else if (currentImage == handsUpImage && label == "Holding hands up") {
    score += 1;
  } else if (currentImage == armRollImage && label == "Rolling arms") {
    score += 1;
  }
}

Here’s how it works: when the machine learning model processes the user’s dance moves, it returns a label indicating the detected pose, like “Clapping” or “Dabbing.”

Then, the updateScore function comes into play. It’s responsible for comparing the detected pose to the pose displayed on the screen (like clapping or dabbing images) and, if they match, incrementing the user’s score.

Another puzzle was dealing with user stillness. Initially, even if users stood frozen, the machine learning model would guess a dance move. This unexpected stillness would inadvertently increase the user’s score. The solution I came up with was to train my model to identify moments of “no movement,” ensuring it could distinguish between action and inaction.

Since I could not find the right poses online that were also consistent with the overall aesthetics of the game, I decided to sketch the different poses on my ipad and integrated that into my sketch which was another aspect I really enjoyed working on.

What I like the most is how little features such as the dancing characters or the music choice really add to the whole experience. Before you enter the disco, I added a special touch to the music. At the beginning, it’s like you’re standing right outside a disco, and you can hear the muffled sounds of music from inside. It sets the stage and builds anticipation. But as soon as you enter, the music changes to something lively and funky, perfectly matching the energetic and vibrant atmosphere of the game.

Reflections

I knew working with machine learning would not be easy however, taking on this challenge working on this project all the more exciting. It was a blend of fun and anxiety I would say. I’m particularly proud of how I managed to keep track of scores and how I elevated the game’s aesthetics. Details like animated characters grooving on the dance floor and hand-illustrated dance move images added an extra layer of vibrancy.

If I could revise one aspect of my project, it would be fine-tuning my model to better recognize distinct dance poses. This, I believe, would further enhance the user’s dance-off experience, making it even more enjoyable and accurate.

REFERENCES

Dan Shiffman’s teachable machine playlist: https://www.youtube.com/watch?v=kwcillcWOg0&list=PLRqwX-V7Uu6aJwX0rFP-7ccA6ivsPDsK5&index=3

 

Overall Concept:

Through this project, I take a chance to reflect on my work and progress during the first half of the semester. In broad terms, it is a compilation of my work in this class, designed as a mini-game where the user themselves has to jump over obstacles I faced. The idea for this concept came to me when I realized how happy I was with my evolution in terms of artistic expression as the class went on, and I wanted everyone to get a glimpse of this evolution.

The game itself is simple enough and can be thought of as a simplified version of the Google Chrome Dinosaur game – with only ground obstacles. (it is deceptively hard!). However, the design is more complex. For example, since I want my viewers to wait and watch the artworks unfold – I made it such that no new obstacles spawn when the cat is stationary. Lastly, I went for a dreamy – after image trail- look for aesthetic purposes.

Working and Strengths:

As I stated earlier, the game itself can be thought of as a simplified version of the Chrome Dinosaur game. Your journey starts the moment you step into the cat’s paws. The left and right arrow keys move the cat, and the spacebar allows it to jump. Additionally, pressing 1 lets you click a screenshot and save it to your device. Whereas, pressing 2 lets you restart the game. You also have the option to toggle the audio on/off.

The progression of the game happens at the user’s own pace. Obstacles only spawn as the cat keeps moving, and they are randomized in size – being one of three variants: “Error”, “Deadline” and “Breakdown”. The cat’s movements include a sitting animation cycle and a walking animation cycle. Aesthetically, I went for a simple color pallet making my main character – a cat white, and the background black, that complimented my color usage in previous projects. The font style I chose resembles handwritten text – a decision made to further highlight the personal tone of the work. I also went for a generally upbeat theme to add to the work!

 

My cat sprites in motion!

 

On a technical level, the collision detection and mechanics were surprisingly hard to implement due to the length of the cat. Even when I got the implementation right, it was nigh impossible to play the game as intended (as it would lead to a continuous triggering of the collision mechanism). I needed to find a way to make this mechanism work without increasing the cat’s speed and worsening the overall experience.

Ultimately, I realized I could also increase the speed on the X axis while jumping, that made the game a lot smoother.

function jump() {
  if (catY >= groundHeight - catHeight / 2) {
    velocity += jumpForce;
    speed = boostSpeed; // boost the speed during the jump
  }
}    //jump function

function checkCollision(obs) {
  return (
    camX + width / 2 + (catWidth / 2) * 0.6 -50 > obs.x &&
    camX + width / 2 - (catWidth / 2) * 0.6 +40< obs.x + obs.width &&
    catY + (catHeight) * 0.6+30 > obs.y &&
    catY - (catHeight) * 0.6 -30< obs.y + obs.height
  );
}

Another area whose technicality I am proud of is the cat animations. I only had a sprite sheet for a right walk cycle. In code, I used this to also work for a left walk cycle (by flipping). But more importantly, I managed to find a way to connect my sitting cycle with my walk cycle. The cat now sits down when you are not moving, and then gets back up (in reverse the sit cycle) when you start again!

/ Check direction of camera movement to adjust cat's direction
    if (camX > lastCamX) {
      direction = -1;
    } else if (camX < lastCamX) {
      direction = 1;
    }

    // If camera is not moving and the cat was walking previously
    if (camX == lastCamX && wasMoving) {
      wasMoving = false;
      isWalking = false;
      isSitting = true;
      sitFrames.reverse();
      currentFrame = 0;
    }
// If camera starts moving and the cat was sitting previously
    if (camX != lastCamX && !wasMoving) {
      wasMoving = true;
      isSitting = false;
      isWalking = true;
      sitFrames.reverse();
      currentFrame = 0;
    }

// Draw the cat
    let frameToDraw;
    if (isWalking) {
      frameToDraw = walkFrames[currentFrame % 12];
      if (frameCount % 2 == 0)
        //making the animation slower
        currentFrame = (currentFrame + 1) % 12;
    } else if (isSitting) {
      frameToDraw = sitFrames[currentFrame];
      if (frameCount % 2 == 0 && currentFrame < 6) {
        currentFrame++;
      }
    }

On a more fun note, while working on the project I realized a peculiarity of the code – if you try to run away from the obstacles – they start bunching up and appear together more frequently – a fact we all know to be true from experience.

Reflections and Future Work:

Throughout this project, I ran into several problems – a couple of which I already highlighted in the above section. In general, staging the whole game was quite the task (as it involved basically a moving background created by a camera movement (that we simulate by a constant translation based on arrow keys)). This could only be resolved with meticulous testing.

Moving on, I’ve identified two crucial components of my project that might require some improvement: the way it appears and the way it functions. I believe I can further improve the project’s aesthetics and visual appeal. To make it more appealing, the background and cat’s appearance can be improved. Additionally, I want to make the text on the obstacles more legible. On a functional level – the hitboxes of the collision functions aren’t yet perfectly set – I’d like to explore using an elliptical hitbox than a rectangular one.

Finally, I would love to include my expectations and plans for the second half of the semester, as well as any abstract ideas or goals I may have as the ending scene – something that I didn’t have time to implement.

 

References:

Music:

Voxel Revolution – Kevin Macleod

Digital Lemonade – Kevin Macleod

Link to the sketch: https://editor.p5js.org/oa2302/full/Yn69_3-Qb

Concept:

The overall concept for my midterm project I have described in my first progress post regarding this assignment. I was inspired by another student’s project, as well as reminiscent of the 2D mobile puzzle games that I loved to play, and thus, I decided to create my own puzzle game. I think puzzle games are meant to be mysterious and invoke a lot of thinking in order to solve them, and thus, when I initially planned on what my midterm would be, I was highly ambitious. Nevertheless, I am extremely happy with the outcome, and exploring this type of concept, though challenging, was highly engaging and fun for me.

Though I initially planned to do 4 different puzzles, as I was creating the first puzzle, I realized that creating 4 different ones was not a task I could do in such a short amount of time. Nevertheless, I chose two puzzles that I could make work well with my beginner coding experience. These two puzzles are finding the differences and solving a riddle. Once these two puzzles are solved, the indicators above the door will turn green, and the door will open, signifying you have won.

What really made this project take so long for me was the fact that I decided to draw each element and scene by hand, and thus, creating each required scene took much more time than it could’ve if I had just used images from the internet. Nevertheless, this makes my work original and authentic.

Process / How It Works:

I began by creating 2 main scenes in a digital drawing application on my iPad. I drew an introduction screen where I would generate the title, start, and instructions buttons. Then, I designed the game scene, focusing on a nearly monochrome 2D scene. Since I wanted each element in the game to be interactive, I decided to draw each element in a separate layer, and then create a button using a transparent PNG file for just that object. Similarly, I created the start and instruction buttons. I also created a state machine using if conditions and constant variables, which allows you to travel between a zoom-in of each object or the main game screen.

The code I am most proud of is most likely breaking every part of the game into pieces, which I made into functions, and thus, my actual function draw() loop only contains the if conditions.

//state machine using if conditions and the infinite draw function loop
function draw() {
  if (state == INTRO_SCREEN) {
    introScreen();
  } else if (state == RULES_SCREEN) {
    rulesScreen();
  } else if (state == GAME_SCREEN) {
    mainMusic.amp(0.4);
    gameScreen();
  } else if (state == PLANT_ZOOM) {
    showPlant();
  } else if (state == PAINTING_ZOOM) {
    showPainting();
  } else if (state == CLOCK_ZOOM) {
    showClock();
  } else if (state == BOOKS_ZOOM) {
    showBooks();
  }
}

function introScreen() {
  //show or hide buttons
  startButton.show();
  rulesButton.show();
  backButton.hide();
  backToGameButton.hide();

  //hide objects
  exitDoor.hide();

  //main
  image(startBG, 0, 0);
  push();
  fill(20);
  stroke(200);
  strokeWeight(5);
  textFont(font1B, 190);
  text("NO WAY OUT", width / 2, 290);
  pop();
}

function rulesScreen() {
  //show or hide buttons
  startButton.hide();
  rulesButton.hide();
  backButton.show();

  //hide objects
  exitDoor.hide();

  image(startBG, 0, 0);
  push();
  fill(20);
  stroke(200);
  strokeWeight(5);
  textFont(font1B, 150);
  text("INSTRUCTIONS", width / 2, 290);
  pop();
  push();
  strokeWeight(20);
  stroke(200, 200, 200, 100);
  fill(125);
  rectMode(CENTER);
  rect(width / 2, 550, width - 100, 310);
  pop();
  textFont(font1A, 30);
  textWrap(WORD);
  text(instructions, width / 2, 450, width - 170);
}

function gameScreen() {
  //hide buttons
  startButton.hide();
  rulesButton.hide();
  backToGameButton.hide();
  answerInput.hide();
  submitButton.hide();
  difference1.hide();
  show_difference1.hide();
  difference2.hide();
  show_difference2.hide();
  difference3.hide();
  show_difference3.hide();
  difference4.hide();
  show_difference4.hide();

  //show all objects
  exitDoor.show();
  plantPot.show();
  clockButton.show();
  booksButton.show();
  paintingButton.show();

  //background and above door indicator
  image(gameBG, 0, 0);
  image(indicator, 257, 175);
  let indicator1 = new Indicator(300, 210);
  let indicator2 = new Indicator(380, 210);


  if (puzzle1solved == true) {
    indicator1.setCorrect();
  } else if (puzzle2solved == true) {
    indicator2.setCorrect();
  }
  indicator1.display();
  indicator2.display();
}

Reflection:

Overall, I believe that this midterm project has been a major milestone in my journey of learning how to code. Ever since I was small, I always dreamed of being able to create games and about all the endless ideas I had. Finally, I can proudly say that I am on the track to fulfilling my dream. Additionally, I believe that by creating all my elements as buttons, it was much easier to manipulate when they were shown and hidden. I am sure this can also be done in many other ways, and I will explore those when the right time comes. Nevertheless, the main issue I faced was expecting too much from myself. I spent over 10 hours in total, drawing, coding, and figuring this game out, however I was unable to reach the goal of 4 different puzzles. I also found it challenging to create puzzles, such as a sudoku, within this game. Nevertheless, I am extremely happy with the final result.

CONCEPT

Link to the sketch: https://editor.p5js.org/MR_Shark/full/oYWGrKk87

My initial idea for the project was to give a user an opportunity to experience playing the piano. While working on my project, I realized that I could create not only a musical instrument simulator, but I could also create a whole educational project aimed at teaching people how to play simple piano pieces. Furthermore, this project can provide people with a tool that could give a partial overview of what it feels like to play the piano without acquiring an actual instrument.

DESCRIPTION OF THE PROJECT

The user experience begins with a start screen and the music piece composed by Ludovico Einaudi – “Fly”

 

Clicking on the screen takes the user to the actual piano simulator.

The keys are assigned to the user’s keyboard so that they replicate the actual piano experience. However, due to the limited width of the computer keyboard, the piano is divided into two halves.

The octaves from 3 to 5 are available to the user as they cover the scope of the most compositions.

Clicking the buttons at the bottom of the page makes the canvas display the notes of the piano pieces that a user can learn how to play. Clicking the left button displays the notes as they appear on the screen (e.g. A, D, F# etc.). Clicking on the right button displays the notes as they appear on a user’s keyboard (e.g. Q,V, [ , M etc.)

Parts of the code I’m proud of

soundFormats("mp3", "ogg");
  mainscreensound = loadSound(
    "x2mate.com - Ludovico Einaudi - Fly (Intouchables Soundtrack) (192 kbps).mp3"
  );
  let a = 0;
  for (let letter = unchar("A"); letter < 72; letter++) {
    for (i = 3; i < 6; i++) {
      let fileName = char(letter) + i + ".mp3";
      soundfiles[a] = loadSound(fileName);
      a = a + 1;
    }
    if (letter != 67 && letter != 70) {
      for (i = 3; i < 6; i++) {
        let fileName = char(letter) + "b" + i + ".mp3";
        soundfiles[a] = loadSound(fileName);
        a = a + 1;
      }
    }
  }

I had a problem with importing a large number of audio files into the program, and I’m proud of the way I managed to solve the problem with importing a large number of sound files (thank you, Professor Shiloh) using char and unchar functions.

Furthermore, I enjoyed creating classes for the piano keyboard. The code for creating black keys is designed in such a way that they appear as meaningful class objects with properties:

class BlackKey {
  constructor(x, y) {
    this.note = blackkeys[y % 5];
    this.octave = Math.floor(y / 5);
  }

  display(x, y) {
    fill("black");
    rect(30 + 30 * x, 300, 20, 75, 3);
    fill("white");
    textFont(font);
    textAlign(CENTER);
    text(blackkeys[y % 5], 40 + 30 * x, 365);
  }
}

REFLECTIONS

The project helped me implement the knowledge I gained throughout the lectures and readings. I see a lot of room for improvement in this project, either technical or graphical. I believe that if I choose to continue the work on the project, creating a broader range of choices for users, improving graphics, and adding more musical instruments, this project could potentially become a widely known useful tool for people trying to get an understanding of what the experience of playing musical instruments feels like without actually possessing the instrument.

Concept:

Link to the sketch: https://editor.p5js.org/ff2185/sketches/J9qU9jniU

The concept behind this idea was born from the coffee shop experience shown in class. When I saw it, I immediately recognized an opportunity to tinker around and produce an experience that would be similar to the one offered in the example. I was heavily interested in generating an interactive experience that would feature various types of “mini-games” or small interactive screens. In this case, I ended up choosing 3 options that would fit perfectly into the bar theme:

1. Piano Simulator: The first experience would be the piano, as it was something that I have been interested in since we started working with audio. Thankfully, I managed to implement a piano that I found in a sketch someone shared online (see reference). By adapting this piano into the system that my sketch was running, I was able to offer the user the capacity to play a fully functional piano made with the p5js.oscillator() in p5js.
2. Bartender Game: The bartender game was inspired by an old famous game for people born around the 2000’s. This game is called “The Right Mix”, and the idea is simple: Tap on a drink and add it to the mix. If the mix is good, then the score is high. If it isn’t, the score will be low.
3. Seated experience: Sit down and enjoy a drink at the bar. You will hear music playing in the background while you enjoy it and take it slow.
   

 

Design and Implementation:

For starters, the biggest challenge I knew I would face would be the state management. Having so many different interactive screens that do multiple varied functions is a big trouble considering how P5Js works. The reason behind this is mainly that P5JS is a framework that requires the user interaction to be screen-dependent: if you want to output something, it has to be relative to the coordinates. Same with the input. This is the only way to interact with the screen: that is why it’s called a sketch, it’s all about drawing.

The first part was to design the bar front, which I did manually by putting shapes in certain coordinates:

After this, I decided to start coding the piano. The piano sound is made naturally from p5js, as opposed to the other option which was to add the sounds of each key manually. As the second option was a nightmare in terms of implementation and code cleanliness, I decided to research how to generate it naturally. This is the result:

if(state == "piano"){
    if (keyCode === 65) { //osc refers to the oscillator
    rSide[0].red();
    osc[0].start();
    osc[0].freq(midiToFreq(root));
    envo[0].play();
  } else if (keyCode === 87) {
    black[0].red();
    osc[1].start();
    osc[1].freq(midiToFreq(root + 1));
    envo[1].play();
  } else if (keyCode === 83) {
    mid[1].red();
    osc[2].start();
    osc[2].freq(midiToFreq(root + 2));
    envo[2].play();
  } else if (keyCode === 69) {
    black[1].red();
    osc[3].start();
    osc[3].freq(midiToFreq(root + 3));
    envo[3].play();
  } else if (keyCode === 68) {
    lSide[2].red();
    osc[4].start();
    osc[4].freq(midiToFreq(root + 4));
    envo[4].play();
  } else if (keyCode === 70) {
    rSide[3].red();
    osc[5].start();
    osc[5].freq(midiToFreq(root + 5));
    envo[5].play();
  } else if (keyCode === 84) {
    black[3].red();
    osc[6].start();
    osc[6].freq(midiToFreq(root + 6));
    envo[6].play();
  } else if (keyCode === 71) {
    mid[4].red();
    osc[7].start();
    osc[7].freq(midiToFreq(root + 7));
    envo[7].play();
  } else if (keyCode === 89) {
    black[4].red();
    osc[8].start();
    osc[8].freq(midiToFreq(root + 8));
    envo[8].play();
  ............

The next step was to design the interior of the bar. After we linked the bar front door click with the interior, we had to design the background for it. For this, my approach was to utilize Ai generated pictures as a starter and work over that. After generating the background, I decided to add it in Photoshop and create the hover options by selecting the different elements and adding a shiny border to them:

For the drink game, I coded a simple interaction where the user can click on the cup and fill it. When it fills to the brim, the bartender will say thanks and the user can leave.

Lastly, the jukebox was made by utilizing the p5.TouchGUI library. This library allowed me to utilize buttons and generate a GUI with a volume slider in a very simple manner. By adding 2 lists, one containing the album covers and the other one containing MP3 files for every song, I managed to create a proper, functional music player with user-inputted files.

Reflections:

For this project, I am ultimately proud of my improvement on the creative side. The thing I struggled with the most was imagining the concepts and functions of my sketches, rather than how to do them. In this midterm assignment, I consider that finally challenged myself in terms of coding and creativity, and finally made an experience I find enjoyable.

Regarding improvements that I could make, I personally consider that there are multiple design decision that were a by-product of my coding decision rather than a conscious design choice. I would have like to utilize the TouchGUI library more to generate a better user experience around the bar (maybe jump through screens easily?).

In general terms, I have learned a lot doing this project and I hope to keep improving my coding and creative skills by the end of this semester.

Reference:

• Piano: https://editor.p5js.org/mrbombmusic/sketches/ryeLfZTd-
• TouchGUI: https://github.com/L05/p5.touchgui
• TouchGUI tutorial: https://www.youtube.com/watch?v=w1hEitRLKog
• Ai-generated images: https://playgroundai.com/
• Music: https://www.youtube.com/channel/UCGnClsipVxB9GhlX2W2ujBw

Concept:

Link to the sketch: https://editor.p5js.org/Yupuuu/full/p-mvwnETP

This project takes its inspiration from the coffee shop example given by Professor Shiloh in class. My idea is to integrate some cultural elements into such an experience, Therefore, I decided to create an experience of a Peking Opera Village, in which users can listen to and learn about Peking Opera and hopefully can eventually appreciate it. The overall idea for this project is that the user enters the village through an arch and arrives at a traditional Peking Opera theater. Here, the users are able to see the main stage, some paintings on the wall, tables, tea and teapots, the opera information, and even play a little trivial game.

Design and Implementation:

The most significant design of the experience is the main stage. For this part, I wanted to create something like this:

Recognizing the complexities of such a sophisticated design, I decided to make a simplified version of it. The simplified version consists of the use of both online images and primitive shapes of p5js. The essential elements that constitute a Chinese style stage remain, such as the yellow dragon and phoenix  patterns, the little fences, and the color of Chinese dark red. Combing these elements, I have created this stage for my project:

Particularly, I am proud of the lanterns swinging animation. This adds a bit fun to this static display of the stage. This is realized by the following code:

lantern.resize(55, 0);
 // the left lantern
 push();
 translate(690, 0);
 rotate(angle);
 image(lantern, -25, 0);
 pop();

 // the right lantern
 push();
 translate(780, 0);
 rotate(angle);
 image(lantern, -25, 0);
 pop();

 // the swing effect of the lantern
 angle += lan_speed;
 if (angle >= 10) {
   lan_speed = -lan_speed;
 } else if (angle <= -10) {
   lan_speed = -lan_speed;
 }

Moreover, whenever the user hovers their mouse onto an interactive object, the cursor will become a hand, indicating the interactivity. This explicitly navigates the user’s interaction in the scene without disturbing the holistic aesthetics of the setting. This is realized by the following code:

if (mouseX >= 28 && mouseX <= 48) {
      // back button
      if (mouseY >= 320 && mouseY <= 348) {
        cursor(HAND);
      }
    } else if (mouseX >= 200 && mouseX <= 838) {
      if (mouseY >= 0 && mouseY <= 418) {
        // stage
        cursor(HAND);
      }
      if (mouseX >= 755 && mouseX <= 805) {
        //paper on table
        if (mouseY >= 493 && mouseY <= 545) {
          cursor(HAND);
        }
      }
    } else if (mouseX >= 45 && mouseX <= 170) {
      //painting on the left
      if (mouseY >= 0 && mouseY <= 158) {
        cursor(HAND);
      }
    } else if (mouseX >= 900 && mouseX <= 1050) {
      // opera today
      if (mouseY >= 10 && mouseY <= 105) {
        cursor(HAND);
      }
    } else {
      cursor(ARROW);
    }

Another interesting part of my project is the little trivia challenge for the user. They can click on the paper on one of the table in the main scene, and it will show them the challenge. To complete the challenge, the user will have to read the Opera Today on the wall to get some information about the opera clip I chose. Then, the user will be asked three questions about the information of the opera. Currently, there is not scoring or reward system with this challenge. This just adds some interactivity to the experience.

Reflections:

For this project, I think the overall output is satisfactory. However, it is clear that to truly replicate an experience in a Peking Opera theater need a lot work than this. Due to the limit of time, I wasn’t able to duplicate all the details. If time allows, for example, I would add some audience, waiters, ect, to make the environment more lively and realistic. Moreover, I could add more sounds, like crowd sounds, to make the soundscape of the experience more realistic. Lastly, I could improve the challenge to make it record score and have a reward system. But this indeed need a lot more efforts and time.

One problem I ran into and was not completely solved is the lagging of the character when moving. As the program progresses, the opera singer on stage’s motion will become more and more lagging. I suspect that the problem is due to the huge memory the program is taking when running for a long time. I tried remove(), but this didn’t work as expected. Then I used clear() whenever the scene changes; this works a bit, but not perfectly. So far, I am not sure how I can solve this problem.

Concept:

For my midterm, I envisioned creating an immersive underwater experience, inspired by my childhood favorite movie “Finding Nemo.” The whole idea is to bring human interaction to the fish world. As a result, I wanted to create a street feel life but under the sea. There fish can go shopping and view artworks in a museum.

I created the whole atmosphere from scratch. I combined vector-based images for the background and other objects using Illustrator. I saved the files as PNG and then added them to P5JS.

Highlight:

The code was the most challenging part mainly for creating layers and buttons to click on. I started the process of implementation by trying to figure out all the variables I needed for each part. Then I loaded all the images and audio I wanted to use for the project. After that, I began to find the logic for moving from one scene to the next. I created a class called experience manager where I give instructions on what to display in the layers when a specific button is clicked. This class is a boolean. As a result, when a specific variable is set to be true it will be visible on the screen: the sound, the images, and the buttons.

// class to manage which slide to display.
class ExperienceManager {
  constructor() {
    this.onHome = true;
    this.onStreet = false;
    this.onMall = false;
    this.onMuseum = false;
  }
  display() {
    if (this.onHome) {
      image(HomeImg, 0, 0, 500, 500);
      SoundWaves.play();
      SoundMall.stop();
      SoundMuseum.stop();
      fishObj = new ClickedObj(fish, 215, 320, 90, 90);
      // Call the textGraphics function
      Title("Life Under the Sea", width / 2, 90, 30);
      // Call the instruction box function
      drawInstructionBox(110, 130, 280, 180, 23);
      this.onStreet = false;
      this.onMall = false;
      this.onMuseum = false;
    } else if (this.onStreet) {
      SoundWaves.play();
      SoundMall.stop();
      SoundMuseum.stop();
      image(seaSt, 0, 0, 500, 500);
      arrowStObj = new ClickedObj(arrow, 10, 20, 25, 25);
      mallObj = new ClickedObj(malltrn, 25, 320, 150, 150);
      museumObj = new ClickedObj(museumtrn, 38, 190, 110, 110);
      this.onHome = false;
      this.onMall = false;
      this.onMuseum = false;
    } else if (this.onMall) {
      //     hide clicks
      SoundMall.play();
      SoundMuseum.stop();
      SoundWaves.stop();
      fishObj.x = -100;
      museumObj.x = -100;
      // bags on click display
      image(mall, 0, 0, 500, 500);
      arrowMallObj = new ClickedObj(arrow, 10, 20, 25, 25);

      this.onStreet = false;
      this.onHome = false;
      this.onMuseum = false;
    } else if (this.onMuseum) {
      image(museum, 0, 0, 500, 500);
      SoundWaves.stop();
      SoundMall.stop();
      SoundMuseum.play();
      arrowMuseumObj = new ClickedObj(arrow, 10, 20, 25, 25);
      this.onHome = false;
      this.onMall = false;
      this.onStreet = false;
    }
  }
}

I also created a class for clickable objects like the fish, and arrows that go to the previous layer, the mall, the museum, the bags, and artworks. This class has a lot of conditionals in it. For some, I made clickable images but for the bags and artwork, I felt it’s better to have curser values that refer to a specific bag of artwork and then display an image for that. Then I created a function for the home screen instructions and passed the text through an array of strings. And added a rectangle behind it. Finally, I added the title of the interaction set a color, and a font in the title function.

// this class helps because it removes alot of hard coding so when I change the coordinates of an inage the program will recognise and so i dont need to change it everywhere.
class ClickedObj {
  constructor(imgFileName, x, y, imgWidth, imgHeight) {
    this.imgFileName = imgFileName;
    this.x = x;
    this.y = y;
    this.imgWidth = imgWidth;
    this.imgHeight = imgHeight;
    this.drawObjct();
  }
  drawObjct() {
    if (this.imgFileName) {
      image(this.imgFileName, this.x, this.y, this.imgWidth, this.imgHeight);
      // print("image draw sucessfull");
    }
  }
  isSelected() {
    if (
      mouseX > this.x &&
      mouseX < this.x + this.imgWidth &&
      mouseY > this.y &&
      mouseY < this.y + this.imgHeight
    ) {
      return true;
    } else return false;
  }
}

// to go to the next layer of the street so all clickable objects are here.
function mouseClicked() {
  //  to go  back to the home layer and instructions and click other objects like the fish etc.
  if (expMgr.onHome) {
    if (fishObj.isSelected()) {
      // print("fish 1 is clicked");
      expMgr.onHome = false;
      expMgr.onStreet = true;
      expMgr.onMall = false;
      expMgr.onMuseum = false;
      expMgr.display();
      //print("done")
    }
  } else if (expMgr.onStreet) {
    if (mallObj.isSelected()) {
      // print("mall is clicked");
      expMgr.onHome = false;
      expMgr.onStreet = false;
      expMgr.onMall = true;
      expMgr.onMuseum = false;
      expMgr.display();
    } else if (museumObj.isSelected()) {
      // print("museum is clicked");
      expMgr.onHome = false;
      expMgr.onStreet = false;
      expMgr.onMall = false;
      expMgr.onMuseum = true;
      expMgr.display();
    } else if (arrowStObj.isSelected()) {
      // print("arrow is clicked");
      expMgr.onHome = true;
      expMgr.onStreet = false;
      expMgr.onMall = false;
      expMgr.onMuseum = false;
      expMgr.display();
    }
  } else if (expMgr.onMall) {
    if (arrowMallObj.isSelected()) {
      // print("arrow is clicked");
      expMgr.onHome = false;
      expMgr.onStreet = true;
      expMgr.onMall = false;
      expMgr.onMuseum = false;
      expMgr.display();
    } else if (mouseX >= 83 && mouseX <= 124 && mouseY >= 90 && mouseY <= 135) {
      image(rdBag, width / 4, height / 4, 210, 210);
    } else if (
      mouseX >= 83 &&
      mouseX <= 120 &&
      mouseY >= 200 &&
      mouseY <= 236
    ) {
      image(LneBag, width / 4, height / 4, 210, 210);
    } else if (
      mouseX >= 83 &&
      mouseX <= 121 &&
      mouseY >= 220 &&
      mouseY <= 331
    ) {
      image(BrwnBag, width / 4, height / 4, 210, 210);
    } else if (
      mouseX >= 353 &&
      mouseX <= 379 &&
      mouseY >= 91 &&
      mouseY <= 140
    ) {
      image(YlwBag, width / 4, height / 4, 210, 210);
    } else if (
      mouseX >= 391 &&
      mouseX <= 422 &&
      mouseY >= 96 &&
      mouseY <= 141
    ) {
      image(OrngBag, width / 4, height / 4, 210, 210);
    } else if (
      mouseX >= 345 &&
      mouseX <= 381 &&
      mouseY >= 192 &&
      mouseY <= 238
    ) {
      image(bluBag, width / 4, height / 4, 210, 210);
    } else if (
      mouseX >= 397 &&
      mouseX <= 423 &&
      mouseY >= 202 &&
      mouseY <= 235
    ) {
      image(GrnBg, width / 4, height / 4, 210, 210);
    } else if (
      mouseX >= 355 &&
      mouseX <= 381 &&
      mouseY >= 297 &&
      mouseY <= 336
    ) {
      image(pnkBag, width / 4, height / 4, 210, 210);
    } else if (
      mouseX >= 391 &&
      mouseX <= 427 &&
      mouseY >= 312 &&
      mouseY <= 332
    ) {
      image(whtBag, width / 4, height / 4, 210, 210);
    }
  } else if (expMgr.onMuseum) {
    if (arrowMuseumObj.isSelected()) {
      // print("arrow is clicked");
      expMgr.onHome = false;
      expMgr.onStreet = true;
      expMgr.onMall = false;
      expMgr.onMuseum = false;
      expMgr.display();
    } else if (
      mouseX >= 126 &&
      mouseX <= 177 &&
      mouseY >= 96 &&
      mouseY <= 199
    ) {
      image(art1, width / 3, height / 3, 250, 300);
    } else if (
      mouseX >= 212 &&
      mouseX <= 293 &&
      mouseY >= 102 &&
      mouseY <= 179
    ) {
      image(art3, width / 3, height / 3, 250, 300);
    } else if (
      mouseX >= 330 &&
      mouseX <= 384 &&
      mouseY >= 97 &&
      mouseY <= 191
    ) {
      image(art2, width / 3, height / 3, 250, 300);
    }
  }
}

To be honest the hardest part was the clickable objects class because at the beginning the objects would layer on one another and would still be clickable even if they were not visible. However, I fixed it by passing the clicks to the specific experience they fit in. I also faced a lot of bugs because P5js is case-sensitive, and it would take me hours to realize that but now I know, and it is the first thing I look for if I have a bug.

 

Reflection and ideas for future work or improvements:

I am satisfied with the overall result of the project. For the future, I would add more places to visit and add some motion to the website. Maybe a fish moving, some submarine. I would also want to add a music icon where users can pick a song and pause it if they want. I also want the home page to be more interactive, maybe create graphics with text or water bubbles that move around the canvas.

Resources:

https://itch.io/games/made-with-p5js

https://www.youtube.com/watch?v=MzD7W6Vt6LA

https://www.youtube.com/results?search_query=p5js+object+orianted+

https://p5js.org/reference/#/p5/REMOVE

https://p5js.org/examples/sound-load-and-play-sound.html

https://www.rapidtables.com/web/color/blue-color.html

https://p5js.org/examples/interaction-snake-game.html

https://stackoverflow.com/questions/58477636/transitioning-from-one-scene-to-the-next-with-p5-js

 

 

Concept

Initially, I had planned on making a project with gravity manipulation as a core mechanic. However, I did not particularly like the ideas that I came up with. One particular idea was to create a voxel-based game where players could create objects that would fall to the ground. Upon impact, the land would be destroyed based on the momentum of impact. However, this proved to be difficult. I might attempt this for a future project, but the idea I settled on took its roots in this idea of a voxel-based landscape, where a voxel is a 3-dimensional pixel.

My idea was to implement a landscape constructed with voxels, and the player could play around with. For ease of computation and aesthetic preferences, I decided to make the voxels change height instead of stacking voxels to create the terrain. Additionally, I gave the players the ability to change from 3D to 2D and vice versa.

Lastly, to complete the experience, I added sounds that vary based on the nature of the landscape. The user can control different attributes of the landscape, and experience what it sounds like.

Note: Please enter Fullscreen mode to view the controls.

P.S: Headphones are recommended!

Design

I really enjoy pixel art, which is why I wanted my landscape to be pixel-based instead of being a continuous plane. Some of my previous projects have had the same style, so I wanted to stick to something that I knew design-wise.

To move from 3D to 2D and vice versa, I decided to implement the landscape to be comprised of boxes whose height varies according to Perlin noise. The noise depends on the spatial coordinates of each box in the space, as well as their height in the previous frame. The noise also depends on time, so the landscape evolves right in front of your eyes.

I particularly like the way I transition from the 2D view to the 3D view. The plane rotates as it grows and morphs into a landscape, which is a sleek effect and adds to the experience in my opinion.

The above video is an early sample from the project, where the project was supposed to be a landscape editor with physical interactions.

Code Highlights

Although I’m proud of many parts of the code, there are a few pieces that add a functionality that ties the entire project together.

For example, the following code changes the soundscape by calculating the percentage of each type of terrain on the current landscape and sets the volume of each corresponding sound based on how much of that terrain is present.

//set soundstage according to the composition of the landscape
 if (timeEvolution) {
   playSounds();
   //set audio volume based on the number of tiles of each type
   waterSound.setVolume(
     map(landscape.waterCount, 0, landscape.totalGridPoints, 0, 0.2)
   );
   sandSound.setVolume(
     map(landscape.sandCount, 0, landscape.totalGridPoints, 0, 0.2)
   );
   landSound.setVolume(
     map(landscape.landCount, 0, landscape.totalGridPoints, 0, 0.2)
   );
   mountainSound.setVolume(
     map(landscape.mountainDirtCount, 0, landscape.totalGridPoints, 0, 0.6)
   );
   mountainRockSound.setVolume(
     map(landscape.mountainRockCount, 0, landscape.totalGridPoints, 0, 0.1)
   );
   snowSound.setVolume(
     map(landscape.snowCount, 0, landscape.totalGridPoints, 0, 0.2)
   );
 } else {
   stopSounds();
 }

Despite it being a very simple case of using the map() function, the depth this code snipped creates for the overall piece is amazing, giving the whole experience a much-needed atmospheric vibe.

Project Design — The Good

The way the land is created is by creating a Landscape object, which is a container that manages multiple Tile objects. Each Tile is represented as a box with a particular height. The way the height is determined is using Perlin Noise, which takes in the spatial coordinates of each tile as well as the point in time and spits out a number between 0 and 1. This number is then mapped between a minimum and maximum height-scale, and so we get a landscape comprised of multiple tiles that vary in height.

Perlin noise was the perfect tool to use for generating landscape. Landscapes are typically very smooth, with areas close to each other having similar elevation. Conveniently, Perlin noise outputs similar values for similar input values, giving us areas of higher elevation and other areas of lower ones.

The last part that makes this project come together is using the calculated noise value for different things. With some thresholds, you can specify what heights should be water and which should be grass, so on and so forth. With this, one can color the landscape differently based on the elevation, which really produced a natural looking landscape.

Also, I feel like having sliders being the only controls was a good idea. Sliders are very intuitive to use, and one can figure out how they integrate with the project in no time. I wanted to have other controls in the beginning — like a brush that allows you to morph the canvas — but realized that those controls were just muddying the experience. Instead, I decided to focus on the aesthetic choices like the sound design, which I believe added way more than the brush tool would have.

Project Design — The Not So Good

Working on this project was frustrating. Funnily enough, creating the landscape and everything related to that took no time. However, what I did not realize at that time was how difficult it would be to integrate 2D portions, like the menu and buttons, back into the project.

Firstly, I decided to use a library called EasyCam. This allowed me to easily create a camera object that could be programmed with animations so that it shifted from one perspective to another with a smooth transition, with the click of a button. However, as I started working on the 2D elements, I realized what I had gotten myself into.

Turns out, you cannot really create 2D portions when using the WEBGL renderer. My first thought was to create multiple canvases and swap them when switching to the menu, but that does not work as the renderer is still WEBGL which is 3D.

Then I tried to experiment with something known as Instance mode. Basically, everything that we run in a P5js sketch is part of a larger P5js object. What you can do is create multiple p5js objects to be able to use namespaces, which is just a larger scope that you can specify when declaring object. I did try that, and everything ran, but I was still confused on how to create the menu using instance mode. Thinking back, I realize that I should have stuck to that route, as it would have been much more elegant compared to the solution I came up with.

In the end, I decided to use an idea that I learnt from one of Daniel Schiffman’s videos of turning the menu into a texture and applying it to a plane. Here is where the use of EasyCam proved to be a bad choice. I basically had two states: a game state and a menu state, which would show either the instructions menu or the main game menu. Since the plane lives in a 3D world, the camera being at a different position when going into the menu means the menu plane will be viewed from a different angle. To top this off, the mouse coordinates are skewed when compared to the world coordinated because the world is seen through the lens of the camera. All of this made it hard to implement a menu that was in the right orientation, and buttons that were clickable.

However, I was able to make a crude solution by resetting the camera every time the game state changes. This way, the camera returns to the neutral position when in the menu, and goes back to being movable in the game state.

Overall, I’m proud of the end result, but looking at the guts of the project I see that there are many aspects that can be improved. Still, it was a project that I spent many hours working on, and I’m glad I had fun along the way.

Resources

How to set up JavaScript namespace and classes properly? — Stack Overflow

createGraphics() p5.js Tutorial — The Coding Train

createGraphics() as WebGL Texture — The Coding Train

3D Terrain Generation with Perlin Noise in Processing — The Coding Train

How to Code Procedural Terrain with Perlin Noise (JavaScript & p5.js) — RachelfTech

Font Used

Overbyte

 

 

Concept

For my midterm project, I decided not to make a game, and rather focus on creating a program for an emotional experience. I think that midterms are always associated with stress and people need to find a way to balance these emotions and have time to focus on their well-being. Therefore, I decided to create a meditation program that will allow its users to calm themselves and enjoy self-care exercises.

Design

I started off by visualizing my project. Below you can see the developed design.

 

Here is a more detailed look at the three exercises I’m planning to create:

Frightening moments

The most frightening part for me is to understand how to create several pages that can be manipulated with buttons. I tried to look at the students’ codes, however, they were very complicated and I could not understand them. Therefore, with the help of the professor, I decided to use a switch statement to manipulate different cases (scenes) within the program.

Another complexity was regarding the buttons. For my program, I wanted to have a button as an image. So, a built-in function of creating a button was not appropriate. After doing some search, I eliminated this issue by finding a code that could help me to create and program my own buttons that are images. You can view the code here.

Also, I felt overwhelmed with doing everything within one code as it is very easy to find and analyze the bug when you run a long code. Therefore, I decided to test my exercises in a separate and much shorter code, which I can later implement in the main long code. Thus, I designed two separate codes for the breathing and journaling exercises.

Breathing exercise

For now, I’m still working on debugging the code that causes issues with the switch statement and buttons. Nevertheless, I am proud that the code functions more or less. As I am finished with debugging, I am going to add two last parameters that I need: shapes and sounds.