So, Far…

During this week’s progress report on my midterm project, I focused on a crucial aspect: integrating sound into my steelpan simulation. To achieve this, I created a specialized “object” within my code to neatly organize the sound files for each note. This object, named “noteSounds” in the code, serves as a sort of playlist where I store all the distinct sounds I intend to use. Just like individual songs in a playlist, each sound file is stored separately for easy access.

To ensure precision and avoid any mix-up between notes, I carefully assigned notes to their respective rows on the steelpan. This separation of notes into rows played a key role in achieving accuracy. In essence, rather than directly assigning a sound to a note, I associated it with a specific position on the canvas.

The process involved loading these sound files before using them, much like preparing a playlist before listening to songs. When you click on a note, the code identifies which note you’ve clicked and plays the corresponding sound file. Think of it as striking a key on a piano to produce a sound.

Admittedly, grasping this concept and implementing it did consume a significant amount of time. However, breaking down my code and attaching sound to one row at a time proved to be an effective strategy. It allowed me to avoid overwhelming complexity and troubleshoot more efficiently.

One challenge I encountered was the limited availability of sound files for the steelpan notes on the website I initially used as a source. So, while I downloaded all the available files, they were incomplete and did not correctly represent the notes of the steelpan.

To overcome this, I reached out to a family member who is skilled in playing the steelpan. They played each note and sent me the sound files directly. This resourcefulness enabled me to continue progressing with my project.

For the outer row of notes, I have successfully integrated sound using the available sound files. However, for the middle and core rows, I’ve temporarily included placeholder values. These will be replaced once I receive the sound files from my uncle, who is assisting with this aspect.

Essentially the code works but, for the moment I have commented out the sound objects until the correct sound files are in my possession.

Future Additions:

Looking ahead, my future additions to the project include refining the instructions screen to ensure its functionality and user-friendliness. This should take approximately one day to complete. Subsequently, I will focus on enhancing the overall aesthetics of my project, bringing it closer to completion.

My ultimate goal is to create an engaging and immersive steelpan experience, reminiscent of Trinidad and Tobago’s renowned “Panorama” event, where Steelband compete passionately. I aspire to replicate this cultural experience within my sketch to offer users a genuine taste of the Trinidad and Tobago steelpan tradition.

Below is an embedded sketch of my work:

<iframe “width: 640px; height: 360px; overflow: hidden;” src=”https://editor.p5js.org/KhaleeqaAasiyah/full/pXR0hhpH4″>

 Lucid Dream  –  a world of illusions

This week, I began refining the main page of my project. I introduced a new element called “Balls,” which are three-dimensional spheres composed of points that blend with the background. I strategically positioned these balls, situating some within the largest sphere and others around a smaller central sphere nested within the larger one.

Following the creation of these balls, which serve as links to different pages with distinct visual designs, I added a question mark inside each of them. This was done to pique the user’s curiosity and encourage them to explore each ball and its associated art design.

After meticulously arranging all the elements on the main page, I encountered a performance issue. The program struggled to smoothly render the constantly changing positions of the thousands of points on the canvas. To address this, I dedicated some time to optimizing the code for better processing with p5, successfully resolving this performance challenge.

However, another issue arose. The continuous rotation of the visual space made it challenging for users to select a specific ball of interest. To enhance user-friendliness and reduce stress, I modified the code to halt the orbital rotation and display the sphere in the correct orientation once the mouse entered the large circle.

Finally, after completing these aforementioned steps, I incorporated sound into the project and set it to loop. I chose a non-copyrighted audio piece reminiscent of the Interstellar theme song to infuse the project with a sense of grandeur and enhance its immersive quality.

Reflection and future plans

The main page is nearly finished, and my next step involves connecting the balls to various pages, each showcasing different art pieces. Additionally, I plan to incorporate audio into these individual art pieces to enhance the overall experience.

 

 

So far so good. Currently, I’m done with 90% of the project. What I have left to do is to change the benchmarks for each level and also replace the circle with characters. I’m still contemplating changing the drop objects to sprites.

Highlights so far:

Places in the code I found challenging are the preloading of the images to use as Backgrounds. Apparently, you can not have two preload functions in two classes. I learned this the hard way.

let B1;
let B2;
let B3;
let B4;
let B5;
let B6;
let B7;
let B8;
let IB;
let GB;
let F1;
let F2;


function preload(){
  B1=loadImage("Background/B1.jpg");
  B2=loadImage("Background/B2.jpg");
  B3=loadImage("Background/B3.jpg");
  B4=loadImage("Background/B4.jpg");
  B5=loadImage("Background/B5.jpg");
  B6=loadImage("Background/B6.jpg");
  B7=loadImage("Background/B7.jpg");
  B8=loadImage("Background/B8.jpg");
  IB=loadImage("Background/IB.jpg");
  GB=loadImage("Background/GB.jpg");
  F1=loadFont("Fonts/Font1.ttf");
  F2=loadFont("Fonts/Font2.ttf");
  
}

 

Another challenging part was the collision of the play with the drop objects. getting the collision to be accurate and finding the right place in my code to place the collision function(method) was the main challenge.

One more challenging part of the code was creating the hover effects for the circle shape. This took me some time to figure out a quite simple method to do so.

createhelp(){
    fill(100);
    if(dist(mouseX,mouseY,370,30)<=20){
      fill(60);
    }
    if(dist(mouseX,mouseY,370,30)<=20&&mouseIsPressed){
      this.helpbool=true;
    }
    circle(370,30,40);
    fill(255,255,0);
    textSize(30);
    textFont(NORMAL);
    text("?",370,40);

 

The Background movement was also an Issue for me as unlike the example given in class, my project was moving the background from top to down.

Also defining the outer boundaries was a challenge in this project but with the use of min(mouseX, width-radius) and min(mouseY, height-radius) I was able to address this.

y.showBG(x.getlvl());
   circle(min(mouseX, 390),min(mouseY,490),20);
   gameoverbool=x.collision();

 

I also faced some challenges with the changing of the levels as the code uses more than 3 classes dependent on each other. All in all I really enjoy programming this project. I am working on adding the sound at the moment and then hopefully I would work on the characters and more. Feel free to give me your suggestions.

Current work:

For the midterm project, I had decided to present an interactive representation of the Indian classical dance of Kathak as mentioned in the previous post. I now have a clear image in mind for my project with the entire user experience being presented in the form of a book that allows you to flip pages and explore a new dimension of Kathak on each page.

The main idea would be to use sprite sheet and produce animations of different dance movements. I collected online images depicting the Kathak movements and then combined them to form a sprite sheet.

The aim of my project is to serve an introductory guide into the world of Kathak. A new concept that I explored during this process was the function to import videos into a sketch, this enabled me to create the flipping page effect.

I still have more details to be executed, however I have divided the entire process into steps which is helping me proceed with ease.

Concept: 

“WHAT A WASTE” is an educational game aimed at educating students about the importance of proper waste segregation. Nowadays, the recycling rate in the UAE drops below 10%, mainly due to a lack of awareness and segregation infrastructure, while the government’s sustainable agenda for 2030 aims to divert 75% of the waste produced in the UAE from landfills. Inspired by real-life scenarios, such as students inadvertently placing non-recyclable items like half-full Coca-Cola bottles into recycling bins, the game’s core objective is to prevent contamination in recycling bins. Players are tasked with identifying and sorting recyclable items correctly to avoid contaminating the recycling stream. By doing so, they learn that unsorted waste not only hinders the recycling process but also increases the workload for recycling centers and waste management companies. Through engaging gameplay and practical examples, “WHAT A WASTE” seeks to equip students with the knowledge and habits needed to contribute effectively to recycling efforts, supporting the UAE’s sustainability goals and promoting a cleaner environment.

Rule of the game:

In the “WHAT A WASTE” game, players click the start button to begin. As they do, a variety of trash items will start falling down from the screen. The first crucial step is to catch the plastic waste and place it in the plastic recycling bin. When players successfully catch clean plastic trash, they earn 2 points. However, if they fail to catch any trash, their score remains at zero. The challenge intensifies when players accidentally catch the wrong trash, such as dirty plastic plates or paper boxes, as this causes them to lose one of their remaining lives. This game format not only tests players’ knowledge of proper recycling but also provides an engaging and interactive way to reinforce the importance of correct waste sorting and segregation. When the player accumulates 15 points, they win the game.

Things to Improve:

The first thing I want to improve is the PNG that I use for the waste. Since it is a large size PNG, it is very easy to lose the game. I need to change the size and add more photos of the waste.

Secondly, I need to add design and songs on the start and end pages and also include the instructions on the start page.

Lastly, I would like to update my code to include a restart button and make the game restart after I win or lose the game.

 

 

Levels Progression and Difficulty:

• I’ve introduced multiple levels that progress in difficulty. Users can now experience a gradual increase in challenge as they advance through the game.
• For this draft, I’ve used the same sample image for all levels to focus on increasing the difficulty. The puzzle becomes more complex as the number of columns and rows increases, engaging players further.

Background Music and User Interface:

• To enhance the gaming experience, I’ve added background music on the main screen. This sets the mood for puzzle-solving while ensuring user comfort.
• The music then pauses when users click on the level number 

Sound Effects for Interactivity:

• Responding to user feedback, I’ve incorporated sound effects when tiles are moved. These audio cues make the gameplay more engaging.

Moves Counter:

• To track progress and add a competitive element, I’ve added a moves counter. It records the number of moves players take to complete the puzzle, encouraging strategic thinking and efficient puzzle-solving.

Checking Puzzle Completion:

• I’ve implemented an “isSolved()” function to check whether the puzzle has been completed. This function iterates through the board array, comparing current values with expected indices from the tiles array. (The result will later on be displayed on the canvas)

 Next steps:

As part of my future plans for the project, I intend to replace the sample images with ones that are personally meaningful to me, reflecting my hobbies in horses, nature, and photography. This addition will infuse the game with a personal touch, creating a more enjoyable and relatable experience for users. Moving forward, I plan to further enhance the visual appeal of each level by adding backgrounds that match the theme of the image. This approach will create a more immersive and cohesive experience for players. For instance, if the puzzle features a nature image, the background will incorporate elements from the natural world, such as trees, leaves, or landscapes. 

Additionally, I’m aware that the code is currently quite messy and scattered. While my initial focus was to ensure all functionalities work seamlessly, I plan to undertake a substantial code refactoring effort. In doing this, each level will have its own class, thus enabling independent adjustments to difficulty levels, as well as significantly improving code organization and management.

Reference: https://youtu.be/uQZLzhrzEs4?si=YztBzKDQ2A0Pr1Vk

 

so for now i still did not start with the coding part i am getting to it ,  but i finally found a idea for my midterm.

i saw a project of a spinning record, and i really liked this idea, it is going to have sound and animation , and i will add more to it  from buttons to music, and possibly add abstract art for the background.

The spinning record will play an instrumental mp3 file that i composed on the software Logic. on the bottom i will add a sprite sheet of a character dancing to the uploaded music.

Sprite sheet:

Spinning record example:

Logic instrumental:

 

 

 

AURA PART 2 UPDATE

If you can remember, I was basing my original concept on this painting that I made a couple of weeks ago.

Here is an update of how things are going.

PROGRESS:

So for this iteration of the project, I was mainly focusing on getting the user to be able to type in an initial of their name and the particles in the sketch to change colors accordingly via a CSV file I made connecting each letter of the alphabet to a different set of colors. FUN I know!

I was able to get this part of the project going and start mixing and matching the colors so that the user can actually see multiple colors being displayed in the sketch. 

COOL CODE:

 //TYPE IN INITIAL
  let sample = "A";
  
  // find the row index of the input sample initial and set that value to idx
  for (let i = 0; i < alphabet.length; i++) {
    if (alphabet[i] == sample) {
      idx = i; //4
    }
  } 
  // get corresponding colors 
  colors_for_sample = [color1[idx], color2[idx], color3[idx]];


  for (let i = 0; i < num; i++) {
    particles.push(createVector(random(width), random(height)));
  }
}

function draw() {
  background;
  for (let i = 0; i < num; i++) {
    let p = particles[i];
    square(p.x, p.y, 2);
    
    let color1 = color(colors_for_sample[0]);
    let color2 = color(colors_for_sample[1]); 
    
    fill(lerpColor(color1, color2, noise(p.x * noiseScale, p.y * noiseScale)));// lerpColor() function is used to interpolate two colors to find a third color between them.
    noStroke();

In the next step of my project I hope to add a background image behind my sketch, as well as some sound to make things a bit more interesting.

 

 

 

Initially, my plan was to create an engaging game by incorporating various elements, including sprites, background parallax scrolling, object-oriented programming (OOP), and TensorFlow.js integration for character control through either speech recognition or body motion detection.

However, for several reasons, I have changed my midterm project idea. The game I was initially going to create would have likely been a remake of an existing game, and it didn’t sound very authentic. My goal for taking this class is to challenge myself creatively, and I gained valuable insights during Thursday’s class, which greatly influenced the ideas I’m going to implement in my project. The part I was missing was probably deciding which machine learning model to use. After observing Professor Aya’s demonstration of the poseNet model in class, my project’s direction became clearer. I have transitioned from creating a game to crafting a digital art piece.

As I write this report on October 7 at 4:29 PM, I have been experimenting with the handpose model from the ml5 library. Handpose is a machine-learning model that enables palm detection and hand-skeleton finger tracking in the browser. It can detect one hand at a time, providing 21 3D hand key points that describe important palm and finger locations.

I took a systematic approach, first checking the results when a hand is present in the frame and when it isn’t.

 

My next step was to verify the accuracy of the points obtained from the model. I drew green ellipses using the model’s points to ensure they corresponded to the correct locations on the hand. I noticed that the points were mirrored, which was a result of my attempt to mirror the webcam feed.

I resolved this issue by placing the drawing function for the points between the push() and pop() functions I used to mirror the feed.

I also discovered that the object returned from the prediction included a bounding box for the hand. I drew out the box to observe how it was affected by the hand’s movements. I plan to use the values returned in topLeft and bottomRight to control the volume of the soundtrack I intend to use in the application.

I have also spent time brainstorming how to utilize the information from the model to create the piece. The relevant information I receive from the model includes landmarks, bounding box, and handInViewConfidence. I am contemplating whether to compute a single point from the model’s points or to utilize all the points to create the piece. To make a decision, I have decided to test both approaches to determine which produces the best result.

In light of this, I created a new sketch to plan how to utilize the information from the model. In my first attempt, I created a Point class that takes x, y, and z coordinates, along with the handInViewConfidence. The x, y, and z coordinates are mapped to values between 0 and 255, while the handInViewConfidence is mapped to a value between 90 and 200 (these values are arbitrary). All this information is used to create two colors, which are linearly interpolated to generate a final color.

After creating the sketch for the Point class, I incorporated it into the existing sketch for drawing the landmarks on the hand. I adjusted the drawKeyPoint() function to create points that were added to an array of points. The point objects were then drawn on the canvas from the array.

// A function to create points for the detected keypoints
function loadKeyPoints() {
  for (let i = 0; i < hand.length; i += 1) {
    const prediction = hand[i];
    for (let j = 0; j < prediction.landmarks.length; j += 1) {
      const keypoint = prediction.landmarks[j];
      points.push(new Point(keypoint[0], keypoint[1], 
                            keypoint[2], prediction.handInViewConfidence))
    }
  }
}

 

I also worked on creating different versions of the sketch. For the second version I created, I used curveVertex() instead of point() in the draw function of the Point class to see how the piece would turn out. I liked the outcome, so I decided to include it as a different mode in the program.

In my efforts to make the sketch more interactive, I have been attempting to utilize the SoundClassification model from the ml5 library as well. I tried working with the “SpeechCommands18w” model and my own custom pre-trained speech commands. However, both models I have tried are not accurate. I have had to repeat the same command numerous times because the model fails to recognize it. I am exploring alternative solutions and ways to potentially improve the model’s accuracy.

Although I am still working on the core of my project, I have begun designing the landing page and other sub-interfaces of my program. Below are sketches for some of the pages.

 

Summary

The progress I’ve made so far involves shifting my initial plan from creating a game to crafting a digital art piece. This change came after attending a class that provided valuable insights, particularly in selecting a machine learning model. I’ve been working with the handpose model, addressing issues like mirroring points and exploring the use of bounding box data for sound control.

I’m also brainstorming ways to utilize landmarks and handInViewConfidence from the model to create the art piece, testing various approaches to mapping data to colors. Additionally, I’ve been experimenting with the SoundClassification model from the ml5 library, though I’ve encountered accuracy challenges.

While the core of my project is still in progress, I’ve started designing the program’s landing page and sub-interfaces. Overall, I’ve made progress in refining my project idea and addressing technical aspects while exploring creative possibilities.

Below  is a screenshot of the rough work I’ve been doing.

I made some significant changes to my p5.js game, which I’ve now named “Pop Frenzy”. One of the key updates was the addition of a menu feature, aiming to improve user experience. At first, I attempted to use a while loop to transition between game states, but I ran into some issues when trying to create a final screen for game loss. To address this, I opted for if statements and introduced a variable called “choice” to switch between the menu, game, and game-over screens. This approach allowed for smoother transitions and better screen management.

In terms of visuals, I decided to dynamically change the game screen’s color intensity. As the player loses more balls, the screen shifts from a vibrant red to a darker, more intense shade. This alteration effectively conveys a sense of rising challenge and urgency, heightening player engagement.

In addition, I took the initiative to incorporate various sound effects, breathing life into the gameplay. I included sounds for button clicks, ball pops, game-over scenarios, and even background music to create a more immersive experience. This auditory dimension adds depth and excitement to the game.

Here’s a screenshot of the final loser screen where the player is given the option to either go to the menu or press the shout button that outputs a sound(merely for fun interactivity).

The Menu Screen:

To make things look more neat, I went with a blue title to match everything. As well as, adding some text of the description of the game to get started.

I incorporated a screen allowing players to pause or resume the game by pressing a designated key, which in my case is the escape key. This functionality was achieved by implementing the built-in functions loop() and noLoop() from p5.js. When the escape key is pressed, noLoop() is executed, causing p5.js to halt the execution of the draw function. Subsequently, when escape is pressed again, the game seamlessly resumes from where it was left off.

To facilitate the addition of a paused menu screen, I strategically placed the function containing the loop() and noLoop() calls at the end of the entire code. This ensures that when the escape button is triggered during gameplay, one final draw cycle occurs before suspension. Within this concluding draw cycle, p5.js generates a screen displaying the “game paused” message. This approach ensures a smooth transition between active gameplay and the paused state.

Here’s a snippet of that function:

function keyPressed() {
  if (keyCode === ESCAPE)
  {
    if(balllost==8)
      {
        gamePaused=false;
      }
    else
    {
    gamePaused = !gamePaused; // Toggle gamePaused variable
    }
    if (gamePaused) {
      noLoop(); // Pause the game
    } else {
      loop(); // Resume the game
    }
  }
}

The initial ‘if’ statement basically ensures that if the escape button is pressed in the loser screen, the draw function will still keep looping and won’t cause issues. Since the player already lost, there’s no need for a pause screen at the loser screen.

Overall, I’m quite pleased with how these modifications have elevated the game’s overall experience. The combination of refined visuals, dynamic color shifts, and a diverse range of sound effects contributes to a highly engaging and immersive gameplay. Adapting to challenges along the way has been a learning experience, and I’m proud of the creative and technical progress I’ve made in this project.