My midterm project began as a whimsical idea: imagine a disco party where your favorite cartoon characters hit the dance floor. But this idea seemed too simple and I could not figure out how to make it more interactive and engaging for the user. As I began coding, I stumbled upon Dan Shiffman’s Teachable Machine playlist on YouTube. His playlist sparked a new inspiration – why not infuse ML into my project as well? Plus, a repeated suggestion I got from my friends regarding making the project more engaging was to somehow make the users mimic the dancing characters movement.
This fusion of disco vibes and machine learning transformed my simple “disco experience” into an engaging game. Now, users don’t just watch the dance moves; they become the stars of the show, mimicking poses detected by the webcam. To enhance the scene, I added dancing characters, some spotlights, and floor highlights.
Sketch
Problems/WHAT IM PROUD OF
Since working with a machine learning model was a new experience for me, I faced quite a lot of challenges throughout the coding process. One significant hurdle was integrating my trained machine learning model to recognize user poses. I had a hard time figuring out how I can increment the users score according to the poses detected by the model.
function classifyVideo() {
classifier.classify(video, gotResults);
}
function gotResults(error, results) {
if (error) {
console.error(error);
return;
}
// Store the label and classify again
label = results[0].label;
classifyVideo();
}
function updateScore() {
if (currentImage == clappingImage && label == "Clapping") {
score += 1;
} else if (currentImage == dabbingImage && label == "Dabbing") {
score += 1;
} else if (currentImage == handsUpImage && label == "Holding hands up") {
score += 1;
} else if (currentImage == armRollImage && label == "Rolling arms") {
score += 1;
}
}
Here’s how it works: when the machine learning model processes the user’s dance moves, it returns a label indicating the detected pose, like “Clapping” or “Dabbing.”
Then, the updateScore function comes into play. It’s responsible for comparing the detected pose to the pose displayed on the screen (like clapping or dabbing images) and, if they match, incrementing the user’s score.
Another puzzle was dealing with user stillness. Initially, even if users stood frozen, the machine learning model would guess a dance move. This unexpected stillness would inadvertently increase the user’s score. The solution I came up with was to train my model to identify moments of “no movement,” ensuring it could distinguish between action and inaction.
Since I could not find the right poses online that were also consistent with the overall aesthetics of the game, I decided to sketch the different poses on my ipad and integrated that into my sketch which was another aspect I really enjoyed working on.
What I like the most is how little features such as the dancing characters or the music choice really add to the whole experience. Before you enter the disco, I added a special touch to the music. At the beginning, it’s like you’re standing right outside a disco, and you can hear the muffled sounds of music from inside. It sets the stage and builds anticipation. But as soon as you enter, the music changes to something lively and funky, perfectly matching the energetic and vibrant atmosphere of the game.
Reflections
I knew working with machine learning would not be easy however, taking on this challenge working on this project all the more exciting. It was a blend of fun and anxiety I would say. I’m particularly proud of how I managed to keep track of scores and how I elevated the game’s aesthetics. Details like animated characters grooving on the dance floor and hand-illustrated dance move images added an extra layer of vibrancy.
If I could revise one aspect of my project, it would be fine-tuning my model to better recognize distinct dance poses. This, I believe, would further enhance the user’s dance-off experience, making it even more enjoyable and accurate.
Through this project, I take a chance to reflect on my work and progress during the first half of the semester. In broad terms, it is a compilation of my work in this class, designed as a mini-game where the user themselves has to jump over obstacles I faced. The idea for this concept came to me when I realized how happy I was with my evolution in terms of artistic expression as the class went on, and I wanted everyone to get a glimpse of this evolution.
The game itself is simple enough and can be thought of as a simplified version of the Google Chrome Dinosaur game – with only ground obstacles. (it is deceptively hard!). However, the design is more complex. For example, since I want my viewers to wait and watch the artworks unfold – I made it such that no new obstacles spawn when the cat is stationary. Lastly, I went for a dreamy – after image trail- look for aesthetic purposes.
Working and Strengths:
As I stated earlier, the game itself can be thought of as a simplified version of the Chrome Dinosaur game. Your journey starts the moment you step into the cat’s paws. The left and right arrow keys move the cat, and the spacebar allows it to jump. Additionally, pressing 1 lets you click a screenshot and save it to your device. Whereas, pressing 2 lets you restart the game. You also have the option to toggle the audio on/off.
The progression of the game happens at the user’s own pace. Obstacles only spawn as the cat keeps moving, and they are randomized in size – being one of three variants: “Error”, “Deadline” and “Breakdown”. The cat’s movements include a sitting animation cycle and a walking animation cycle. Aesthetically, I went for a simple color pallet making my main character – a cat white, and the background black, that complimented my color usage in previous projects. The font style I chose resembles handwritten text – a decision made to further highlight the personal tone of the work. I also went for a generally upbeat theme to add to the work!
My cat sprites in motion!
On a technical level, the collision detection and mechanics were surprisingly hard to implement due to the length of the cat. Even when I got the implementation right, it was nigh impossible to play the game as intended (as it would lead to a continuous triggering of the collision mechanism). I needed to find a way to make this mechanism work without increasing the cat’s speed and worsening the overall experience.
Ultimately, I realized I could also increase the speed on the X axis while jumping, that made the game a lot smoother.
Another area whose technicality I am proud of is the cat animations. I only had a sprite sheet for a right walk cycle. In code, I used this to also work for a left walk cycle (by flipping). But more importantly, I managed to find a way to connect my sitting cycle with my walk cycle. The cat now sits down when you are not moving, and then gets back up (in reverse the sit cycle) when you start again!
/ Check direction of camera movement to adjust cat's direction
if (camX > lastCamX) {
direction = -1;
} else if (camX < lastCamX) {
direction = 1;
}
// If camera is not moving and the cat was walking previously
if (camX == lastCamX && wasMoving) {
wasMoving = false;
isWalking = false;
isSitting = true;
sitFrames.reverse();
currentFrame = 0;
}
// If camera starts moving and the cat was sitting previously
if (camX != lastCamX && !wasMoving) {
wasMoving = true;
isSitting = false;
isWalking = true;
sitFrames.reverse();
currentFrame = 0;
}
// Draw the cat
let frameToDraw;
if (isWalking) {
frameToDraw = walkFrames[currentFrame % 12];
if (frameCount % 2 == 0)
//making the animation slower
currentFrame = (currentFrame + 1) % 12;
} else if (isSitting) {
frameToDraw = sitFrames[currentFrame];
if (frameCount % 2 == 0 && currentFrame < 6) {
currentFrame++;
}
}
On a more fun note, while working on the project I realized a peculiarity of the code – if you try to run away from the obstacles – they start bunching up and appear together more frequently – a fact we all know to be true from experience.
Reflections and Future Work:
Throughout this project, I ran into several problems – a couple of which I already highlighted in the above section. In general, staging the whole game was quite the task (as it involved basically a moving background created by a camera movement (that we simulate by a constant translation based on arrow keys)). This could only be resolved with meticulous testing.
Moving on, I’ve identified two crucial components of my project that might require some improvement: the way it appears and the way it functions. I believe I can further improve the project’s aesthetics and visual appeal. To make it more appealing, the background and cat’s appearance can be improved. Additionally, I want to make the text on the obstacles more legible. On a functional level – the hitboxes of the collision functions aren’t yet perfectly set – I’d like to explore using an elliptical hitbox than a rectangular one.
Finally, I would love to include my expectations and plans for the second half of the semester, as well as any abstract ideas or goals I may have as the ending scene – something that I didn’t have time to implement.
The overall concept for my midterm project I have described in my first progress post regarding this assignment. I was inspired by another student’s project, as well as reminiscent of the 2D mobile puzzle games that I loved to play, and thus, I decided to create my own puzzle game. I think puzzle games are meant to be mysterious and invoke a lot of thinking in order to solve them, and thus, when I initially planned on what my midterm would be, I was highly ambitious. Nevertheless, I am extremely happy with the outcome, and exploring this type of concept, though challenging, was highly engaging and fun for me.
Though I initially planned to do 4 different puzzles, as I was creating the first puzzle, I realized that creating 4 different ones was not a task I could do in such a short amount of time. Nevertheless, I chose two puzzles that I could make work well with my beginner coding experience. These two puzzles are finding the differences and solving a riddle. Once these two puzzles are solved, the indicators above the door will turn green, and the door will open, signifying you have won.
What really made this project take so long for me was the fact that I decided to draw each element and scene by hand, and thus, creating each required scene took much more time than it could’ve if I had just used images from the internet. Nevertheless, this makes my work original and authentic.
Process / How It Works:
I began by creating 2 main scenes in a digital drawing application on my iPad. I drew an introduction screen where I would generate the title, start, and instructions buttons. Then, I designed the game scene, focusing on a nearly monochrome 2D scene. Since I wanted each element in the game to be interactive, I decided to draw each element in a separate layer, and then create a button using a transparent PNG file for just that object. Similarly, I created the start and instruction buttons. I also created a state machine using if conditions and constant variables, which allows you to travel between a zoom-in of each object or the main game screen.
The code I am most proud of is most likely breaking every part of the game into pieces, which I made into functions, and thus, my actual function draw() loop only contains the if conditions.
//state machine using if conditions and the infinite draw function loop
function draw() {
if (state == INTRO_SCREEN) {
introScreen();
} else if (state == RULES_SCREEN) {
rulesScreen();
} else if (state == GAME_SCREEN) {
mainMusic.amp(0.4);
gameScreen();
} else if (state == PLANT_ZOOM) {
showPlant();
} else if (state == PAINTING_ZOOM) {
showPainting();
} else if (state == CLOCK_ZOOM) {
showClock();
} else if (state == BOOKS_ZOOM) {
showBooks();
}
}
function introScreen() {
//show or hide buttons
startButton.show();
rulesButton.show();
backButton.hide();
backToGameButton.hide();
//hide objects
exitDoor.hide();
//main
image(startBG, 0, 0);
push();
fill(20);
stroke(200);
strokeWeight(5);
textFont(font1B, 190);
text("NO WAY OUT", width / 2, 290);
pop();
}
function rulesScreen() {
//show or hide buttons
startButton.hide();
rulesButton.hide();
backButton.show();
//hide objects
exitDoor.hide();
image(startBG, 0, 0);
push();
fill(20);
stroke(200);
strokeWeight(5);
textFont(font1B, 150);
text("INSTRUCTIONS", width / 2, 290);
pop();
push();
strokeWeight(20);
stroke(200, 200, 200, 100);
fill(125);
rectMode(CENTER);
rect(width / 2, 550, width - 100, 310);
pop();
textFont(font1A, 30);
textWrap(WORD);
text(instructions, width / 2, 450, width - 170);
}
function gameScreen() {
//hide buttons
startButton.hide();
rulesButton.hide();
backToGameButton.hide();
answerInput.hide();
submitButton.hide();
difference1.hide();
show_difference1.hide();
difference2.hide();
show_difference2.hide();
difference3.hide();
show_difference3.hide();
difference4.hide();
show_difference4.hide();
//show all objects
exitDoor.show();
plantPot.show();
clockButton.show();
booksButton.show();
paintingButton.show();
//background and above door indicator
image(gameBG, 0, 0);
image(indicator, 257, 175);
let indicator1 = new Indicator(300, 210);
let indicator2 = new Indicator(380, 210);
if (puzzle1solved == true) {
indicator1.setCorrect();
} else if (puzzle2solved == true) {
indicator2.setCorrect();
}
indicator1.display();
indicator2.display();
}
Reflection:
Overall, I believe that this midterm project has been a major milestone in my journey of learning how to code. Ever since I was small, I always dreamed of being able to create games and about all the endless ideas I had. Finally, I can proudly say that I am on the track to fulfilling my dream. Additionally, I believe that by creating all my elements as buttons, it was much easier to manipulate when they were shown and hidden. I am sure this can also be done in many other ways, and I will explore those when the right time comes. Nevertheless, the main issue I faced was expecting too much from myself. I spent over 10 hours in total, drawing, coding, and figuring this game out, however I was unable to reach the goal of 4 different puzzles. I also found it challenging to create puzzles, such as a sudoku, within this game. Nevertheless, I am extremely happy with the final result.
My initial idea for the project was to give a user an opportunity to experience playing the piano. While working on my project, I realized that I could create not only a musical instrument simulator, but I could also create a whole educational project aimed at teaching people how to play simple piano pieces. Furthermore, this project can provide people with a tool that could give a partial overview of what it feels like to play the piano without acquiring an actual instrument.
DESCRIPTION OF THE PROJECT
The user experience begins with a start screen and the music piece composed by Ludovico Einaudi – “Fly”
Clicking on the screen takes the user to the actual piano simulator.
The keys are assigned to the user’s keyboard so that they replicate the actual piano experience. However, due to the limited width of the computer keyboard, the piano is divided into two halves.
The octaves from 3 to 5 are available to the user as they cover the scope of the most compositions.
Clicking the buttons at the bottom of the page makes the canvas display the notes of the piano pieces that a user can learn how to play. Clicking the left button displays the notes as they appear on the screen (e.g. A, D, F# etc.). Clicking on the right button displays the notes as they appear on a user’s keyboard (e.g. Q,V, [ , M etc.)
Parts of the code I’m proud of
soundFormats("mp3", "ogg");
mainscreensound = loadSound(
"x2mate.com - Ludovico Einaudi - Fly (Intouchables Soundtrack) (192 kbps).mp3"
);
let a = 0;
for (let letter = unchar("A"); letter < 72; letter++) {
for (i = 3; i < 6; i++) {
let fileName = char(letter) + i + ".mp3";
soundfiles[a] = loadSound(fileName);
a = a + 1;
}
if (letter != 67 && letter != 70) {
for (i = 3; i < 6; i++) {
let fileName = char(letter) + "b" + i + ".mp3";
soundfiles[a] = loadSound(fileName);
a = a + 1;
}
}
}
I had a problem with importing a large number of audio files into the program, and I’m proud of the way I managed to solve the problem with importing a large number of sound files (thank you, Professor Shiloh) using char and unchar functions.
Furthermore, I enjoyed creating classes for the piano keyboard. The code for creating black keys is designed in such a way that they appear as meaningful class objects with properties:
The project helped me implement the knowledge I gained throughout the lectures and readings. I see a lot of room for improvement in this project, either technical or graphical. I believe that if I choose to continue the work on the project, creating a broader range of choices for users, improving graphics, and adding more musical instruments, this project could potentially become a widely known useful tool for people trying to get an understanding of what the experience of playing musical instruments feels like without actually possessing the instrument.
The concept behind this idea was born from the coffee shop experience shown in class. When I saw it, I immediately recognized an opportunity to tinker around and produce an experience that would be similar to the one offered in the example. I was heavily interested in generating an interactive experience that would feature various types of “mini-games” or small interactive screens. In this case, I ended up choosing 3 options that would fit perfectly into the bar theme:
Piano Simulator: The first experience would be the piano, as it was something that I have been interested in since we started working with audio. Thankfully, I managed to implement a piano that I found in a sketch someone shared online (see reference). By adapting this piano into the system that my sketch was running, I was able to offer the user the capacity to play a fully functional piano made with the p5js.oscillator() in p5js.
Bartender Game: The bartender game was inspired by an old famous game for people born around the 2000’s. This game is called “The Right Mix”, and the idea is simple: Tap on a drink and add it to the mix. If the mix is good, then the score is high. If it isn’t, the score will be low.
Seated experience: Sit down and enjoy a drink at the bar. You will hear music playing in the background while you enjoy it and take it slow.
Design and Implementation:
For starters, the biggest challenge I knew I would face would be the state management. Having so many different interactive screens that do multiple varied functions is a big trouble considering how P5Js works. The reason behind this is mainly that P5JS is a framework that requires the user interaction to be screen-dependent: if you want to output something, it has to be relative to the coordinates. Same with the input. This is the only way to interact with the screen: that is why it’s called a sketch, it’s all about drawing.
The first part was to design the bar front, which I did manually by putting shapes in certain coordinates:
After this, I decided to start coding the piano. The piano sound is made naturally from p5js, as opposed to the other option which was to add the sounds of each key manually. As the second option was a nightmare in terms of implementation and code cleanliness, I decided to research how to generate it naturally. This is the result:
The next step was to design the interior of the bar. After we linked the bar front door click with the interior, we had to design the background for it. For this, my approach was to utilize Ai generated pictures as a starter and work over that. After generating the background, I decided to add it in Photoshop and create the hover options by selecting the different elements and adding a shiny border to them:
For the drink game, I coded a simple interaction where the user can click on the cup and fill it. When it fills to the brim, the bartender will say thanks and the user can leave.
Lastly, the jukebox was made by utilizing the p5.TouchGUI library. This library allowed me to utilize buttons and generate a GUI with a volume slider in a very simple manner. By adding 2 lists, one containing the album covers and the other one containing MP3 files for every song, I managed to create a proper, functional music player with user-inputted files.
Reflections:
For this project, I am ultimately proud of my improvement on the creative side. The thing I struggled with the most was imagining the concepts and functions of my sketches, rather than how to do them. In this midterm assignment, I consider that finally challenged myself in terms of coding and creativity, and finally made an experience I find enjoyable.
Regarding improvements that I could make, I personally consider that there are multiple design decision that were a by-product of my coding decision rather than a conscious design choice. I would have like to utilize the TouchGUI library more to generate a better user experience around the bar (maybe jump through screens easily?).
In general terms, I have learned a lot doing this project and I hope to keep improving my coding and creative skills by the end of this semester.
This project takes its inspiration from the coffee shop example given by Professor Shiloh in class. My idea is to integrate some cultural elements into such an experience, Therefore, I decided to create an experience of a Peking Opera Village, in which users can listen to and learn about Peking Opera and hopefully can eventually appreciate it. The overall idea for this project is that the user enters the village through an arch and arrives at a traditional Peking Opera theater. Here, the users are able to see the main stage, some paintings on the wall, tables, tea and teapots, the opera information, and even play a little trivial game.
Design and Implementation:
The most significant design of the experience is the main stage. For this part, I wanted to create something like this:
Recognizing the complexities of such a sophisticated design, I decided to make a simplified version of it. The simplified version consists of the use of both online images and primitive shapes of p5js. The essential elements that constitute a Chinese style stage remain, such as the yellow dragon and phoenix patterns, the little fences, and the color of Chinese dark red. Combing these elements, I have created this stage for my project:
Particularly, I am proud of the lanterns swinging animation. This adds a bit fun to this static display of the stage. This is realized by the following code:
lantern.resize(55, 0);
// the left lantern
push();
translate(690, 0);
rotate(angle);
image(lantern, -25, 0);
pop();
// the right lantern
push();
translate(780, 0);
rotate(angle);
image(lantern, -25, 0);
pop();
// the swing effect of the lantern
angle += lan_speed;
if (angle >= 10) {
lan_speed = -lan_speed;
} else if (angle <= -10) {
lan_speed = -lan_speed;
}
Moreover, whenever the user hovers their mouse onto an interactive object, the cursor will become a hand, indicating the interactivity. This explicitly navigates the user’s interaction in the scene without disturbing the holistic aesthetics of the setting. This is realized by the following code:
if (mouseX >= 28 && mouseX <= 48) {
// back button
if (mouseY >= 320 && mouseY <= 348) {
cursor(HAND);
}
} else if (mouseX >= 200 && mouseX <= 838) {
if (mouseY >= 0 && mouseY <= 418) {
// stage
cursor(HAND);
}
if (mouseX >= 755 && mouseX <= 805) {
//paper on table
if (mouseY >= 493 && mouseY <= 545) {
cursor(HAND);
}
}
} else if (mouseX >= 45 && mouseX <= 170) {
//painting on the left
if (mouseY >= 0 && mouseY <= 158) {
cursor(HAND);
}
} else if (mouseX >= 900 && mouseX <= 1050) {
// opera today
if (mouseY >= 10 && mouseY <= 105) {
cursor(HAND);
}
} else {
cursor(ARROW);
}
Another interesting part of my project is the little trivia challenge for the user. They can click on the paper on one of the table in the main scene, and it will show them the challenge. To complete the challenge, the user will have to read the Opera Today on the wall to get some information about the opera clip I chose. Then, the user will be asked three questions about the information of the opera. Currently, there is not scoring or reward system with this challenge. This just adds some interactivity to the experience.
Reflections:
For this project, I think the overall output is satisfactory. However, it is clear that to truly replicate an experience in a Peking Opera theater need a lot work than this. Due to the limit of time, I wasn’t able to duplicate all the details. If time allows, for example, I would add some audience, waiters, ect, to make the environment more lively and realistic. Moreover, I could add more sounds, like crowd sounds, to make the soundscape of the experience more realistic. Lastly, I could improve the challenge to make it record score and have a reward system. But this indeed need a lot more efforts and time.
One problem I ran into and was not completely solved is the lagging of the character when moving. As the program progresses, the opera singer on stage’s motion will become more and more lagging. I suspect that the problem is due to the huge memory the program is taking when running for a long time. I tried remove(), but this didn’t work as expected. Then I used clear() whenever the scene changes; this works a bit, but not perfectly. So far, I am not sure how I can solve this problem.
For my midterm, I envisioned creating an immersive underwater experience, inspired by my childhood favorite movie “Finding Nemo.” The whole idea is to bring human interaction to the fish world. As a result, I wanted to create a street feel life but under the sea. There fish can go shopping and view artworks in a museum.
I created the whole atmosphere from scratch. I combined vector-based images for the background and other objects using Illustrator. I saved the files as PNG and then added them to P5JS.
Highlight:
The code was the most challenging part mainly for creating layers and buttons to click on. I started the process of implementation by trying to figure out all the variables I needed for each part. Then I loaded all the images and audio I wanted to use for the project. After that, I began to find the logic for moving from one scene to the next. I created a class called experience manager where I give instructions on what to display in the layers when a specific button is clicked. This class is a boolean. As a result, when a specific variable is set to be true it will be visible on the screen: the sound, the images, and the buttons.
// class to manage which slide to display.
class ExperienceManager {
constructor() {
this.onHome = true;
this.onStreet = false;
this.onMall = false;
this.onMuseum = false;
}
display() {
if (this.onHome) {
image(HomeImg, 0, 0, 500, 500);
SoundWaves.play();
SoundMall.stop();
SoundMuseum.stop();
fishObj = new ClickedObj(fish, 215, 320, 90, 90);
// Call the textGraphics function
Title("Life Under the Sea", width / 2, 90, 30);
// Call the instruction box function
drawInstructionBox(110, 130, 280, 180, 23);
this.onStreet = false;
this.onMall = false;
this.onMuseum = false;
} else if (this.onStreet) {
SoundWaves.play();
SoundMall.stop();
SoundMuseum.stop();
image(seaSt, 0, 0, 500, 500);
arrowStObj = new ClickedObj(arrow, 10, 20, 25, 25);
mallObj = new ClickedObj(malltrn, 25, 320, 150, 150);
museumObj = new ClickedObj(museumtrn, 38, 190, 110, 110);
this.onHome = false;
this.onMall = false;
this.onMuseum = false;
} else if (this.onMall) {
// hide clicks
SoundMall.play();
SoundMuseum.stop();
SoundWaves.stop();
fishObj.x = -100;
museumObj.x = -100;
// bags on click display
image(mall, 0, 0, 500, 500);
arrowMallObj = new ClickedObj(arrow, 10, 20, 25, 25);
this.onStreet = false;
this.onHome = false;
this.onMuseum = false;
} else if (this.onMuseum) {
image(museum, 0, 0, 500, 500);
SoundWaves.stop();
SoundMall.stop();
SoundMuseum.play();
arrowMuseumObj = new ClickedObj(arrow, 10, 20, 25, 25);
this.onHome = false;
this.onMall = false;
this.onStreet = false;
}
}
}
I also created a class for clickable objects like the fish, and arrows that go to the previous layer, the mall, the museum, the bags, and artworks. This class has a lot of conditionals in it. For some, I made clickable images but for the bags and artwork, I felt it’s better to have curser values that refer to a specific bag of artwork and then display an image for that. Then I created a function for the home screen instructions and passed the text through an array of strings. And added a rectangle behind it. Finally, I added the title of the interaction set a color, and a font in the title function.
// this class helps because it removes alot of hard coding so when I change the coordinates of an inage the program will recognise and so i dont need to change it everywhere.
class ClickedObj {
constructor(imgFileName, x, y, imgWidth, imgHeight) {
this.imgFileName = imgFileName;
this.x = x;
this.y = y;
this.imgWidth = imgWidth;
this.imgHeight = imgHeight;
this.drawObjct();
}
drawObjct() {
if (this.imgFileName) {
image(this.imgFileName, this.x, this.y, this.imgWidth, this.imgHeight);
// print("image draw sucessfull");
}
}
isSelected() {
if (
mouseX > this.x &&
mouseX < this.x + this.imgWidth &&
mouseY > this.y &&
mouseY < this.y + this.imgHeight
) {
return true;
} else return false;
}
}
// to go to the next layer of the street so all clickable objects are here.
function mouseClicked() {
// to go back to the home layer and instructions and click other objects like the fish etc.
if (expMgr.onHome) {
if (fishObj.isSelected()) {
// print("fish 1 is clicked");
expMgr.onHome = false;
expMgr.onStreet = true;
expMgr.onMall = false;
expMgr.onMuseum = false;
expMgr.display();
//print("done")
}
} else if (expMgr.onStreet) {
if (mallObj.isSelected()) {
// print("mall is clicked");
expMgr.onHome = false;
expMgr.onStreet = false;
expMgr.onMall = true;
expMgr.onMuseum = false;
expMgr.display();
} else if (museumObj.isSelected()) {
// print("museum is clicked");
expMgr.onHome = false;
expMgr.onStreet = false;
expMgr.onMall = false;
expMgr.onMuseum = true;
expMgr.display();
} else if (arrowStObj.isSelected()) {
// print("arrow is clicked");
expMgr.onHome = true;
expMgr.onStreet = false;
expMgr.onMall = false;
expMgr.onMuseum = false;
expMgr.display();
}
} else if (expMgr.onMall) {
if (arrowMallObj.isSelected()) {
// print("arrow is clicked");
expMgr.onHome = false;
expMgr.onStreet = true;
expMgr.onMall = false;
expMgr.onMuseum = false;
expMgr.display();
} else if (mouseX >= 83 && mouseX <= 124 && mouseY >= 90 && mouseY <= 135) {
image(rdBag, width / 4, height / 4, 210, 210);
} else if (
mouseX >= 83 &&
mouseX <= 120 &&
mouseY >= 200 &&
mouseY <= 236
) {
image(LneBag, width / 4, height / 4, 210, 210);
} else if (
mouseX >= 83 &&
mouseX <= 121 &&
mouseY >= 220 &&
mouseY <= 331
) {
image(BrwnBag, width / 4, height / 4, 210, 210);
} else if (
mouseX >= 353 &&
mouseX <= 379 &&
mouseY >= 91 &&
mouseY <= 140
) {
image(YlwBag, width / 4, height / 4, 210, 210);
} else if (
mouseX >= 391 &&
mouseX <= 422 &&
mouseY >= 96 &&
mouseY <= 141
) {
image(OrngBag, width / 4, height / 4, 210, 210);
} else if (
mouseX >= 345 &&
mouseX <= 381 &&
mouseY >= 192 &&
mouseY <= 238
) {
image(bluBag, width / 4, height / 4, 210, 210);
} else if (
mouseX >= 397 &&
mouseX <= 423 &&
mouseY >= 202 &&
mouseY <= 235
) {
image(GrnBg, width / 4, height / 4, 210, 210);
} else if (
mouseX >= 355 &&
mouseX <= 381 &&
mouseY >= 297 &&
mouseY <= 336
) {
image(pnkBag, width / 4, height / 4, 210, 210);
} else if (
mouseX >= 391 &&
mouseX <= 427 &&
mouseY >= 312 &&
mouseY <= 332
) {
image(whtBag, width / 4, height / 4, 210, 210);
}
} else if (expMgr.onMuseum) {
if (arrowMuseumObj.isSelected()) {
// print("arrow is clicked");
expMgr.onHome = false;
expMgr.onStreet = true;
expMgr.onMall = false;
expMgr.onMuseum = false;
expMgr.display();
} else if (
mouseX >= 126 &&
mouseX <= 177 &&
mouseY >= 96 &&
mouseY <= 199
) {
image(art1, width / 3, height / 3, 250, 300);
} else if (
mouseX >= 212 &&
mouseX <= 293 &&
mouseY >= 102 &&
mouseY <= 179
) {
image(art3, width / 3, height / 3, 250, 300);
} else if (
mouseX >= 330 &&
mouseX <= 384 &&
mouseY >= 97 &&
mouseY <= 191
) {
image(art2, width / 3, height / 3, 250, 300);
}
}
}
To be honest the hardest part was the clickable objects class because at the beginning the objects would layer on one another and would still be clickable even if they were not visible. However, I fixed it by passing the clicks to the specific experience they fit in. I also faced a lot of bugs because P5js is case-sensitive, and it would take me hours to realize that but now I know, and it is the first thing I look for if I have a bug.
Reflection and ideas for future work or improvements:
I am satisfied with the overall result of the project. For the future, I would add more places to visit and add some motion to the website. Maybe a fish moving, some submarine. I would also want to add a music icon where users can pick a song and pause it if they want. I also want the home page to be more interactive, maybe create graphics with text or water bubbles that move around the canvas.
Initially, I had planned on making a project with gravity manipulation as a core mechanic. However, I did not particularly like the ideas that I came up with. One particular idea was to create a voxel-based game where players could create objects that would fall to the ground. Upon impact, the land would be destroyed based on the momentum of impact. However, this proved to be difficult. I might attempt this for a future project, but the idea I settled on took its roots in this idea of a voxel-based landscape, where a voxel is a 3-dimensional pixel.
My idea was to implement a landscape constructed with voxels, and the player could play around with. For ease of computation and aesthetic preferences, I decided to make the voxels change height instead of stacking voxels to create the terrain. Additionally, I gave the players the ability to change from 3D to 2D and vice versa.
Lastly, to complete the experience, I added sounds that vary based on the nature of the landscape. The user can control different attributes of the landscape, and experience what it sounds like.
Note: Please enter Fullscreen mode to view the controls.
P.S: Headphones are recommended!
Design
I really enjoy pixel art, which is why I wanted my landscape to be pixel-based instead of being a continuous plane. Some of my previous projects have had the same style, so I wanted to stick to something that I knew design-wise.
To move from 3D to 2D and vice versa, I decided to implement the landscape to be comprised of boxes whose height varies according to Perlin noise. The noise depends on the spatial coordinates of each box in the space, as well as their height in the previous frame. The noise also depends on time, so the landscape evolves right in front of your eyes.
I particularly like the way I transition from the 2D view to the 3D view. The plane rotates as it grows and morphs into a landscape, which is a sleek effect and adds to the experience in my opinion.
The above video is an early sample from the project, where the project was supposed to be a landscape editor with physical interactions.
Code Highlights
Although I’m proud of many parts of the code, there are a few pieces that add a functionality that ties the entire project together.
For example, the following code changes the soundscape by calculating the percentage of each type of terrain on the current landscape and sets the volume of each corresponding sound based on how much of that terrain is present.
//set soundstage according to the composition of the landscape
if (timeEvolution) {
playSounds();
//set audio volume based on the number of tiles of each type
waterSound.setVolume(
map(landscape.waterCount, 0, landscape.totalGridPoints, 0, 0.2)
);
sandSound.setVolume(
map(landscape.sandCount, 0, landscape.totalGridPoints, 0, 0.2)
);
landSound.setVolume(
map(landscape.landCount, 0, landscape.totalGridPoints, 0, 0.2)
);
mountainSound.setVolume(
map(landscape.mountainDirtCount, 0, landscape.totalGridPoints, 0, 0.6)
);
mountainRockSound.setVolume(
map(landscape.mountainRockCount, 0, landscape.totalGridPoints, 0, 0.1)
);
snowSound.setVolume(
map(landscape.snowCount, 0, landscape.totalGridPoints, 0, 0.2)
);
} else {
stopSounds();
}
Despite it being a very simple case of using the map() function, the depth this code snipped creates for the overall piece is amazing, giving the whole experience a much-needed atmospheric vibe.
Project Design — The Good
The way the land is created is by creating a Landscape object, which is a container that manages multiple Tile objects. Each Tile is represented as a box with a particular height. The way the height is determined is using Perlin Noise, which takes in the spatial coordinates of each tile as well as the point in time and spits out a number between 0 and 1. This number is then mapped between a minimum and maximum height-scale, and so we get a landscape comprised of multiple tiles that vary in height.
Perlin noise was the perfect tool to use for generating landscape. Landscapes are typically very smooth, with areas close to each other having similar elevation. Conveniently, Perlin noise outputs similar values for similar input values, giving us areas of higher elevation and other areas of lower ones.
The last part that makes this project come together is using the calculated noise value for different things. With some thresholds, you can specify what heights should be water and which should be grass, so on and so forth. With this, one can color the landscape differently based on the elevation, which really produced a natural looking landscape.
Also, I feel like having sliders being the only controls was a good idea. Sliders are very intuitive to use, and one can figure out how they integrate with the project in no time. I wanted to have other controls in the beginning — like a brush that allows you to morph the canvas — but realized that those controls were just muddying the experience. Instead, I decided to focus on the aesthetic choices like the sound design, which I believe added way more than the brush tool would have.
Project Design — The Not So Good
Working on this project was frustrating. Funnily enough, creating the landscape and everything related to that took no time. However, what I did not realize at that time was how difficult it would be to integrate 2D portions, like the menu and buttons, back into the project.
Firstly, I decided to use a library called EasyCam. This allowed me to easily create a camera object that could be programmed with animations so that it shifted from one perspective to another with a smooth transition, with the click of a button. However, as I started working on the 2D elements, I realized what I had gotten myself into.
Turns out, you cannot really create 2D portions when using the WEBGL renderer. My first thought was to create multiple canvases and swap them when switching to the menu, but that does not work as the renderer is still WEBGL which is 3D.
Then I tried to experiment with something known as Instance mode. Basically, everything that we run in a P5js sketch is part of a larger P5js object. What you can do is create multiple p5js objects to be able to use namespaces, which is just a larger scope that you can specify when declaring object. I did try that, and everything ran, but I was still confused on how to create the menu using instance mode. Thinking back, I realize that I should have stuck to that route, as it would have been much more elegant compared to the solution I came up with.
In the end, I decided to use an idea that I learnt from one of Daniel Schiffman’s videosof turning the menu into a texture and applying it to a plane. Here is where the use of EasyCam proved to be a bad choice. I basically had two states: a game state and a menu state, which would show either the instructions menu or the main game menu. Since the plane lives in a 3D world, the camera being at a different position when going into the menu means the menu plane will be viewed from a different angle. To top this off, the mouse coordinates are skewed when compared to the world coordinated because the world is seen through the lens of the camera. All of this made it hard to implement a menu that was in the right orientation, and buttons that were clickable.
However, I was able to make a crude solution by resetting the camera every time the game state changes. This way, the camera returns to the neutral position when in the menu, and goes back to being movable in the game state.
Overall, I’m proud of the end result, but looking at the guts of the project I see that there are many aspects that can be improved. Still, it was a project that I spent many hours working on, and I’m glad I had fun along the way.
For my midterm project, I decided not to make a game, and rather focus on creating a program for an emotional experience. I think that midterms are always associated with stress and people need to find a way to balance these emotions and have time to focus on their well-being. Therefore, I decided to create a meditation program that will allow its users to calm themselves and enjoy self-care exercises.
Design
I started off by visualizing my project. Below you can see the developed design.
Here is a more detailed look at the three exercises I’m planning to create:
Frightening moments
The most frightening part for me is to understand how to create several pages that can be manipulated with buttons. I tried to look at the students’ codes, however, they were very complicated and I could not understand them. Therefore, with the help of the professor, I decided to use a switch statement to manipulate different cases (scenes) within the program.
Another complexity was regarding the buttons. For my program, I wanted to have a button as an image. So, a built-in function of creating a button was not appropriate. After doing some search, I eliminated this issue by finding a code that could help me to create and program my own buttons that are images. You can view the code here.
Also, I felt overwhelmed with doing everything within one code as it is very easy to find and analyze the bug when you run a long code. Therefore, I decided to test my exercises in a separate and much shorter code, which I can later implement in the main long code. Thus, I designed two separate codes for the breathing and journaling exercises.
Breathing exercise
For now, I’m still working on debugging the code that causes issues with the switch statement and buttons. Nevertheless, I am proud that the code functions more or less. As I am finished with debugging, I am going to add two last parameters that I need: shapes and sounds.
Concept:
“Pride Dragon Generator” is essentially a very simplified version of a Picrew avatar maker, in which users can select a LGBTQ+ identity and receive a dragon with spike colors of the corresponding pride flag. They can then press “s” to save the image to their device.
Image of “Bi” dragon saved to device
This project was inspired by my identity as a queer Chinese-American who has been immersed in queer culture for years, but did not know the terms in Chinese until very recently. These terms are not taught in language classes, and the queerness is oftentimes not spoken about in Chinese or Chinese diasporic families. Additionally, queer narratives in popular culture are overwhelmingly western-centric. Thus, I wanted to combine Chinese imagery (the color red, the lantern images, guqin music, dragons) with queer imagery to represent this intersection of identities. I hope this program makes the process of learning queer Chinese terms a little more cute and memorable than by just typing them into Google Translate.
Parts I’m Proud Of: The thing I am most proud of in creating this project is not necessarily any single bit of code, since the code is quite basic (to those well versed in p5js —aka not me), but rather stringing it all together to form a functional work of art. In each week’s assignment, I gained individual skills such as drawing with shapes, making classes, and using the mouseClicked and keyPressed functions. The challenge was combining all of these into one project, and at first it kind of felt like starting off on a marathon when you had just learned how to walk. What helped was breaking the project down into chunks, and first establishing the “structure” of the code before actually drawing the details. Here are the steps I followed, roughly in order:
Figuring out how to toggle between two screens with true/false & if statements
Programming the “S” key and back arrow to save image & return to homepage
Creating a Button class and adding functionality (hover and click) to it
Drawing the unchanging elements of the homepage and the dragon page
Creating a Dragon class
Figuring out how to make the scales change color depending on which button was pressed (if I were to do this project again, I would perhaps move step 6 to the beginning and try to figure out the 6 buttons + 6 color schemes while my code was still minimal).
As for the design of the project, I like its minimalistic, cute feel, which I tried to emanate with round rectangle corners, bold Futura font, and pastel colors. I also believe it was a good decision to have the “back” and “save image” functions be keyboard commands, because it decluttered the area under the canvas and foregrounded the LGBTQ+ identity terms that I wanted to highlight.
I also think it was a good idea to have the toggling between scenes controlled by a boolean variable + functions to draw each scene, as it made it very clear which scene was being drawn currently. I had to restructure my program in the middle of my working process, as I realized that it made much more sense to have a dragon class and then call different dragons within one scene than to have different scenes with one dragon each, and call the scenes. Also, originally, the buttons were all on the homepage with only the Chinese terms, which was a bit confusing for the user.
After revising, there was a start button on the homescreen and a “canvas” on the second screen, which brings to mind the universal design of familiar drawing programs, creating a more intuitive experience. I also added English to the buttons, because although my original intent was to omit English and have the user learn the terms by associating them with the pride flag colors, I realized this was not explicit enough and also many people may not know or remember which pride flag colors correspond with which identities.
Problems:
I encountered four main bugs in this code, two are resolved and two are still remaining.
The first bug was with the “press ‘s’ to save image” functionality. The bug was that every key being pressed was triggering the program to save the image, not just the “s” key. The issue was that I had written keyCode === "s" instead of keyCode === 83, as keyCode is only intended to work in natural language with special keys such as LEFT_ARROW, whereas for letter keys you need to find its numerical code. Below is the working code:
function keyPressed() {
// To find keyCode of letter keys:
// console.log(keyCode);
if (keyCode === LEFT_ARROW && homePageVisible === false) {
homePageVisible = true;
scene1Visible = false;
}
if (keyCode === 83 && homePageVisible === false) {
savedImg = myCanvas.get(0, 0, width, height - 115);
savedImg.save("my-pride-dragon", "png");
}
}
The second bug was that the buttons were triggering different scale colors to show up when the user was merely hovering over them, but not clicking them. This was because I was writing code for clicking specific buttons inside the class, which I later learned is unideal because classes are not supposed to know the specifics of each button, they should be as general as possible. Here is the code before, with this section repeated for each button 1-6:
In office hours, I learned that the click within the Button class can be used to make a boolean variable either true or false, and then that true or false can be used in the mouseClicked function to actually detect clicks on different buttons and call each button’s associated actions.
//code for detecting click on buttons to trigger
//different scale colors
if (scene1Visible) {
if (button1.click(mouseX, mouseY)) {
dragon1Visible = true;
dragon2Visible = false;
dragon3Visible = false;
dragon4Visible = false;
dragon5Visible = false;
dragon6Visible = false;
mainDragonVisible = false;
}
The third bug, which I have not resolved yet, is one in which the “bi” dragon (dragon3) sometimes shows up as the “default”, what the user sees when entering sceneOne before clicking on any buttons. By putting print statements in every function where dragon3 is called, I was able to see that that something is happening between the moment the user goes back to the homepage and the moment they returns to sceneOne to call dragon3. All the code for the bi dragon is the same as for the other dragons, so I have not been able to figure out what is causing it, in particular, to be called.
Finally, the background audio does not play when the sketch is embedded into this blog post or in fullscreen mode, but it plays fine when the sketch is run in p5js. I am not sure why this is, but perhaps it could be solved by adding a command for playing the audio on the user’s first mouse click. However, the desired effect is that the audio plays before the user clicks, just as ambient background music.
Areas for Improvement:
There are many areas of improvement, both in terms of efficiency of the code and functions and aesthetics I wish the program had. In terms of efficiency, it would be sleeker to have a “dragon selector” variable that would assign a number to each dragon, so instead of writing all of the code below to ensure that only one dragon is true at a time, I could just call the number of each dragon.
if (scene1Visible) {
if (button1.click(mouseX, mouseY)) {
dragon1Visible = true;
dragon2Visible = false;
dragon3Visible = false;
dragon4Visible = false;
dragon5Visible = false;
dragon6Visible = false;
mainDragonVisible = false;
}
if (button2.click(mouseX, mouseY)) {
dragon1Visible = false;
dragon2Visible = true;
dragon3Visible = false;
dragon4Visible = false;
dragon5Visible = false;
dragon6Visible = false;
mainDragonVisible = false;
}
if (button3.click(mouseX, mouseY)) {
dragon1Visible = false;
dragon2Visible = false;
dragon3Visible = true;
dragon4Visible = false;
dragon5Visible = false;
dragon6Visible = false;
mainDragonVisible = false;
//**bug: sometimes dragon does not set back to
//the default dragon but instead
//shows dragon 3 as the default
//when returning to the drawing page
}
if (button4.click(mouseX, mouseY)) {
dragon1Visible = false;
dragon2Visible = false;
dragon3Visible = false;
dragon4Visible = true;
dragon5Visible = false;
dragon6Visible = false;
mainDragonVisible = false;
}
if (button5.click(mouseX, mouseY)) {
dragon1Visible = false;
dragon2Visible = false;
dragon3Visible = false;
dragon4Visible = false;
dragon5Visible = true;
dragon6Visible = false;
mainDragonVisible = false;
}
if (button6.click(mouseX, mouseY)) {
dragon1Visible = false;
dragon2Visible = false;
dragon3Visible = false;
dragon4Visible = false;
dragon5Visible = false;
dragon6Visible = true;
mainDragonVisible = false;
}
}
}
function keyPressed() {
// code to find keyCode of letter keys:
// console.log(keyCode);
//return to homepage when left arrow is pressed
//reset dragon to default dragon (sometimes does not work because bug)
if (keyCode === LEFT_ARROW && homePageVisible === false) {
homePageVisible = true;
scene1Visible = false;
mainDragonVisible = true;
dragon1Visible = false;
dragon2Visible = false;
dragon3Visible = false;
dragon4Visible = false;
dragon5Visible = false;
dragon6Visible = false;
}
Additionally, since each of my buttons are performing the same hover and click functions, they could perhaps be stored in a button array, instead of being called individually every time:
However, I am glad I did things “the long way” for this project, because it was much easier for me to know what each part of my program was doing when I could see every button I was creating, instead of everything being “hidden” in a for loop. I think making the code more efficient would have come at the cost of me understanding my program as I worked on it.
As for features, I wish I had time to add a key command to change background color, as well as a little “wardrobe” for the dragon (another scene that users could enter to select items, such as sunglasses or a shirt, that they could put on their dragon).
