Adi was my tester for my prototype before finalizing the whole project. My project is very straightforward, so minimal explanation was necessary and he got used to playing the game very quick. He liked that the nodes matched the music that was playing in rhythm and enjoyed the experience in both easier song and the harder song. Although he didn’t get to finish throughout the harder song, “Canon”, but since that is part of the purpose of that song, it worked out as I expected. Again, I felt the need to explain the connection between my two different elements of rhythm game and the whack-a-mole game, so after the user testing, I made a slight change in the UI where the mole picture was changed to a hole, and the shadow node would change to a mole picture when it was apt to hit it for score.
Month: May 2023
Final Project: VR HandVenture
Concept:
As the name of the project suggests, for my final project I created the “VR” gloves that allows the user to control the hand movement and explore different places of Kazakhstan through their panorama images. I have always been fascinated by the beauty of my country, and I want other people to see those beautiful places as well. The destination points that the user can explore include a variety of landscapes, starting from dense forests and mountain lakes, and ending with deserts and canyons.
The reason I named this project “VR HandVenture” is because moving inside the panorama image closely resembles the movement inside a VR world and our right hand is the main instrument through which we can explore the place ( “HandVenture” comes from the word “Adventure”; shoutout to Saamia and Sanjana who helped me come up with this project name that I think perfectly encapsulates the project’s main idea).
After the welcome page and following the instructions page, the user will be prompted to enter the main page, a page displaying Kazakhstan’s map, where he/she later can choose the destination he/she wants to visit. Before clicking on the corresponding destination, a small information about the place, including the name of the place and a short description, will be displayed to the user. The user can explore each destination through moving inside panorama image using the hand movements. The movements that are available to the user are moving left, right and zooming in, out. When the user bends his/her hand to the right the image will move right, when bends to the left the image will move left, and bending index finger will allow the user to zoom in, while bending the thumb will allow to zoom out. When zooming out of the image, after reaching a certain threshold, the user exits the image and reaches a 3d space where the image will now be displayed in form of a sphere.
Implementation:
I attached flex sensors to the glove and used analog input received from them to detect the user’s hand movement. I attached two flex sensors to both sides the glove which are used to sense when the user bends his/her hand right or left and move the panorama image accordingly. I also attached two sensors to the index finger and to the thumb which are used to zoom in and zoom out inside the image. I’ve used pretty useful feedback from the user-testing so I set the limit to how much the user an zoom in and out, as this has been creating problems with the experience because the sensors are too sensitive for detecting when the user continuously bended his/her hand.
Arduino code:
Arduino is used to read analog input from the flex sensors which it then sends to p5.js through serial communication.
const int flexPin1 = A1; //controls movement right
const int flexPin0 = A0; //controls movement left
const int flexPin5 = A5; //controls zooming in
const int flexPin4 = A4; //controls zooming out
int value_left; //save analog value
int value_right;
int value_zoomIn;
int value_zoomOut;
void setup(){
Serial.begin(9600);
//start the handshake
while (Serial.available() <= 0) {
Serial.println("0,0,0,0"); // send a starting message
delay(300);
delay(50);
}
}
void loop(){
// wait for data from p5 before doing something
while (Serial.available()) {
int isMoving = Serial.parseInt();
if (Serial.read() == '\n') {
//read the inputs from flex sensors
value_right = analogRead(flexPin1);
value_left = analogRead(flexPin0);
value_zoomIn = analogRead(flexPin5);
value_zoomOut = analogRead(flexPin4);
delay(5);
//send them to p5.js
Serial.print(value_right);
Serial.print(',');
Serial.print(value_left);
Serial.print(',');
Serial.print(value_zoomIn);
Serial.print(',');
Serial.println(value_zoomOut);
}
}
}
p5.js code:
p5.js code is too large, so I’m embedding the link to my code.
p5.js is handling the main functions of the code such as creating visuals and a panorama-like experience, moving the image according to the value received from the Arduino, creating user interactivity with the map, popping text boxes when the user hovers over the destination rectangles, etc.
Communication between Arduino and p5.js:
Arduino gets the analog input from the flex sensors and sends them to p5.js, which measures that value, and if it exceeds a certain threshold, it moves the panorama image accordingly.
Demo:
Aspects that I’m proud of:
I’m really proud with how everything in this project turned out. I wasn’t really expecting initially that this will resemble a VR experience, so this was a pleasant surprise for me that I discovered during the process of making. I’m also proud with the idea of using gloves to move inside the image as I think that is a pretty cool and novel idea. In terms of the hardware part, I’m glad with how sensors are working as they’re giving pretty accurate answers since they have been placed on right locations on the glove.
Areas for future improvement:
One of the areas for future improvements include adding more destination points to the map, because I have only 6 places so far. Also, I used hot glue to attach the flex sensors to the glove. Even though the connection is good, it may not be long-term so one of the improvements could be sewing the flex senosors to the glove, as it is made of sewable material. In addition to that, it’s highly recommended for the user to hold the hand in a certain position initially (palm facing left side of the body and fingers pointing outwards) so that sensors would work best, and I’m planning on explaining it to user by demonstrating the hand position. One of the better ways to do this for the future would be recording a demo and attaching a video to the instructions page or adding a picture/gif with the corresponding hand posture.t
Explore Nature’s Elements – Final Project by Hana
For the final project, I decided to create an experience which involves interactivity through buttons, led lights, a fan and a screen. During finals week we have so much to work on that we neglect nature and its healing powers, so I decided to bring that indoors and onto our screens as a way to create an enjoyable and relaxing experience.
This project consists of the screen display part which is coded on P5, and the user interactivity with buttons, a potentiometer, a fan and the neopixel ring lights. Here is a demo to show the experience better:
Interaction
One can switch through the different elements to experience them: rain, fire, or earth&wind. For a more immersive feel beyond the simulations on the screen, I decided to have light shows on a neopixel ring with patterns which imitate what is happening on the screen. The magnitude of the potentiometer affects what is happening in the simulations.
Higher magnitude in rain leads to more raindrops and at higher speeds. In the lightshow this translates to the raindrops rotating faster. The fan is activated during rain, as wind is often associated with rain.
Higher magnitude in fire leads to more fire particles and a bigger spread (essentially more fire). In the lightshow this is shown by how high the fire reaches the top of the ring. The fan is off at this time.
Magnitude in earth&wind affects the direction of the wind on the screen. The light show is unrelated to the work of the potentiometer, as the light show is imitating the piling of the leaves on the screen, which happens at a constant rate regardless of the values of the potentiometer. It is interesting to note that the constant rate of the leaf piling creates an hourglass effect on both the screen and the neopixel ring (the pile gets higher after a constant x amount of time). The fan is turned on to imitate the wind.
Arduino
The arduino works by taking as input the values of the switches and the potentiometers, and outputting the neopixel light show patterns as well as turning the fan on and off when appropriate.
The Fan
The fan required a higher voltage than the 5V that arduino supplies, so in order to get it to work I had to use a different power source and use a transistor in order to control the fan through the arduino. As for code, I simply had to do a digitalWrite of HIGH on the appropriate pin when I wanted it on and digitalWrite of LOW when I wanted it off.
Light Shows
I had to come up with my own patterns for the light shows to imitate best what I had come up with for the screen. Because of the ring shape of the neopixel, I had to get very creative, since the screen is a rectangle.
For the raindrops, I decided to create 4 raindrops which go around in a clockwise direction, and the magnitude of the rain would affect the speed at which they go around. To create a more interesting effect, I created the raindrops as a sort of trail, where the pixel at the front was the brightest, and the 2 pixels behind it each get dimmer, followed by fully dim pixels to distinguish between the rain drops. Here is the code for this pattern:
void raindrops(int del) {
//map potentiometer value to del
del = map(del, 40, 1020, 150, 30);
// a total of 24 pixels
// separate into 4 sections by tackling 6 pixels at a time
for(int i=0; i<6; i++) {
//first pixels in the section are dark
pixels.setPixelColor(i, pixels.Color(0, 0, 0));
pixels.setPixelColor(i+6, pixels.Color(0, 0, 0));
pixels.setPixelColor(i+12, pixels.Color(0, 0, 0));
pixels.setPixelColor(i+18, pixels.Color(0, 0, 0));
//second pixels in the section are dark
pixels.setPixelColor(i+1, pixels.Color(0, 0, 0));
pixels.setPixelColor(i+7, pixels.Color(0, 0, 0));
pixels.setPixelColor(i+13, pixels.Color(0, 0, 0));
pixels.setPixelColor(i+19, pixels.Color(0, 0, 0));
//third pixels in the section are dark
pixels.setPixelColor(i+2, pixels.Color(0, 0, 0));
pixels.setPixelColor(i+8, pixels.Color(0, 0, 0));
pixels.setPixelColor(i+14, pixels.Color(0, 0, 0));
pixels.setPixelColor(i+20, pixels.Color(0, 0, 0));
//fourth pixels in the section are dim
pixels.setPixelColor(i+3, pixels.Color(20, 20, 20));
pixels.setPixelColor(i+9, pixels.Color(20, 20, 20));
pixels.setPixelColor(i+15, pixels.Color(20, 20, 20));
pixels.setPixelColor(i+21, pixels.Color(20, 20, 20));
//fifth pixels in the section are brighter
pixels.setPixelColor((i+4)%NUMPIXELS, pixels.Color(100,100,100));
pixels.setPixelColor((i+10)%NUMPIXELS, pixels.Color(100,100,100));
pixels.setPixelColor((i+16)%NUMPIXELS, pixels.Color(100,100,100));
pixels.setPixelColor((i+22)%NUMPIXELS, pixels.Color(100,100,100));
//sixth pixels (at the front) in the section are the brightest
pixels.setPixelColor((i+5)%NUMPIXELS, pixels.Color(200, 200, 200));
pixels.setPixelColor((i+11)%NUMPIXELS, pixels.Color(200, 200, 200));
pixels.setPixelColor((i+17)%NUMPIXELS, pixels.Color(200, 200, 200));
pixels.setPixelColor((i+23)%NUMPIXELS, pixels.Color(200, 200, 200));
pixels.show(); // Send the updated pixel colors to the hardware.
delay(del); //delay affects how fast the raindrops are moving
}
}
For the fire, I decided to have the fire start at the bottom of the ring, and get reach higher the higher the value of the potentiometer. To create a realistic fire crackling effect, I had to experiment a lot with colors and dimness, the colors at the bottom had to be more yellow and brighter, and the colors at the top had to be more red and dimmer – and this all had to be relative to the total height of the fire. To achieve this, I had to do a lot of calculations and a lot of experimenting until I got it right. Here is the code which gave me the desired effect:
void fire(int maxNum){ //maxNum should be 4 to 12
//maxNum refers to the height of the fire. the number corresponds
//to how many pixels will be colored on each side.
maxNum = map(maxNum, 40, 1024, 4, 12);
//changing delay so that the fire crackles and changes more
//the larger the fire is
DELAYVAL = map(maxNum, 4, 12, 50, 25);
for(int i=0; i<maxNum; i++) {
//calculating the dimness in relation to how many pixels there are
//in total
double percentage=1-0.1*map(i, 0, maxNum, 0, 10);
//do not dim the first maxNum/3 pixels
//the fire at the bottom must be brightest
if(i>maxNum/3){
// pixels.setPixelColor(i, pixels.Color(int(random(100,255)*0.1*(maxNum-i)), int(random(40)*0.1*(maxNum-i)), 0));
// pixels.setPixelColor(NUMPIXELS-i, pixels.Color(int(random(100,255)*0.1*(maxNum-i)), int(random(40)*0.1*(maxNum-i)), 0));
pixels.setPixelColor(i, pixels.Color(int(random(100,255)*percentage), int(random(40)*percentage), 0));
pixels.setPixelColor(NUMPIXELS-i, pixels.Color(int(random(100,255)*percentage), int(random(40)*percentage), 0));
}
else{
//full brightness
pixels.setPixelColor(i, pixels.Color(random(150,255), random(10,60), 0));
pixels.setPixelColor(NUMPIXELS-i, pixels.Color(random(150,255), random(10,60), 0));
}
pixels.show(); // Send the updated pixel colors to the hardware.
delay(DELAYVAL); // Pause before next pass through loop
}
//turn off the pixels above the current height, if the height decreases
for(int i=maxNum; i<13; i++) {
pixels.setPixelColor(i, pixels.Color(0,0,0));
pixels.setPixelColor(NUMPIXELS-i, pixels.Color(0,0,0));
}
}
For the leaves, I wanted to imitate the pile of leaves which is being accumulated on the screen. So I get the level of pile that has been reached from P5, and light up a corresponding number of pixels. I use alternating shades of pink to imitate the colors of the leaves on the screen, which are also alternating. Here is the code:
void leaves(int leavesNum) {
for(int i=0; i<leavesNum; i++) {
//turn on fan
digitalWrite(fan, HIGH);
//light up two pixels at a time on each side
//left side
pixels.setPixelColor(2*i, pixels.Color(255, 0, 144));
pixels.setPixelColor(2*i+1, pixels.Color(255, 69, 174));
//right side
pixels.setPixelColor(NUMPIXELS-2*i, pixels.Color(255, 0, 144));
pixels.setPixelColor(NUMPIXELS-2*i+1, pixels.Color(255, 69, 174));
pixels.show();
delay(1);
}
}
Beyond that in arduino, I receive values from the buttons to determine which simulation to experience, and display the appropriate light show, and turn on and off the fan whenever needed. I will discuss the arduino/P5 connection below.
P5
For the simulations, I utilized objects and arrays to create them. If you notice, each of the simulations is made up of small similar particles which are created and disappear in different ways. We have raindrops, fire particles and leaves. The code for each of the classes is quite similar, with small differences to create the different simulations, as the particles look and behave differently for each of the simulations, but they follow a similar structure. Here is the code for one of them:
class RainDrop {
constructor() {
//randomly decide horizontal location
this.x = random(0, width);
//start at the top of the screen
this.y = 0;
//random vertical speed
this.vy = random(5,8);
//random vertical diameter
this.d = random(5,8);
//color variation, create different shades
//of light blue
this.R = random(190,200);
this.G = random(190, 200);
this.B = random(200,255);
}
//finished when they reach the bottom of the screen
finished() {
return this.y > 600;
}
//update postition based on vertical speed
update() {
//add additional boost based on the value of the
//potentiometer to increase vertical speed
this.y += this.vy + map(potVal, 100, 1020, 0, 3);
}
//drawing out the particle
show() {
noStroke();
fill(this.R, this.G, this.B);
ellipse(this.x, this.y, 5, this.d);
}
}
And here is how the objects are created, updated and deleted:
function create_rain() {
//redraw rain background
image(rain_bg, 0, 0, 600, 400);
//max num of raindrops created at a time
//determined by the potentiometer
let maxI = map(potVal, 40, 1020, 2, 10);
if(frameCount%5==0){
for (let i = 0; i < maxI; i++) {
let p = new RainDrop();
rain_drops.push(p);
}
}
//update raindrops, and delete them when they are finished
for (let i = rain_drops.length - 1; i >= 0; i--) {
rain_drops[i].update();
rain_drops[i].show();
if (rain_drops[i].finished()) {
rain_drops.splice(i, 1);
}
}
}
Since the code for the other simulations follows a similar structure, I will not attach it so as not to be repetitive.
P5/Arduino Communication
At first a handshake is established to ensure communication exists before proceeding with the program:
//arduino code
while (Serial.available() <= 0) {
Serial.println("-1"); // send a starting message
delay(300); // wait 1/3 second
}
Then, the arduino constantly sends the values of the potentiometer and the values of which button was pressed last. It reads from P5 the value of the leaf pile, which it uses later for the light show:
int leavesNum = Serial.parseInt();
if (Serial.read() == '\n') {
Serial.print(potPosition);
Serial.print(",");
Serial.println(currentButton);
}
Here is how P5 reads these values and also sends the leaf pile value:
// This function will be called by the web-serial library
// with each new *line* of data. The serial library reads
// the data until the newline and then gives it to us through
// this callback function
function readSerial(data) {
////////////////////////////////////
//READ FROM ARDUINO HERE
////////////////////////////////////
if (data != null) {
// make sure there is actually a message
// split the message
let fromArduino = split(trim(data), ",");
// if the right length, then proceed
if (fromArduino.length == 2) {
// store values
//potentiometer value
potVal = fromArduino[0];
currentButton = fromArduino[1];
}
//////////////////////////////////
//SEND TO ARDUINO HERE
//////////////////////////////////
let sendToArduino = leaf_bg_choice+1 + "\n";
writeSerial(sendToArduino);
}
}
Strengths
As a computer science student, I am comfortable with programming. For this project I wanted to challenge myself creatively. From the creativity of the idea itself, to the screen simulations, light shows and even the crafty parts of assembling the interactive box – I pushed myself creatively as much as I could. This was an incredible learning experience for me. I learned that I can be quite resourceful even with hand work such as creating the box out of cardboard and paper, creating a cardboard ring and pinning down cloth to create a ‘lampshade’ for the neopixel. I also had so many connections and circuits, and that lead to many challenges, but it also taught me so much.
As for the user experience, I think it serves its purpose quite well – it is very simple in its usage, that the testers which you will see below required no instruction before or after to learn how to use it. And it is quite a relaxing experience, you just get to play around and enjoy the different elements, and enjoy my creative efforts at visual arts – whether it is the screen simulations or the light shows. The testers particularly enjoyed the light shows, and were very pleasantly surprised by the fan.
Weaknesses
While working on this project I ran into problem after problem. Starting with the problems I anticipated well in advance, such as learning how to use a neopixel, learning how to connect a fan and learning how to solder. These took only a matter of seeking out help from my professor and/or the internet depending on the task – so they were easier challenges to overcome, I just needed to educate myself on how to do them.
The unexpected problems were quite tough. Since I am dealing with arrays of objects in my code, I had to work on optimizing the code so I do not overwork the computer and have it result to low fps. I already had plenty of experience with that in my midterm project, so this was a relatively easy fix too.
My biggest problem was with the potentiometer. Unfortunately, I am still unsure that I solved the issue at the root – but I learned how to avoid the problem. As long as I arrange my circuits evenly, and keep the wires neat and straight, the potentiometer will give me values that are stable enough for my purposes. But the problem is that the values are not entirely stable. For my purposes though, the problem is somewhat solved – at least enough for my project to work fine.
I believe a lot of the problem from above came from using a breadboard, if I used soldering instead I would have a more stable connection and thus more stable values. In a case where this was a more permanent installation, I would most definitely have to switch out of the breadboard – but this works currently.
User testing
I had two people come and test the project with no instructions beforehand. Both of them had no problems figuring out exactly how the experience works and being able to play around with it exactly as intended. They understood exactly what the buttons and the potentiometer did and would check the screen and the neopixel for the results of their work with the potentiometer.
The simplicity of the interactions worked very well, and both testers reported to have had an enjoyable and relaxing experience.
One area of improvement is with the buttons. The buttons themselves are not very sensitive, and the cardboard box is soft, so the user has to push hard to change the simulation. I put support right underneath the buttons to combat this, but it still wasn’t enough to entirely eliminate the inconvenience. However, the user does not need to push too hard, so it is easy to get used to pushing it the right amount – for my own tests I experienced no problems, but people using it for the first time faced a few challenges.
To fix the responsiveness of the buttons, I would switch out the cardboard box for something stiffer and stronger, like acrylic or wood.
Link to code
Are You Stressed? – Final Project
Concept:
For my final project, I created an interactive tool that helps to visualize stress levels during finals seasons. It allows college students to relieve stress through squeezing and screaming and produce a unique visual mapping of their stress. The tool incorporates a microphone and a stress ball on a board. The pressure and sound input are used to determine stress levels based on duration and intensity, and the results are displayed on the screen as moving color dots. The more stressed you are, the bigger and redder your circles are and the faster they move.
Implementation:
The interaction is straightforward – the nature of a stress ball makes it obvious that one needs to squeeze it and to guide the users to use the microphone, I put a little “Talk to me :)” label on top of it. The users interact with the project by either squeezing the stress ball or screaming (or talking, but preferably screaming) into the microphone.
The arduino part of the code is very straightforward as well – arduino simply takes the input values, averages over 10 consecutive values to smooth out the readings, and sends the values to P5JS.
Without interaction with Arduino, the P5JS code creates circles on the screen with randomized but bounded positions and radii and makes them move up the screen again, with randomized but bounded speed. The circles out of the frame are removed and every 100 frames there is a new circle generated.
Color circles when the user is not squeezing the stress ball
P5JS takes the input values and maps them onto 3 variables: the radius, color, and speed of all the circles present on the screen. The color and size of the circles are dependent on the force sensor’s readings. When the stress ball is squeezed, the dots get enlarged and change their color to red.
Color circles when the user squeezes the ball
The code for displaying a single circle is as follows:
display() {
let radius = map(fVal, 0, 1023, 0, this.size);
let colorR = map(fVal, 0, 1023, 0, 255);
let colorB = map(fVal, 0, 1023, 255, 0);
let colorG = 0;
for (let i = 0; i <= radius; i=i+2) {
noStroke();
fill(color(colorR, colorG, colorB, 2));
ellipse(this.x, this.y, i, i*this.f);
}
}
The speed of the color circles depends on the value of the microphone. I found the base rate of a room is under 600 on the microphone reading, so I set the detection threshold to 600. To control the randomized but bounded speed of the circles, I added a variable speedFactor that I use when changing the speed.
//Defining global variables related to speed
let minSpeed = 1;
let maxSpeed = 5;
let speedFactor = 1;
//randomizing the speed for each circle
let s = random(minSpeed, maxSpeed)/4;
//update function in the Circle class that updates based on the speedfactor
update() {
this.y -= this.speed*speedFactor;
}
//changing the speedfactor according to the microphone reading (and by using speedfactor changing all of the randomized speeds by a constant scale)
if(micVal>600){
speedFactor = micVal/600;
}else{
speedFactor = 1;
}
//NOTE these are snippets from different parts of the code
Challenges
I had 2 big challenges in this project. First, it was connecting the hardware as it was my first time soldering or building a shell for my circuit. I ended up messing up some sensors and wires, and in the end not using them but sticking to a very simplistic design of my sensors on abox.
Another big challenge was making the gradient circle in P5JS that fades toward the edges. I tried 4-5 different methods, including drawing just the outline of a circle, using the HSB color mode, lerp function, and the one that I stuck with – drawing a lot of circles of the same color and very small alpha value so that the layering of many circles creates the saturated color in the middle. I am still not fully satisfied with this method though as when there are more than 15 circles on the screen, it gets very laggy and there are way too many objects on the screen (from a computational perspective, not a visual). I would love to find a better, more efficient way to do this without having to draw 4000 (number of circle groups, 10 in my case)*(maxRadius, 400 in my case) circles in each frame.
However, I am still proud of my final project and really like the idea of using squeezable objects (like the stress ball I used) to be part of computer controlling (I also created a very small version of a flappy bird game that uses the squeezer as a controller, which is simplistic but really fun).
User Testing
Without instructions, users were able to figure out that they are supposed to squeeze the stress ball and speak into the mic. The connection between the stress ball and the color circles on the screen was pretty obvious, but the positioning of the microphone made it a little awkward to both watch and interact with the tool, which could be an area for improvement.
Final Project Final Documentation
Concept and Design:
For my final project, I created a table piano using the ultra sonic sensor and connected it to the P5js interface. The start screen allows users to choose between three modes: free mode, game mode, and record mode.
In free mode, users can play the piano for as long as they want and try to figure out a tune on their own. Every time a note is pressed on the piano, the P5js interface shows a different picture for each note so that the experience is more immersive. The free mode allows the user to practice their skills and be as creative as they want to be.
Record mode as the name suggests records and store each key played by the user and the concatenation of these notes then can be played back in the form of a tune. This allows the user to listen back to what they play and make amends to what needs to be changed in the future. This proves to be a good tool for those who like to be precise and perfect.
In game mode, numbers fall from the sky, and the user would have to press the appropriate number on the table piano. The game eventually keeps getting harder as the score increases. The game gets harder in the form of game speed and the spacing between the numbers. Every un-pressed note/number will result in losing a life and after a certain number of lives lost, the game ends. The current score of the user and the high score of the user is also displayed at the same time encouraging the users to get more competitive in a healthy way. The game mode is basically created to add some fun to piano playing and also to build muscle memory of where what keys are and strong reflexes. This would in turn help the user with reading off of music sheets while playing actual pianos.
Final Product
Interaction Design:
The project is designed in a way that makes it much more interactive as compared to my previous projects in the class. From the hardware connected to Arduino to the software element in P5js, everything is interactive. The main theme of the project is a table piano and users have to use their hands to play keys and notes. This allows the user to get a different feel of the project as compared to simple buttons on a keyboard or keys on a touch screen phone. When it comes to the P5js user interface, it is designed in a way, that gives full control to the user. There are buttons that change color when you hover over it, suggesting that it is a button. There are instruction screens in all three modes and all the screens have back buttons as well that allow the user to go back to the home screen. Furthermore, before recording, stopping while recording, or playing the recorded tone, the user can use interactive buttons provided to control all of these functions. The user can also see their score going up and lives decreasing while playing the game. In the end, the game has an option of being restarted as well.
Description of P5js code:
The P5js code is developed using Object Oriented Programming to keep the work flow easier, manageable, more efficient. Two classes have been created: the noteDrop class and the Game class. The noteDrop class handles the movements and functionality of a single falling note. On the other hand, the game class is where all the game logic is implemented. The objects of the noteDrop class are stored in an array and then the array is used in various ways depending on the game state.
To keep track of the different game states, variables and boolean values have been used. These variables are updated based on user input and the game logic. Additionally, messages are sent to and received from Arduino to synchronize the game state. This allows for better user experience where the hardware and software work in perfect harmony.
To ensure a smooth flow of the game, I have also implemented a preload function that loads all the necessary assets before the game begins. This helps to prevent any issues with lag or delays during gameplay.
In order to make the code more manageable and easier to understand, the game has been divided into functions and different game states. This also allows for easier editing and maintenance of the codebase. Here is the code:
new p5(); //to counter the issue with random function
//declare global variables
let highscore = -1;
let globalGameSpeed = 1.5;
let gameObject;
let menuBackgroundPicture1;
let menuBackgroundPicture2;
let menuBackgroundPicture3;
let menuBackgroundPicture4;
let endBackgroundPicture;
let recordScreen1;
let recordScreen2;
let recordScreen3;
let recordScreen4;
let recordScreen5;
let freeScreen0;
let freeScreen1;
let freeScreen2;
let freeScreen3;
let freeScreen4;
let freeScreen5;
let freeScreen6;
let freeScreen7;
let freeScreen8;
let freeScreen9;
let freeScreen10;
let freeScreen11;
let freeScreen12;
let freeScreen13;
let freeScreen14;
let freeScreen15;
let freeScreen16;
let gameScreenBG;
let gameInstructions;
let num1;
let num2;
let num3;
let num4;
let num5;
let num6;
let num7;
let num8;
let num9;
let num10;
let num11;
let num12;
let num13;
let num14;
let num15;
let num16;
let playfair_font; //declare the font named playfair
let background_music;
function preload() //preload all the assets
{
menuBackgroundPicture1 = loadImage("menu_background1.png");
menuBackgroundPicture2 = loadImage("menu_background2.png");
menuBackgroundPicture3 = loadImage("menu_background3.png");
menuBackgroundPicture4 = loadImage("menu_background4.png");
endBackgroundPicture = loadImage("end_game.png");
playfair_font = loadFont("playfairDisplay_font.ttf");
recordScreen1 = loadImage("record_instruction1.png");
recordScreen2 = loadImage("record_instruction2.png");
recordScreen3 = loadImage("record_instruction3.png");
recordScreen4 = loadImage("record_instruction4.png");
recordScreen5 = loadImage("record_instruction5.png");
freeScreen0 = loadImage("free01.png");
freeScreen1 = loadImage("free1.png");
freeScreen2 = loadImage("free2.png");
freeScreen3 = loadImage("free3.png");
freeScreen4 = loadImage("free4.png");
freeScreen5 = loadImage("free5.png");
freeScreen6 = loadImage("free6.png");
freeScreen7 = loadImage("free7.png");
freeScreen8 = loadImage("free8.png");
freeScreen9 = loadImage("free9.png");
freeScreen10 = loadImage("free10.png");
freeScreen11 = loadImage("free11.png");
freeScreen12 = loadImage("free12.png");
freeScreen13 = loadImage("free13.png");
freeScreen14 = loadImage("free14.png");
freeScreen15 = loadImage("free15.png");
freeScreen16 = loadImage("free16.png");
gameScreenBG = loadImage("gameModeBG.png");
gameInstructions = loadImage("gameModeinstructions.png");
num1 = loadImage("num1-removebg-preview.png");
num2 = loadImage("num2-removebg-preview.png");
num3 = loadImage("num3-removebg-preview.png");
num4 = loadImage("num4-removebg-preview.png");
num5 = loadImage("num5-removebg-preview.png");
num6 = loadImage("num6-removebg-preview.png");
num7 = loadImage("num7-removebg-preview.png");
num8 = loadImage("num8-removebg-preview.png");
num9 = loadImage("num9-removebg-preview.png");
num10 = loadImage("num_a-removebg-preview.png");
num11 = loadImage("num_b-removebg-preview.png");
num12 = loadImage("num_c-removebg-preview.png");
num13 = loadImage("num_d-removebg-preview.png");
num14 = loadImage("num_e-removebg-preview.png");
num15 = loadImage("num_f-removebg-preview.png");
num16 = loadImage("num_g-removebg-preview.png");
background_music = loadSound("music2.mp3");
}
class NoteDrop //noteDrop class that commands the movement of each note
{
constructor(x, y, noteWidth, noteHeight, noteNumber, canvasWidth, canvasHeight) //constructor initializes everything
{
this.note_x = x;
this.note_y = y;
this.note_height = noteHeight;
this.note_width = noteWidth;
this.canvas_width = canvasWidth;
this.canvas_height = canvasHeight;
this.visible = false;
this.note_number = noteNumber;
}
movement() //movement of the note from top to bottom of the canvas
{
this.note_y = this.note_y + globalGameSpeed; //
if ((this.note_y + this.note_height >= 0) && (this.note_y + this.note_height <= 20)) //as soon as the bottom of the note enters the canvas
{
this.visible = true; //make it visible and this allows for catching it
}
}
display() //displays the note
{
this.movement();
if (this.visible == true) //only display the note if it is visible
{
if (this.note_number == 1) //for each note
{
image(num1, this.note_x, this.note_y); //display its appropriate picture
}
else if (this.note_number == 2)
{
image(num2, this.note_x, this.note_y);
}
else if (this.note_number == 3)
{
image(num3, this.note_x, this.note_y);
}
else if (this.note_number == 4)
{
image(num4, this.note_x, this.note_y);
}
else if (this.note_number == 5)
{
image(num5, this.note_x, this.note_y);
}
else if (this.note_number == 6)
{
image(num6, this.note_x, this.note_y);
}
else if (this.note_number == 7)
{
image(num7, this.note_x, this.note_y);
}
else if (this.note_number == 8)
{
image(num8, this.note_x, this.note_y);
}
else if (this.note_number == 9)
{
image(num9, this.note_x, this.note_y);
}
else if (this.note_number == 10)
{
image(num10, this.note_x, this.note_y);
}
else if (this.note_number == 11)
{
image(num11, this.note_x, this.note_y);
}
else if (this.note_number == 12)
{
image(num12, this.note_x, this.note_y);
}
else if (this.note_number == 13)
{
image(num13, this.note_x, this.note_y);
}
else if (this.note_number == 14)
{
image(num14, this.note_x, this.note_y);
}
else if (this.note_number == 15)
{
image(num15, this.note_x, this.note_y);
}
else if (this.note_number == 16)
{
image(num16, this.note_x, this.note_y);
}
}
}
}
class Game //game class contains all the functionality and game attributes
{
constructor()
{
//defines canvas dimensions
this.canvasWidth = 800;
this.canvasHeight = 600;
//defines note dimensions
this.noteWidth = 75;
this.noteHeight = 75;
this.gameHealth = 10;
this.gameScore = 0;
//these arrays contain various note numbers. They are two different arrays in order to avoid overlap of notes
this.notes1 = [1, 2, 3, 4, 5, 6, 7, 8];
this.notes2 = [9, 10, 11, 12, 13, 14, 15, 16];
this.noteArray = []; //contains the arrays that are displayed
this.recordArray = []; //contains the array that is recorded (not used since array now recorded in arduino)
//boolean variables to control the game
this.recordBool = false;
this.playBool = false;
this.playingScreen = false;
this.instructionsBool = true;
this.backgroundNumber = 0;
this.arrayCounter = 0;
this.notesInAnArray = 8;
this.clickCount = 0;
//dimensions of various buttons
this.menuButtonWidth = 282
this.menuButtonHeight = 56;
this.menuButtonX = 38;
this.menuButton1Y = 357;
this.menuButton2Y = 445;
this.menuButton3Y = 533;
this.recordButtonX = 212;
this.recordButtonY = 369;
this.recordButtonWidth = 393;
this.recordButtonHeight = 80;
this.homeButtonX = 22;
this.homeButtonY = 33;
this.instructionButtonX = 690;
this.instructionButtonY = 33;
this.homeButtonWidth = 95;
this.homeButtonHeight = 40;
//used to increase game speed
this.difficultyParameter = 20;
this.gameState = "menuScreen";
this.noteSpacing = 2.5; //this is the random spacing between notes in order to increase difficulty
for (let i = 0; i < this.notesInAnArray; i++) //constructs and initializes note objects
{
this.noteArray[i] = new NoteDrop(random(0, this.canvasWidth - this.noteWidth), (i + 1)*(-this.noteHeight * this.noteSpacing), this.noteWidth, this.noteHeight, this.notes1[i], this.canvasWidth, this.canvasHeight);
}
}
rainMovement() //rain like movement of notes from the top of canvas to the bottom
{
for (let i = 0; i < this.notesInAnArray; i++)
{
this.noteArray[i].display();
fill(0,0,0);
if (this.noteArray[i].note_y > this.canvasHeight && this.noteArray[i].visible == true) //if the note is not caught and goes below canvas
{
this.gameHealth = this.gameHealth - 1; //you lose a life
if (this.gameHealth == 0) //if the lives go to zero
{
this.gameState = "endScreen"; //the game ends
}
this.noteArray[i].visible = false; //the note also becomes invisible
}
if (this.noteArray[i].note_y > this.canvasHeight) //as soon as the note goes below the canvas
{
this.arraySwitch(i); //switch its number and it should re-emerge at the top of the canvas
}
}
}
arraySwitch(indexOfArray) //this function changes the note numbers in order to make them random and avoid overlap
{
if (this.arrayCounter % 2 == 0) //arrayCounter keeps track of what array of notes is being used
{
//if array counter is even that means first array is being used and so
this.noteArray[indexOfArray].note_number = this.notes2[indexOfArray]; //you change the array note numbers and get them from notes2 array
this.noteArray[indexOfArray].note_y = this.noteArray[this.notesInAnArray -1].note_y - ((indexOfArray + 1) * (this.noteHeight * this.noteSpacing)); //change the y value and ensure all of them are almost equidistant and there is no overlap
this.noteArray[indexOfArray].note_x = random(0, this.canvasWidth - this.noteWidth); //the x coordinates are random
this.noteArray[indexOfArray].visible = false; //ensure that the visibility is off
}
else
{
//same thing here but the notes are taken from the first array instead of the second one
this.noteArray[indexOfArray].note_number = this.notes1[indexOfArray];
this.noteArray[indexOfArray].note_y = this.noteArray[this.notesInAnArray -1].note_y - ((indexOfArray + 1) * (this.noteHeight * this.noteSpacing));
this.noteArray[indexOfArray].note_x = random(0, this.canvasWidth - this.noteWidth);
this.noteArray[indexOfArray].visible = true;
}
if (indexOfArray == this.notesInAnArray - 1) //when all the 8 notes in an array have gone beneath the canvas,
{
shuffle(this.notes1, true); //shuffle both arrays and
shuffle(this.notes2, true);
this.arrayCounter = this.arrayCounter + 1; //increment the counter to now go for the other array
}
}
recordNotes(valueOfNote) //records the value of notes and appends them to the array (not used)
{
append(this.recordArray, valueOfNote);
}
playRecordedNotes() //plays all the notes that are recorded
{
let noteString = "102\n"; //sends a message to arduino and allows arduino to know when to play the notes
writeSerial(noteString);
this.playBool = true; //the game is now in playing mode
this.recordBool = false; //and not in recording mode
}
display() //display function of the whole game
{
if (this.gameState == "menuScreen") //if the game is in menu mode
{
image(menuBackgroundPicture1, 0, 0); //display the menu picture
//and if the mouse hovers over a button, change the color of the button
if (mouseX >= this.menuButtonX && mouseX <= this.menuButtonX + this.menuButtonWidth && mouseY >= this.menuButton1Y && mouseY <= this.menuButton1Y + this.menuButtonHeight)
{
image(menuBackgroundPicture2, 0, 0);
}
else if (mouseX >= this.menuButtonX && mouseX <= this.menuButtonX + this.menuButtonWidth && mouseY >= this.menuButton2Y && mouseY <= this.menuButton2Y + this.menuButtonHeight)
{
image(menuBackgroundPicture3, 0, 0);
}
else if (mouseX >= this.menuButtonX && mouseX <= this.menuButtonX + this.menuButtonWidth && mouseY >= this.menuButton3Y && mouseY <= this.menuButton3Y + this.menuButtonHeight)
{
image(menuBackgroundPicture4, 0, 0);
}
}
else if (this.gameState == "gameScreen") //if game mode is on
{
if (this.instructionsBool == true) //see if the instructions page is up
{
image(gameInstructions, 0, 0); //provide instructions to the user
}
else //else it would be game mode
{
image(gameScreenBG, 0, 0); //add the background to the game
this.rainMovement(); //make sure the notes keep falling
textSize(15);
fill(255,255, 255);
text("SCORE: ", 10, 20); //display score
text(this.gameScore, 75, 20);
text("LIVES: ", this.canvasWidth - 87, 20); //display lives
text(this.gameHealth, this.canvasWidth - 30, 20);
}
}
else if (this.gameState == "freeScreen") //if the game is in free mode
{
this.freeModeBackgrounds(); //display different backgrounds for different notes played
}
else if (this.gameState == "recordScreen") //if the game is in record mode
{
if (this.recordBool == false && this.playBool == false) //see if its the first page
{
image(recordScreen1, 0, 0); //then display the "start recording" image
if (mouseX >= this.recordButtonX && mouseX <= this.recordButtonX + this.recordButtonWidth && mouseY >= this.recordButtonY && mouseY <= this.recordButtonY + this.recordButtonHeight)
{
image(recordScreen2, 0, 0); //hovering over the button would change the button color
}
}
else if (this.recordBool == true && this.playBool == false) //if the button is pressed, then start recording
{
let valueOfState = "98\n"; //tell arduino that the game is now in recording mode so that it changes state as well
writeSerial(valueOfState);
image(recordScreen3, 0, 0); //show image that says stop recording and play
if (mouseX >= this.recordButtonX && mouseX <= this.recordButtonX + this.recordButtonWidth && mouseY >= this.recordButtonY && mouseY <= this.recordButtonY + this.recordButtonHeight)
{
image(recordScreen4, 0, 0); //hovering over the button changes the button color
}
}
else if (this.recordBool == false && this.playBool == true) //when the button is pressed that says stop recording
{
image(recordScreen5, 0, 0); //image is display that says playing
this.playRecordedNotes(); //recorded notes are then played
}
}
else if (this.gameState == "endScreen") //when the lives are lost and game ends
{
let stateOfArduino = "103\n"; //tell the arduino that game has ended (sets arduino back to the start position)
writeSerial(stateOfArduino);
image(endBackgroundPicture, 0, 0); //show game end image
if (this.gameScore > highscore) //calculate if you have a highscore
{
highscore = this.gameScore; //if yes then change the highscore value to your current score
}
fill(255, 255, 255);
textSize(23);
text("HIGH SCORE: ", 295, 365); //display high score
text(highscore, 460, 365);
text("YOUR SCORE: ", 295, 475); //display your score
text(this.gameScore, 460, 475);
}
else
{
background(255, 0, 0);
}
}
backgroundVariableChange(keyValue) //changes the background variable based on the data received
{
this.backgroundNumber = keyValue;
}
freeModeBackgrounds() //the background variable is used to display appropriate pictures for each note
{
if (this.backgroundNumber == 0) //every note has a different picture associated with it
{
image(freeScreen0, 0, 0);
}
else if (this.backgroundNumber == 1)
{
image(freeScreen1, 0, 0);
}
else if (this.backgroundNumber == 2)
{
image(freeScreen2, 0, 0);
}
else if (this.backgroundNumber == 3)
{
image(freeScreen3, 0, 0);
}
else if (this.backgroundNumber == 4)
{
image(freeScreen4, 0, 0);
}
else if (this.backgroundNumber == 5)
{
image(freeScreen5, 0, 0);
}
else if (this.backgroundNumber == 6)
{
image(freeScreen6, 0, 0);
}
else if (this.backgroundNumber == 7)
{
image(freeScreen7, 0, 0);
}
else if (this.backgroundNumber == 8)
{
image(freeScreen8, 0, 0);
}
else if (this.backgroundNumber == 9)
{
image(freeScreen9,0, 0);
}
else if (this.backgroundNumber == 10)
{
image(freeScreen10, 0, 0);
}
else if (this.backgroundNumber == 11)
{
image(freeScreen11, 0, 0);
}
else if (this.backgroundNumber == 12)
{
image(freeScreen12, 0, 0);
}
else if (this.backgroundNumber == 13)
{
image(freeScreen13, 0, 0);
}
else if (this.backgroundNumber == 14)
{
image(freeScreen14, 0, 0);
}
else if (this.backgroundNumber == 15)
{
image(freeScreen15, 0, 0);
}
else if (this.backgroundNumber == 16)
{
image(freeScreen16, 0, 0);
}
}
notePressed(keyValue) //this is used in game Mode - keyValue is the key pressed on the table piano sent by arduino to p5js
{
for (let i = 0; i < this.notesInAnArray; i++) //in all the arrays
{
if (this.noteArray[i].visible == true) //if the note is visible (on screen)
{
if (keyValue == this.noteArray[i].note_number) //check if the key pressed matches a note
{
this.gameScore = this.gameScore + 1; //increase the game score
this.noteArray[i].visible = false; //make the visibility false
if (this.gameScore % this.difficultyParameter == 0 && this.noteSpacing > 1.5) //make the game harder but not very hard
{
this.noteSpacing = this.noteSpacing - 0.3; //decrease the distance between the notes
globalGameSpeed = globalGameSpeed + 0.5; //increase game speed
}
break; //avoid checking other keys because already one has been found
}
}
}
}
}
//setup function
function setup() {
createCanvas(800, 600); //creates canvas of desired size
textFont(playfair_font); //use the playfair font to keep everything consistent
}
gameObject = new Game(); //create object of the game class
function draw() {
background(220); //arbitary background for debugging
// if (gameObject.gameState == "menuScreen") //only play the music when the user is in the menu
// {
if (!background_music.isPlaying()) //ensure the music keeps looping forever
{
background_music.play(); //but also ensures that it doesnt start from the start every time
}
// }
// else //if the game is in any other state
// {
// if (background_music.isPlaying()) //check if the music is playing
// {
// background_music.stop(); //if yes then stop the music
// }
// }
gameObject.display(); //displays all the screens
}
function keyTyped() //key typed built in function
{
if (gameObject.gameState == "endScreen") //when the game has ended
{
if (key === 'h' || key === 'H') //you can press h to return to home/restart game
{
gameObject = new Game(); //a new object is assigned to the variables and the constructor initializes everything
}
}
if (gameObject.gameState == "menuScreen") //if the game is in menu screen
{
if (keyCode == ENTER) //use enter to
{
setUpSerial(); //set up serial
}
}
return false;
}
function readSerial(data) //callback function for read serial
{
if (data != null) //if the data is good
{
if (gameObject.gameState == "gameScreen") //if game mode is on
{
if (data >= 1 && data <= 16) //and the data is between the range we want (the keys)
{
gameObject.notePressed(data); //tell the game that a note has been pressed and which one
}
}
if (gameObject.gameState == "recordScreen" && gameObject.recordBool == true) //if the game is in record mode and is recording
{
if (data >= 1 && data <= 16) //if any note is pressed
{
gameObject.recordNotes(data); //record the notes (not used)
}
}
if (gameObject.gameState == "freeScreen") //if the game is in free mode
{
if (data >= 1 && data <= 16) //and the data is what we want
{
gameObject.backgroundVariableChange(data); //we know a note has been played and so change the backgrounds
}
}
}
}
function mouseClicked() //mouseclicked function
{
if (gameObject.gameState == "menuScreen") //if the game is in menu screen
{
//whatever button is pressed, change the gameState to that specific state
if (mouseX >= gameObject.menuButtonX && mouseX <= gameObject.menuButtonX + gameObject.menuButtonWidth && mouseY >= gameObject.menuButton1Y && mouseY <= gameObject.menuButton1Y + gameObject.menuButtonHeight)
{
gameObject.gameState = "freeScreen";
let screenState2 = "97\n"; //tell arduino that we are in free mode now
writeSerial(screenState2);
}
else if (mouseX >= gameObject.menuButtonX && mouseX <= gameObject.menuButtonX + gameObject.menuButtonWidth && mouseY >= gameObject.menuButton2Y && mouseY <= gameObject.menuButton2Y + gameObject.menuButtonHeight)
{
gameObject.gameState = "gameScreen";
gameObject.instructionsBool = true;
}
else if (mouseX >= gameObject.menuButtonX && mouseX <= gameObject.menuButtonX + gameObject.menuButtonWidth && mouseY >= gameObject.menuButton3Y && mouseY <= gameObject.menuButton3Y + gameObject.menuButtonHeight)
{
gameObject.gameState = "recordScreen"; //change the mode to record mode
}
}
if (gameObject.gameState == "recordScreen") //if the game is in record mode
{
//check to see if the button is pressed or not and then change boolean variables accordingly
if (mouseX >= gameObject.recordButtonX && mouseX <= gameObject.recordButtonX + gameObject.recordButtonWidth && mouseY >= gameObject.recordButtonY && mouseY <= gameObject.recordButtonY + gameObject.recordButtonHeight && gameObject.recordBool == false && gameObject.playBool == false)
{
gameObject.recordBool = true;
gameObject.clickCount = 0; //this is so that button is not accidentally double clicked
}
else if (mouseX >= gameObject.recordButtonX && mouseX <= gameObject.recordButtonX + gameObject.recordButtonWidth && mouseY >= gameObject.recordButtonY && mouseY <= gameObject.recordButtonY + gameObject.recordButtonHeight && gameObject.recordBool == true && gameObject.playBool == false)
{
gameObject.clickCount++;
if (gameObject.clickCount >= 3)
{
gameObject.recordBool = false;
gameObject.playBool = true;
gameObject.playingScreen = true;
}
}
if (mouseX >= gameObject.homeButtonX && mouseX <= gameObject.homeButtonX + gameObject.homeButtonWidth && mouseY >= gameObject.homeButtonY && mouseY <= gameObject.homeButtonY + gameObject.homeButtonHeight && gameObject.playingScreen == true)
{
gameObject.playingScreen = false;
gameObject.gameState = "menuScreen"; //go to the menu state if the home button is pressed
}
}
if (gameObject.gameState == "freeScreen") //if the game is in free mode
{
if (mouseX >= gameObject.homeButtonX && mouseX <= gameObject.homeButtonX + gameObject.homeButtonWidth && mouseY >= gameObject.homeButtonY && mouseY <= gameObject.homeButtonY + gameObject.homeButtonHeight)
{
//if the home button is pressed
gameObject.backgroundNumber = 0; //reinitalize variable so that instruction screen pops up again
gameObject.gameState = "menuScreen"; //go back to menu state
let screenState3 = "104\n"; //tell arduino that we are going back to menu screen
writeSerial(screenState3);
}
}
if (gameObject.gameState == "gameScreen") //if the game is in game mode
{
if (gameObject.instructionsBool == true && mouseX >= gameObject.instructionButtonX && mouseX <= gameObject.instructionButtonX + gameObject.homeButtonWidth && mouseY >= gameObject.instructionButtonY && mouseY <= gameObject.instructionButtonY + gameObject.homeButtonHeight)
{
//if the instructions button is pressed
gameObject.instructionsBool = false; //set the boolean to false
let gameScreenState = "99\n"; //tell arduino that we are in game mode now
writeSerial(gameScreenState);
}
if (gameObject.instructionsBool == true && mouseX >= gameObject.homeButtonX && mouseX <= gameObject.homeButtonX + gameObject.homeButtonWidth && mouseY >= gameObject.homeButtonY && mouseY <= gameObject.homeButtonY + gameObject.homeButtonHeight)
{
//if the home button is pressed
gameObject.instructionsBool = true; //ensure the next time instruction page pops up
gameObject.gameState = "menuScreen"; //go back to menu screen mode
let stateOfArduino2 = "103\n"; //tell arduino as well that you have gone back to menu screen mode
writeSerial(stateOfArduino2);
}
}
}
remove();
Description of Arduino Code:
The program reads data from an ultrasonic sensor and responds by generating tones through a buzzer and turning on an LED. The program has four different states, menu mode, game mode, record mode, and free mode, which are determined by input from the P5js interface.
In the beginning, the code defines the pins for the ultrasonic sensor, buzzer, buttons, and LED, and initializes several variables used in the program, including pressed, distance, arduinoState, recordCounter, and recordArray.
The first state is menu mode, indicated by arduinoState == 100. In this mode, the program waits for input from the connected P5js program. If it receives the value 99, it changes the state to game mode, if it receives 98, it changes the state to record mode, and if it receives 97, it changes the state to free mode.
In game mode, the program turns on the LED to indicate that the ultrasonic sensor is working, reads sensor data, and generates tones based on the distance measured. It also checks for input from the P5js program and leaves the state if it receives the value 103.
In record mode, the program reads data from the ultrasonic sensor, turns on the LED, and waits for input from the P5js program. If it receives the value 102, it plays back the recorded notes stored in recordArray, resets the state to menu mode, and sets recordCounter to 0.
In free mode, the program reads data from the ultrasonic sensor, turns on the LED, and checks for input from the P5js program. It leaves the state if it receives the value 104.
Finally, the playRecordedNote() function is called in the loop() function to generate tones based on the notes stored in recordArray. The function maps the values in the array to specific frequencies and durations using the tone() function.
All over the place, the Serial.read() function is used to skip over irrelevant data that may be present in the serial. Also the break feature is used excessively to break out of the serial.available() loop once the required value is read. Here is the code:
//Define the pins used for the ultrasonic sensor, buzzer, buttons, and LED
const int pingPin = 2; //Trigger Pin of Ultrasonic Sensor
const int echoPin = 3; //Echo Pin of Ultrasonic Sensor
const int buzzerPin = 8;
const int LEDbutton = 7;
//Initialize variables used in the program
int pressed = 0;
long distance = 0;
int arduinoState = 100;
int recordCounter = 0;
int recordArray[100];
int stateValue = 0; //helps change the state of Arduino (sent by p5js)
//Include the pitches library for generating tones
#include "pitches.h"
void setup()
{
//Start serial communication at 9600 baud
Serial.begin(9600);
//Set the ultrasonic sensor pins as output and input respectively
pinMode(pingPin, OUTPUT);
pinMode(echoPin, INPUT);
//Set the LED pin as an output
pinMode(LEDbutton, OUTPUT);
//Turn off the LED initially
digitalWrite(LEDbutton, LOW);
}
void loop()
{
if (arduinoState == 100) //menu mode
{
digitalWrite(LEDbutton, LOW); //this means that the sensor is not working
while (Serial.available()) //if serial is available
{
stateValue = Serial.parseInt(); //then parseInt and see what p5js has sent us
if (stateValue == 99) //game mode
{
arduinoState = 99; //set arduino state to 99 as well which is game mode
digitalWrite(LEDbutton, LOW);
break; //we have the state so we can now go to that state
}
else if (stateValue == 98) //record mode
{
arduinoState = 98; //set arduino to record mode as well
digitalWrite(LEDbutton, LOW);
break; //we have the state so we can leave
}
else if (stateValue == 97) //free mode
{
arduinoState = 97; //set arduino state to free mode
digitalWrite(LEDbutton, LOW);
break; //we have the state so we can leave
}
}
}
else if (arduinoState == 99) //game Mode
{
sensorReading(); //start the sensor
digitalWrite(LEDbutton, HIGH); //turn on LED to indicate the sensor is on
while (Serial.available()) //if serial is available
{
int changeOfState = Serial.parseInt(); //use the parseInt functiion
if (changeOfState == 103) //if the value is 103, this means that the p5js program has left the game Mode
{
arduinoState = 100; //so arduino can also leave the game mode state
break;
}
Serial.read(); //keep reading and ignoring the serial values until 103 is reached
}
}
else if (arduinoState == 98)
{
digitalWrite(LEDbutton, HIGH); //turn on LED to indicate the sensor is on
sensorReading(); // start the sensor
int dataNote;
while (Serial.available()) //see if serial is available
{
int playSound = Serial.parseInt(); //parse the serial for integer
if (playSound == 102) //if it is 102, we know that p5js wants us to play the recorded sound
{
for (int i = 0; i < recordCounter; i++) //go through the array that has recorded notes
{
playRecordedNote(recordArray[i]); //play each note
recordArray[i] = 0; //then empty that location for the next time
}
arduinoState = 100; //go back to state 100 which is menu state
recordCounter = 0; //initialize the record counter to 0 as well for the next time
break; //break the loop
}
Serial.read();
}
}
else if (arduinoState == 97) //if you are in free mode
{
sensorReading(); // start the sensor
digitalWrite(LEDbutton, HIGH); //turn on LED to indicate the sensor is on
while (Serial.available()) //check if serial is available
{
int stateChange = Serial.parseInt(); //parse the serial for integers
if (stateChange == 104) //if it is 104
{
arduinoState = 100; //p5js has left free mode and is in menu mode so arduino does the same
break;
}
Serial.read(); //keep reading serial to ignore irrelevant information
}
}
}
void playRecordedNote(int valueOfNote) //play recorded notes
{
if (valueOfNote == 1)
{
tone(buzzerPin, NOTE_C4, 400); //for 400ms
}
else if (valueOfNote == 2)
{
tone(buzzerPin, NOTE_E4, 400);
}
else if (valueOfNote == 3)
{
tone(buzzerPin, NOTE_G4, 400);
}
else if (valueOfNote == 4)
{
tone(buzzerPin, NOTE_A4, 400);
}
else if (valueOfNote == 5)
{
tone(buzzerPin, NOTE_C5, 400);
}
else if (valueOfNote == 6)
{
tone(buzzerPin, NOTE_D5, 400);
}
else if (valueOfNote == 7)
{
tone(buzzerPin, NOTE_E5, 400);
}
else if (valueOfNote == 8)
{
tone(buzzerPin, NOTE_G5, 400);
}
else if (valueOfNote == 9)
{
tone(buzzerPin, NOTE_A5, 400);
}
else if (valueOfNote == 10)
{
tone(buzzerPin, NOTE_B5, 400);
}
else if (valueOfNote == 11)
{
tone(buzzerPin, NOTE_D6, 400);
}
else if (valueOfNote == 12)
{
tone(buzzerPin, NOTE_E6, 400);
}
else if (valueOfNote == 13)
{
tone(buzzerPin, NOTE_G6, 400);
}
else if (valueOfNote == 14)
{
tone(buzzerPin, NOTE_A5, 400);
}
else if (valueOfNote == 15)
{
tone(buzzerPin, NOTE_B5, 400);
}
else if (valueOfNote == 16)
{
tone(buzzerPin, NOTE_C5, 400);
}
else
{
tone(buzzerPin, NOTE_C5, 400);
}
delay(408); // a small delay so it sounds nice
}
//Function to read the ultrasonic sensor and play a tone based on the distance measured
void sensorReading()
{
//Send a short low pulse
digitalWrite(pingPin, LOW);
delay(2); //delay to avoid complications
digitalWrite(pingPin, HIGH); //sends a high pulse for 10 microseconds
delay(10);
digitalWrite(pingPin, LOW);
distance = pulseIn(echoPin, HIGH); //Measure the duration of the ultrasonic pulse and calculate the distance
distanceNotes(distance); //play the notes based on the distance
delay(100);
}
void distanceNotes(long distance)
{
if (distance >= 2920) //if the distance is greater than 2920
{
noTone(8); //then dont play anything
Serial.println(0); //send 0 to serial to indicate nothing is being played
pressed = 0; //reinitialize the key pressed variable to 0 so other keys can be pressed
}
else if (distance >= 2800 && pressed == 0) //for each distance, there is a specific note
{
tone(buzzerPin, NOTE_C5, 400);
if (arduinoState == 98) //f the arduino is in record mode
{
recordArray[recordCounter] = 16; //record the note number in the array
recordCounter++; //increment counter for the future
}
Serial.println(16); //print the serial and send to p5js for it to be dealt with appropriately
pressed = 1; //do this to avoid the same note being played repeatedly
}
else if (distance >= 2600 && pressed == 0)
{
tone(buzzerPin, NOTE_B5, 400);
Serial.println(15);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 15;
recordCounter++;
}
}
else if (distance >= 2420 && pressed == 0)
{
tone(buzzerPin, NOTE_A5, 400);
Serial.println(14);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 14;
recordCounter++;
}
}
else if (distance >= 2260 && pressed == 0)
{
tone(buzzerPin, NOTE_G6, 400);
Serial.println(13);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 13;
recordCounter++;
}
}
else if (distance >= 2060 && pressed == 0)
{
tone(buzzerPin, NOTE_E6, 400);
Serial.println(12);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 12;
recordCounter++;
}
}
else if (distance >= 1960 && pressed == 0)
{
tone(buzzerPin, NOTE_D6, 400);
Serial.println(11);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 11;
recordCounter++;
}
}
else if (distance >= 1690 && pressed == 0)
{
tone(buzzerPin, NOTE_B5, 400);
Serial.println(10);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 10;
recordCounter++;
}
}
else if (distance >= 1515 && pressed == 0)
{
tone(buzzerPin, NOTE_A5, 400);
Serial.println(9);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 9;
recordCounter++;
}
}
else if (distance >= 1345 && pressed == 0)
{
tone(buzzerPin, NOTE_G5, 400);
Serial.println(8);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 8;
recordCounter++;
}
}
else if (distance >= 1180 && pressed == 0)
{
tone(buzzerPin, NOTE_E5, 400);
Serial.println(7);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 7;
recordCounter++;
}
}
else if (distance >= 1005 && pressed == 0)
{
tone(buzzerPin, NOTE_D5, 400);
Serial.println(6);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 6;
recordCounter++;
}
}
else if (distance >= 825 && pressed == 0)
{
tone(buzzerPin, NOTE_C5, 400);
Serial.println(5);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 5;
recordCounter++;
}
}
else if (distance >= 660 && pressed == 0)
{
tone(buzzerPin, NOTE_A4, 400);
Serial.println(4);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 4;
recordCounter++;
}
}
else if (distance >= 530 && pressed == 0)
{
tone(buzzerPin, NOTE_G4, 400);
Serial.println(3);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 3;
recordCounter++;
}
}
else if (distance > 310 && pressed == 0)
{
tone(buzzerPin, NOTE_E4, 400);
Serial.println(2);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 2;
recordCounter++;
}
}
else if (distance <= 310 && pressed == 0)
{
tone(buzzerPin, NOTE_C4, 400);
Serial.println(1);
pressed = 1;
if (arduinoState == 98)
{
recordArray[recordCounter] = 1;
recordCounter++;
}
}
}
Communication Between Arduino and P5js:
P5js and Arduino are fully connected in this implementation. They communicate with each other to ensure that the program runs smoothly and there is harmony between the software and hardware components of the project. The project has various modes that allows for a greater user experience and so the program is divided in a way that breaks down different modes in game states. This goes for both P5js and Arduino. P5js sends data to Arduino that allows for mode changes in the Arduino program. Without P5js communicating the state it is on, Arduino would have never been able to sync with the program. P5js also tells Arduino when to start recording, stop recording, and then when to play the recorded tune. It also sends data to Arduino in order to communicate when to turn the sensor on or off. Arduino, on the other hand, in the game mode calculates the distance using its ultrasonic sensor and then evaluates what note each distance corresponds to and then finally sends it to P5js interface. The interface then uses the data to either play the game and “catch the notes.” Furthermore, in the free mode as well, Arduino filters out the bad data and then sends only the right notes to P5js in order to be displayed using various images.
Reflection and Improvement
I’m particularly proud of the user interface that I have designed for the project. I put a lot of thought and effort into making it user-friendly and intuitive, while also allowing for complete freedom and interactivity. The buttons and other interactive features really make the interface come alive, and I think users will find it enjoyable to use.
In addition to the user interface, I am also really proud of how I tackled the game states in the project. Initially, I was worried that I wouldn’t be able to get everything done in time, but through lots of user testing and feedback, I was able to create a code that I’m actually really proud of. The game states add a level of complexity and depth to the project, and I think users will appreciate the extra challenge.
Finally, I’m really happy with the arraySwitch function that I developed for the project. It was a difficult task to create two different arrays with different types of notes and then switch between them every time one array was exhausted. But in the end, it was definitely worth it because it ensures that the same note won’t come up twice in a row, which makes the game more interesting and challenging.
I am very proud of the end result of my project, but there are still areas that I can improve on. For instance, I expected the ultrasonic sensor to be more accurate, but I was disappointed with the level of precision it provided. In addition, using the ultrasonic sensor made it difficult for users to play notes quickly, as the hand gestures required were awkward and unlike those used for a normal piano. In the future, I will need to explore alternative sensor options to improve user experience.
Another area for improvement would be to incorporate more buttons or LEDs on the Arduino side of the project to make it more interactive. This would enhance the overall experience for users and make it more engaging. Finally, time management was a significant challenge for me during this project, and as a result, I was unable to 3D print or craft a box for the Arduino. As a result, the aesthetics of the final product (hardware) were what I had in mind. In the future, I will need to better manage my time to ensure that I have enough time to complete all aspects of the project, including the aesthetics.
User Testing
According to my friend’s testing of my project, they had some initial confusion with the user interface, as there were no back buttons and some buttons didn’t respond when hovered over. Additionally, they had trouble understanding when to start playing the notes and what each mode was meant for. However, through incorporating their feedback, I was able to make improvements to the instructions screen and provide clearer information on the interface, which ultimately led to a more interactive and controllable experience. My friend found the P5js portion of the program to be smooth and bug-free, which is working well.
On the other hand, my friend had issues with the sensor as it registered a note even while their hand was still moving in front of it. I had to explain to them that only one finger/hand should be used, and it needed to move out of the way of the sensor for the next note to be played. To make this clearer to future users, I could include instructions on the interface or have a printed paper with the instructions. However, the best solution would be to use a different sensor altogether to avoid these issues in the future.
Week 14 – Final project User Testing
Sanjana was my user-tester for the latest version of my final project. As per the instructions, I didn’t give her any instructions before she started testing it, and was able to receive some valuable feedback on the ways I could improve instructions given to the user and discovered one bug that has a potential to be a cool feature haha.
Video:
Feedback:
- Sanjana found it pretty easy to navigate the user interface and how the mapping between the glove and p5.js work. I feel like I needed to explain her that sensors are very sensitive so sometimes even minor bends affect the movement. The parts of the project that are working well are communication between the flex sensors and p5.js and also the graphics.
- Through the user-testing, I realized that I should make it clearer in the instructions page how the user should hold her hand so that sensors would work the best way. I should also mention that the right, left, zoom in and out movements sometimes can be combined when corresponding hand movements are combined (i.e. when the user bends her thumb while bending her hand right)
- This user-testing also helped me realize that that when zooming in or out, there’s some limit till which u can zoom in/out so that the image will not get distorted. By zooming in too much you can get very deep into the image so that only pixels will be visible and the image presented will not make sense. However, by zooming out of the image you can reach a cool 3d space where the image will cover the sphere (the cool feature I was talking about). See below image for reference.
Akyn the device that takes care of everything, so you don’t have to.
- Describe your concept
- Akyn is a Jarvis(Iron Man) like artificial intelligence that helps to make morning and night routines more pleasant. As a student, I can indeed say that many of us are lazy. We keep procrastinating a lot and as a result, get stressed out when the deadline approaches. Akyn is a way to change that. With only 2 buttons, you can escalate your morning and night routines to a new level.
- As part of the morning routine, Akyn will turn on the AC for you and greet you in a respectful manner. Afterward, it will read out(over voice) the emails you have gotten over the night. And as a last action, it will play motivational music so there is no way you still lay in your bed.
- When you go to sleep, Akyn will turn off the lights and AC for you. Afterward, Akyn voices over your alarm time for tomorrow and starts listing the events from your Google Calendar for tomorrow. This way you will start visualizing tomorrow starting from tonight.
- How does the implementation work?
- Description of interaction design
- The interaction design is very simple. There are 2 buttons on the breadboard located right near your bed. Green button for morning routine and Blue button for night routine. By pressing one of them, the respectful routine gets triggered.
- Description of Arduino code
- The Arduino concept is very simple. There are 2 buttons and 2 servo motors connected. In the setup, I connect the motors and start the handshake process with the p5.js. In the main loop, while there is a connection established, I keep waiting for the user to press either of the buttons. Once one of them is pressed the motor(s) start rotating which turns on/off AC/lights. Lastly, 0 or 1 is sent to the p5.js indicating which button is pressed.
-
#include <Servo.h> Servo servo_lights_off; Servo servo_AC; int pin_GM_button = 2; int pin_GN_button = 4; int pin_lights_off = 11; int pin_AC = 9; int state_GM_button = 0; int state_GN_button = 0; int prev_state_GM_button = 0; int prev_state_GN_button = 0; void setup() { servo_lights_off.attach(pin_lights_off); servo_AC.attach(pin_AC); Serial.begin(9600); // start the handshake while (Serial.available() <= 0) { Serial.println("-1"); // send a starting message delay(300); // wait 1/3 second } } void loop() { while (Serial.available()) { state_GM_button = digitalRead(pin_GM_button); state_GN_button = digitalRead(pin_GN_button); // GM button is pressed if (state_GM_button != prev_state_GM_button && state_GM_button == 1) { // turn on the AC servo_AC.write(180); delay(1000); servo_AC.write(0); delay(1000); // send message to P5.js Serial.println("0"); } // GN button is pressed if (state_GN_button != prev_state_GN_button && state_GN_button == 1) { // turn off the lights servo_lights_off.write(0); delay(1000); servo_lights_off.write(180); delay(1000); // turn off the AC, double click is needed to go to Night mode from Morning mode servo_AC.write(180); delay(1000); servo_AC.write(0); delay(2000); servo_AC.write(180); delay(1000); servo_AC.write(0); delay(2000); // send message to P5.js Serial.println("1"); } // update the previous states of the buttons prev_state_GM_button = state_GM_button; prev_state_GN_button = state_GN_button; } }
- Description of p5.js code
- P5.js is where the main magic happens. The main interaction happens in the javascript file while the authorization happens in the index.html page.
- Authorization
- When the user starts the code after a specific amount of time(10 seconds for the user to have some time to read the instructions) authorization script is run. This script sends requests to Google Calendar API and Gmail API over the gapi library. As a result, I am now able to send get requests to the Google API. After the authorization, I load the calendar using a specific query indicating what kind of events I want to see. All of this is saved into the CALENDAR_EVENTS variable
<script> const gapiLoadPromise = new Promise((resolve, reject) => { gapiLoadOkay = resolve; gapiLoadFail = reject; }); const gisLoadPromise = new Promise((resolve, reject) => { gisLoadOkay = resolve; gisLoadFail = reject; }); var tokenClient; (async () => { // First, load and initialize the gapi.client await gapiLoadPromise; await new Promise((resolve, reject) => { // NOTE: the 'auth2' module is no longer loaded. gapi.load('client', {callback: resolve, onerror: reject}); }); await gapi.client.init({ // NOTE: OAuth2 'scope' and 'client_id' parameters have moved to initTokenClient(). }) .then(function() { // Load the Google API's discovery document. gapi.client.load('https://www.googleapis.com/discovery/v1/apis/calendar/v3/rest'); gapi.client.load('https://www.googleapis.com/discovery/v1/apis/gmail/v1/rest'); }); // Now load the GIS client await gisLoadPromise; await new Promise((resolve, reject) => { try { tokenClient = google.accounts.oauth2.initTokenClient({ client_id: google_calendar_USER_ID, scope: 'https://www.googleapis.com/auth/calendar.readonly https://www.googleapis.com/auth/gmail.readonly', prompt: 'consent', callback: '', // defined at request time in await/promise scope. }); resolve(); } catch (err) { reject(err); } }); })(); // get token in case error while authorizing appeared async function getToken(err) { if (err.result.error.code == 401 || (err.result.error.code == 403) && (err.result.error.status == "PERMISSION_DENIED")) { // The access token is missing, invalid, or expired, prompt for user consent to obtain one. await new Promise((resolve, reject) => { try { // Settle this promise in the response callback for requestAccessToken() tokenClient.callback = (resp) => { if (resp.error !== undefined) { reject(resp); } // GIS has automatically updated gapi.client with the newly issued access token. resolve(resp); }; tokenClient.requestAccessToken(); } catch (err) { console.log(err) } }); } else { // Errors unrelated to authorization: server errors, exceeding quota, bad requests, and so on. throw new Error(err); } } function loadCalendar() { const tomorrow = new Date(); tomorrow.setDate(tomorrow.getDate() + 1); tomorrow.setHours(0, 0, 0, 0); const after_tomorrow = new Date(); after_tomorrow.setDate(after_tomorrow.getDate() + 2); after_tomorrow.setHours(0, 0, 0, 0); var calendar_query = { 'calendarId': 'primary', 'timeMin': tomorrow.toISOString(), 'timeMax' : after_tomorrow.toISOString(), 'singleEvents': true, 'orderBy': 'startTime'}; // Try to fetch a list of Calendar events. If a valid access token is needed, // prompt to obtain one and then retry the original request. gapi.client.calendar.events.list(calendar_query) .then(calendarAPIResponse => CALENDAR_EVENTS = JSON.stringify(calendarAPIResponse)) .catch(err => getToken(err)) // for authorization errors obtain an access token .then(retry => gapi.client.calendar.events.list(calendar_query)) .then(calendarAPIResponse => CALENDAR_EVENTS = JSON.stringify(calendarAPIResponse)) .catch(err => console.log(err)); // cancelled by user, timeout, etc. } // call loadCalendar after 3 seconds so that gapi.client initiates connection setTimeout(loadCalendar, 10000); </script>
- When the user starts the code after a specific amount of time(10 seconds for the user to have some time to read the instructions) authorization script is run. This script sends requests to Google Calendar API and Gmail API over the gapi library. As a result, I am now able to send get requests to the Google API. After the authorization, I load the calendar using a specific query indicating what kind of events I want to see. All of this is saved into the CALENDAR_EVENTS variable
- Main logic
- Now once everything is loaded, the client connects the Arduino and p5.js by simply clicking on the screen and selecting the Arduino port. From now on, everything gets triggered once the button is pressed.
- Green button – goodMorning function gets called.
1) Load Gmail information about unread emails
2) Clear up the saved array, leaving only the necessary information to display
3) P5.speech libraries voices over the amount emails received, the subject, and the sender info of each one as well. Upon completion, Mozart’s Eine kleine Nachtmusik classical music starts playing./* voices over the Good Morning mode*/ function goodMorning() { // load the unread emails loadGmail(); // timeout to wait till the above functions finishes execution setTimeout(function() { clearEmails(); }, 2000); setTimeout(function() { myVoice.speak(GM_PHRASES[Math.floor(Math.random()*GM_PHRASES.length)] + "You've got " + emails.length + " emails while you were sleeping."); if (emails.length === 0) { return; } emails.forEach((email, index) => { setTimeout(function() { console.log(numberMapping[index + 1] + ' email is from ' + email.from + ' about ' + email.subject); // -> debug myVoice.speak(numberMapping[index + 1] + ' email is from ' + email.from + ' about ' + email.subject); }, (index + 1) * 10000); // wait for the previous voice to finish before starting next one }); setTimeout(function() { myVoice.speak("Turning on music!"); }, (emails.length + 1) * 10000); setTimeout(function() { playMusic(); }, (emails.length + 1) * 10000 + 3000); // wait till events finishes voicing and the previous line too. }, 6000); // this timer is the sum of the before called timers (1000 + 3000) + 1000 extra miliseconds } - Blue button – goodNight function gets called.
1) Clear up the saved calendar events, leaving only necessary information to display
2) Voice over the number of events and the time the alarm is set to.
3) Voice over each of the events, time, and location/* voices over the events happening tomorrow*/ function goodNight() { events = clearEvents(); setTimeout(function() { const eventsLength = events.length - 1; let p = "Tomorrow you have " + eventsLength + " events. Your alarm is set to " + events[0]['startTime']+ " AM. " + GN_PHRASES[Math.floor(Math.random()*GN_PHRASES.length)]; myVoice.speak(p); // console.log(p); }, 1500); // iterate thoruhg events and voice them over // timeout used since forEach does not wait for the voice to finish which gives undesired output for (let index = 1; index < events.length; index++) { const event = events[index]; setTimeout(function() { console.log(numberMapping[index] + ' event is ' + event['eventName'] + ', happening from ' + event['startTime'] + ' till ' + event['endTime'] + ' at ' + event['location']); myVoice.speak(numberMapping[index] + ' event is ' + event['eventName'] + ', happening from ' + event['startTime'] + ' till ' + event['endTime'] + ' at ' + event['location']); }, index * 13000); } }
- Description of communication between Arduino and p5.js
- Due to the nature of the project, communication is one-sided. P5.js keeps waiting for Arduino to send 0 or 1 after which the main logic happens. Arduino does not need any input from p5.js all it does is wait for the buttons to be pressed.
- Description of interaction design
- What are some aspects of the project that you’re particularly proud of?
- I am very proud of embedding Google API’s into my project. This is something I have not done before nor was it reached in any of the classes. Yet this is something very useful and applicable in a lot of spheres. By doing this, I was able to make the project actually helpful and not just some dummy toy. I believe that the use of this API together with the speech library and servo motors makes the project very unique.
Also, coming up with balanced values for timeout is something to be proud of. All API requests need to be timeout so that the next functions can use the result of that get request. I experimented a lot with these values so that users do not wait too long and the response has enough time to arrive.
- I am very proud of embedding Google API’s into my project. This is something I have not done before nor was it reached in any of the classes. Yet this is something very useful and applicable in a lot of spheres. By doing this, I was able to make the project actually helpful and not just some dummy toy. I believe that the use of this API together with the speech library and servo motors makes the project very unique.
- What are some areas for future improvement?
- In the future, I would be happy to change the timeouts to Future Promises, so that the further functions only get triggered once the previous ones finish. This way it’s not hardcoded and works with any amount of response.
- Working with wires was fun, however, it would be much more flexible and convenient to use either a Bluetooth connection or even a phone. With a little bit of research, I can substitute long and limited wires to a Bluetooth connection between the breadboard and Arduino. With more research, I will be able to write a telegram bot that sends commands directly to Arduino without any buttons.
- Have people try your project without giving them any prompts/instructions and see how they use it
- Are they able to figure it out? Where do they get confused and why? Do they understand the mapping between the controls and what happens in the experience?
- I invited a few friends to try out my project. They really enjoyed the instructions screen and it was very straightforward to authorize Google Account. However, a useful comment I received was to specify the name of the port to which users have to press in order to establish a connection to the Arduino.
- Further interaction was also very clear to my friends. They went near my bed and pressed either a blue or green button.
- What parts of the experience are working well? What areas could be improved?
- They enjoyed a lot the voiceover of the calendar and Gmail. They also liked my choice of music. One thing they stated can be improved is the choice of voices. Since p5 speech libraries hold more than 100 voices, people would like to pick one that they want to hear the most. I believe this is something that can be implemented, nevertheless, I did not want to complicate the project with voice pick since it does not align with the main goal of improvement of morning and night routines.
- What parts of your project did you feel the need to explain? How could you make these areas clearer to someone experiencing your project for the first time?
- I had to explain to them that once the button is pressed they have to wait till the whole routine is finished before pressing a button again. If someone rushes with a button press, the voice will queue up and some of them will be dropped which is highly undesirable. A possible solution for this is to lock the button action until the whole routine has finished(send value from p5.js to Arduino when everything is finished and keep waiting for a response on Arduino). This can be a great addition if I decide to improve the project in the future.
- Are they able to figure it out? Where do they get confused and why? Do they understand the mapping between the controls and what happens in the experience?
Final Project: Space Explorer
Concept
Many of us are fascinated with space and all the heavenly objects floating around far away from us. Our Earth is one of many objects in space. I wanted to make an interactive environment where we could look at these planets and admire their beauty, while also learning a few facts about them. Hence, I used WEBGL in p5Js to create a 3D space where I could spawn planets and stars which one can explore.
Implementation
My implementation of the project is divided into p5Js and Arduino, and has very simple controls, aesthetic visuals, with immersive audio.
Interactive Design
The design of the project is simple. The visual component of the project features a dark space with the nine planets in the solar system (including Pluto) and the Sun. I tried to make it to scale as much as possible but still remaining reasonable with user experience.
With regards to interactivity, I created a controller that lets the user move around in 3D space and look around. Here are the features of the controller:
- Movement: (Joystick) the user can move around in a plane in a 3D space.
- Look Around: (Joystick) the user can pan the camera to look at different angles in the space.
- Zoom: (momentary Switch) the user can zoom in and out. This provides the third dimensional movement in the z-axis. After a certain point of zooming in, the controls and the visuals are mirrored.
- Pointer to Earth: (Switch) this helps the user in navigation and directional feedback. This also increases the spatial awareness; the idea of where the player is relative to Earth.
- Information: (Switch) this will provide the user with some facts about the planet they are closest to.
Arduino
The Arduino provides controller access for the movement and changes in the p5Js. It takes inputs from the joystick and switches, processes it, and sends it to the p5Js as a suitable string of information.
// Movement using joystick
const int x_pos = A0;
const int y_pos = A1;
// zoom in and out
const int z_in = 12;
const int z_out = 11;
// Camera movement
const int camera_x = A5;
// pointer to earth button
const int pointer = 2;
// information about the planet
const int info = 7;
// LED
const int led = 5;
const int n = 7; // number of button inputs the user provides
int send_list[n];
int input_pins[n];
void setup() {
Serial.begin(9600);
pinMode(x_pos, INPUT);
pinMode(y_pos, INPUT);
pinMode(camera_x, INPUT);
pinMode(z_in, INPUT);
pinMode(z_out, INPUT);
pinMode(pointer, INPUT);
pinMode(info, INPUT);
pinMode(led, OUTPUT);
input_pins[0] = x_pos;
input_pins[1] = y_pos;
input_pins[2] = camera_x;
input_pins[3] = z_in;
input_pins[4] = z_out;
input_pins[5] = pointer;
input_pins[6] = info;
while (Serial.available() <= 0) {
Serial.println(0);
delay(300);
}
}
void get_user_input() {
send_list[0] = analogRead(x_pos);
send_list[1] = analogRead(y_pos);
send_list[2] = analogRead(camera_x);
for (int i = 3; i < n; i++) {
send_list[i] = 0;
if (digitalRead(input_pins[i]) == HIGH) {
send_list[i] = 1;
}
}
}
void send_input() {
String to_send = "";
for (int i = 0; i < n-1; i++) {
to_send += String(send_list[i]) + ",";
}
to_send += String(send_list[n-1]) + "\0";
Serial.println(to_send);
}
void loop() {
while (Serial.available() > 0) {
// get user input and parse to send it to p5js.
get_user_input();
send_input();
// receive response from p5js
int response = Serial.read();
analogWrite(led, response);
}
}
It also receives a confirmation from p5Js that the devices is connected. This can be visually seen from the LED which lights up when the connection is made.
The code first gets all the input and creates an array of them.
p5Js
My project is p5Js heavy. The code lets the user connect to Arduino and receive control information to make changes in the canvas. It creates multiple planets which are basically spheres with different sizes, positions, and textures onto them. Each planet then rotates on its axis and are tilted to some degree. The planets are made stationary relative to the Sun, to make navigation and finding the planets easier.
I used the WEBGL library, to create the 3D space and made the camera movable and responsive of the players movement. This created the effect/ feature of moving in the 3D environment.
The code features multiple classes: Player, Camera, Sun, Planet, Info, Star which creates most of the functionality and visuals in the project. I separated out the Arduino interaction into a separate file, arduino.js to make debugging easier.
Communication
The communication between Arduino and P5Js is intuitive. When the Arduino senses change in the controller, it extract the values of the state the controller is in. These values are then stored into an the send_list array. All the values in this array is then converted into a string separated by commas. This string is then sent to P5Js for further processing.
void get_user_input() {
send_list[0] = analogRead(x_pos);
send_list[1] = analogRead(y_pos);
send_list[2] = analogRead(camera_x);
for (int i = 3; i < n; i++) {
send_list[i] = 0;
if (digitalRead(input_pins[i]) == HIGH) {
send_list[i] = 1;
}
}
}
void send_input() {
String to_send = "";
for (int i = 0; i < n-1; i++) {
to_send += String(send_list[i]) + ",";
}
to_send += String(send_list[n-1]) + "\0";
Serial.println(to_send);
}
The P5Js then receives this string, and parses it to retrieve all the individual values of all the input devices in the controller. These are then stored in separate variable and used to control the objects on the canvas .
function readSerial(data) {
if (data != null) {
// get the information from the arduino
let nums = data.split(",");
let len = nums.length;
// parsing the received string to get the useful numbers
let x = 0;
for (let i = 0; i < len; i++) {
user_input[x] = parseInt(nums[i], 10);
x++;
}
// continuing the conversation between p5js and arduino
writeSerial("10");
}
}
After the information is received, P5Js then sends in a confirmation to Arduino to send in more information. This confirmation also determines the brightness of the LED which lights up when P5Js reciprocates back to the Arduino.
Code I am proud of
The hardest part of the entire project was to figure out the movement in 3D space. I had to first learn the orientation of the camera, then move objects using some arrow keys, and make the movement consistent with where the camera was looking at.
This was very confusing. The problem was, when you move without incorporating the camera into the movement, you would always move the same way, no matter where the camera is pointing towards. For instance, you press UP arrow to move forward. Then you change the camera angle to look behind. When you press UP arrow again, instead of you moving forward, you would move backward.
This problem was overcome by having an angle attribute to the player which would map to the camera as such:
class Camera {
constructor() {
// distance between the camera and the virtual player (invisible)
this.distance = 400;
}
// creates the camera and determines the position, where the camera is looking at and at what orientation.
make() {
camera(this.distance * cos(player.angle) + player.pos.x, this.distance * sin(player.angle) + player.pos.y, this.distance/2, player.pos.x, player.pos.y, player.pos.z, 0, 0, -1);
}
}
Notice in the above code, the angle is an attribute to the player. It controls where the camera is looking at and where the player moves, as seen in the code below.
// we are in the move() function in the Player class this.pos.x += move_mag * cos(theta + this.angle - PI/2); this.pos.y += move_mag * sin(theta + this.angle - PI/2);
Credit: I got the idea of the solution through videos of JS64 Youtube Channel.
Future Improvement
There are a lot of things that I could do with this project. I realised that this project is sort of incomplete, although it serves it purpose of being educational, interactive, and aesthetically pleasing. However, to make it more engaging, I could turn it into a game where, I introduce a character who has to search for more inhabitable planets in the vast expanse of space. I could also link it with my Midterm game, as the theme of space travel is similar.
This project only incorporates the solar system, albeit not entirely since it lacks asteroids, moons and comets. I could expand it more to have more planets and better the spatial awareness so that the player is not lost.
Week 14: User Testing
For my latest version of the final project, Aditya was my user tester. He had seen my project before, but had never used to the controls from the Arduino. My Arduino provides navigation controls for the space exploration in p5Js. I did not say what each controls did, and had him figure out on his own.
Feedback
I received some very useful feedback from Aditya. He did take some time to figure out the controls, but he commented that the movement was “smooth”.
He liked the visuals and the overall project. However, he also pointed out some improvements I could make.
- He suggested I add some kind of music in the background so that the user is engaged and immersed.
- The movement was pretty slow and would eventually loose user’s interest.
By observing Aditya, I learnt that his previous experience with controllers, much like mine (similar functionality to X-Box controller), was making it confusing for him to grasp the movement on screen. There is not much I could do to fix this issue, except let him get the hang of it.
Final Project
Concept
I started this project with the purpose of encapturing the essence of this semester, a type of memorabilia. In order to figure out how to do that, I have to go through many types of interaction and setting to land on a viable idea that would require a more simplistic interaction with a scope for storytelling. Finally I decided to use a circular structure which a person can complete 1 by 1 each segment. The purpose would be to finish the whole circle (journey). As I have started to conceptualized the activity to be like a journey, I felt that the literary framework of “Hero’s Journey” would be a great fit. In order to reflect the “Hero’s Journey,” I divided the framework onto 5 segments accordingly: acquiring knowledge/advice, enjoying lively goods, getting distracted from work, working hard to reach goals, and reflection/atonement. Hence, each segment carries these themes respectively from 1 to 5. I included activities and artists that make frequent visits to my day to day life, as by doing this I was able to commentary on our daily lifestyles of growth and change, our hero’s journeys. Something interesting I added is the option of staying on a tile and experiencing it or just skipping by stepping on the next tile. In some cases, skipping a tile is show of no effort, and in others it is a show of determination. Finally, the final page upon completing the tile 5 explains the metaphor on hero’s journey as each of ours journey, and reminds that we are all going through our own journeys next to each other or even together.
Implementation
There is a circuit implementation as well as the software implementation.
For the circuitry, I have had to do some soldering. For the setup of the circle, I have designed a circle with an inner radius of 20 cm and outer radius of 45 cm. The circle is divided into 5 segments which are laser cut to acrylic plates. Under each segment there is an FSR (force sensing resistor), which all are parallel to each other and are in series with 10k Ohm resistor. The values are read analog.
In terms of software, I have had to commit day and nights as there were to many areas to cover which I quickly realized was too much for my plate. For each tile I have utilized different approach. The first tile encompasses a simple representation of knowledge by an image of a quote on an old paper in a library. The second tile has a song playing on a Spotify interface with respective lyrics that touches upon life, youth and joy. The third tile uses the screen recording that I have taken of my own reels feed in instagram. I wanted to incorporate ‘scrolling’ as it is one of the biggest escape/distarction methods I use in my daily life, and as I have recently lost my phone I have been especially realizing how refreshing it is to not have these distractions. In the third tile there is a video that pops up that mentions you have to jump to reach stars which is shadowing. The user has to make the choice of continuing to the next tile themselves, as this is a conscious decision I made. If there is no decision to leave, you are stuck in the loop of social media forever. In the fourth tile, there is a starry night, with a girl sitting on a moon. When the use jumps on the tile the star gets closer and closer to the girl. When the girl reaches the star or user just moves onto the next tile without trying to reach the star, the 5th tile starts. In the 5th tile, there is cycles by Mac Miller playing, with a relaxing image of a midnight beach. This room is designed to be contemplative where the user should just reflect and absorb. When the sequence finishes the final page is prompted. The final page clarifies the concept of the “hero’s journey” and talks about the overarching theme of each tile, and makes a note on how we are all on our own journeys in the company of each other.
The start page. The icon lights up when the mouse is hovered above.
In the implementation side, I had to experiment with different fonts, images, video representation, concept markers and general design. Managing sound and video was especially challenging.
The user only has to take steps as they seem to fit. Similarly they should jump on the 4th tile if that’s something they wish for. They will be having headphones on as well as a tablet in their hands which they can receive all the inputs.
Hero’s Journey – Final Sketch
The link for my sketch.
Arduino Code
My Arduino code is very simple which 5 analog readings are received and passed together at each loop to the serial connection p5.js as follows:
void setup() {
Serial.begin(9600);
}
void loop() {
//receives information from the analog pins accordingly and assigns them
int tile1 = analogRead(A0);
int tile2 = analogRead(A1);
int tile3 = analogRead(A2);
int tile4 = analogRead(A3);
int tile5 = analogRead(A4);
//passes the readings seperated by comma from the pins directly to p5js
Serial.print(tile1);
Serial.print(',');
Serial.print(tile2);
Serial.print(',');
Serial.print(tile3);
Serial.print(',');
Serial.print(tile4);
Serial.print(',');
Serial.println(tile5);
//marks the end of 1 run of data by continuing to a new line
}
p5.js Code
In p5.js code is quite long for this code and therefore please follow the following link to reach the editor that hosts my p5.js code.
I have described the general progress of the experience in terms of p5js in the previous section. One thing interesting about my code is that since creating canvas and creating a video element at the same time is not allowed, instead I displayed the videos by passing them as images and displaying them quite fast. This allowed me to switch videos and come back to any, play around with displays, and music. There is also more freedom in terms of the placement of the video displayed. Basically, the video becomes that is comparatively way easier to manipulate in p5js. However, there needs to be considerations for the frame rate of the sketch. The following is tile 3 code, one of the parts in which I implemented this method:
let organizerCount = 0;
let changeVideo = 0;
let duration = 0;
//the main setter for the videos has to be called once
function organizer() {
short.size(350, 1.78 * 350);
short.loop();
short.hide();
reel.size(350, 1.78 * 350);
reel.loop();
reel.hide();
}
//function checks whether there is pressure on the next tile
//to prompt up the next tile event
function tile3Function() {
if(dataTile4 > 100){
//the videos are stopped and the selector value is changed
reel.stop();
short.stop();
selector = 4;
}
if (!organizerCount) {
//the organizer function is called only once
organizer();
//escape from if condition
organizerCount++;
}
//setup the main reel video with the 'jump for stars' video
//popping up randomly
else {
background(255);
frameRate(60);
//put out video image by image in order to have a canvas to
//manipulate
let img = short.get();
let mg = reel.get();
//formatting
strokeWeight(0);
stroke(0);
fill(0);
//the bacground of the phone is black
rect((width - img.width) / 2, 20, 350, 1.95 * 350);
//randomly prompt up the 'jump for stars' part
if (changeVideo == 2) {
//stop the main reel video
short.pause();
image(mg, (width - mg.width) / 2, (height - mg.height) / 2);
changeVideo = 2;
duration++;
//run the jump for the stars video for a set time
if (duration == 60) {
changeVideo = 0;
//switch back to the main reel
short.play();
duration = 0;
}
}
//the random value generaor prompt the 'jump for stars' video
//and runner for the main reel video
else {
changeVideo = int(random(1, 900));
image(img, (width - img.width) / 2, (height - img.height) / 2);
}
//phone image on the videos that are played
phone.resize(img.width * 2.55, img.height * 1.35);
image(phone, width/10.2, -80);
}
}
Communication between Arduino and p5.js
I have used the exemplary communication that was given to us in the class. Hence, there is the following snippet in my p5js sketch to receive serial data from Arduino, as well as the needed library and html edit for this communication. Reading for the p5js from Arduino checksfor 5 data points and stores them into 5 variables to be used in other parts of the code.
function readSerial(data) {
if (data != null) {
// when there is an actual message, receive and split the message
let fromArduino = split(trim(data), ",");
// if the right length, then proceed and assign one by one to variables
if (fromArduino.length == 5) {
dataTile1 = fromArduino[0];
dataTile2 = fromArduino[1];
dataTile3 = fromArduino[2];
dataTile4 = fromArduino[3];
dataTile5 = fromArduino[4];
print(
dataTile1 +
" " +
dataTile2 +
" " +
dataTile3 +
" " +
dataTile4 +
" " +
dataTile5 +
"\n"
);
}
}
There is also another function that prompts up when space bar is pressed to establish the serial communication between Arduino and the p5js editor.
function keyPressed() {
if (keyCode == 32) {
// important to have in order to start the serial connection!!
setUpSerial();
}
}
Challenges
The whole project was quite stressful for me as I felt early on that there were some loopholes in the logic and storytelling aspect of it. Despite figuring out the physical interaction of it to be simple and clean, I realized that even after giving days to plan out the storytelling did not change how complicated it turned out to be in the end. I was trying to sketch out each tile and transitions between tiles as well as the general storytelling framework I was going to use for so long, that I experienced a little withdrawal in my capability figure this out. So I just simply sat down and drafted what I had on my mind till now.
In terms of Arduino, the most challenging part was to figure out how to evaluate data to detect jumping, as 1 jump would give consecutive no pressure values to p5js, which disrupted the work on my tile 4. Hence, I decided to counter this by setting the frame rate a set value, so that irrespective of the speed that Ardino delivers to the computer the receiver of the data would be ordered and easy to analyze and deal with. After doing that, I was able to see trends in how the force reading changes during a jump and make the necessary adjustments to register jumps one by one.
The p5js section is definitely more complicated part of my code, and without doubt at every tile (out of 5) there was a long design process as I tried out styles. I incorporated videos and sounds heavily as my experience with this semester was completely defined by videos and sounds as well. In doing that, for example, in the distraction (scrolling in Social Media) video I had to find a way to have the video on but also a canvas so that I can have a phone silhouette on as well as switch between 2 video entities randomly. Figuring out this part was very challenging and had to do a lot of research in terms of how to run videos in p5js. I had a similar struggle in controlling sound files as well, which required a similar amount of commitment in its logic.
While doing the p5js, the rest of the methods I used were not completely new but still challenging. An issue I have had problems till now is the completeness and digestibility of the total experience. Despite putting more time into the design and storytelling sketch this time, I felt that there was still aspects that just did not feel complete. For example, I spent such a long time on tile 2 (which a song is listened in the topic of jot of living), I changed the construction of the page at least 10 times, because I was not able to figure out what kind of delivery worked best out of each option in the grand scheme of the project. Looking at the user reactions, I think the project needs a pamphlet prior to being used, that will allow the users to get into the mindset of the project.
Successes
The experience finally running from the start to finish was what I felt the most proud about, as I have been working on it piece by piece for so long. I’m especially the most proud about the simple experiences like in the last tile just watching the night sea roll to sound of Circles playing. Similarly, reaching the stars tile is so simple yet feels so important and beautiful to me. I was happy to be able to put a concept that I felt to be true and constantly felt around me into a project. I always felt that we were all so consumed with our struggles, successes, distractions, reflections all the time, which always circled back to start. Slump would appear every other week. We would overwork ourselves for a couple weeks every now and then. We would become especially introspective every now and then. I just wanted to concretely draw this process out, personalized with the tokens that I believed marked this journey. My main goal all along the project was to have the scene after tile 5, where many circles appear, intersecting each other. This is what I felt thankful for the most throughout the whole semester. We all had to go through our journeys but it felt even more worthy because we got to have each other’s company: intersecting circles/journeys with our intersecting lives.
Future Improvement
For future improvement, I would want to make this project even more immersive. I feel that every moment of it works to establish a feeling or understanding in the user, which requires the full attention of the user. Hence, I was thinking a fully immersive experience that would limit the user’s ability to look away would definitely align with my project. This would also help to emphasize that our power is on how we respond to what comes our way. We get to look away (step away), but we don’t get to just not see. I think that in terms of this project, this was the overarching dynamic that was created and there are many ways this could be implemented by changing the experience’s location from computer screen, to projected screen on 4 walls inside a small chamber created with projection fabric.
User Testing
User testing was done but I cannot paste the specific video/photos due to privacy reasons. However admittedly there was haste in terms of rushing through the steps similar to my previous project in midterm. It seems that I will have to think more about how to handle this issue by changing the interface or giving a booklet of directions to read before users try out the the journey.
Additional User Testing:
User Input Highlights:
The user did not mention hardships in terms of the instructions of the experience.
There were interesting insights and take about the individual perception of the journey.
The linked circle outro was especially a favorite, as according to the user it emphasized how people come by to our lives and even if they have to leave, there is a possibility they will be back as there are two points of intersection between each circles.
The user made interesting comments about the distraction tile, and how it was constant battle between looking at the reels and looking at the “reach the star” reel.
The user mentioned that it did not feel as if it was a circle but was more of a path as they were not the same person at the end of the journey. After talking about my inspirations of hero’s journey concept and how I made parallels between the literary device and the experience, they found it very intriguing. They think that the other users should be redirected to receive more information on hero’s journey upon going through the project as well.
The user felt that instructions were generally clear. They did not know what to do after being trapped in the reflection room, but as this was my intention as a part of the project I though it was exactly how it should be when I saw the user just switching between tiles to try to get out of reflection room. The user thought it was a good example of the things we could not run away from/just leave.