Concept:

This consisted of creating a final project for the class that would combine both Arduino and Processing to create an Interactive game or application. 

Idea:

My initial idea consisted of creating an application that would have two principal phases. The first phase would be an interactive free play piano where the user could explore and play the piano as they wish. This first phase would include features such as recording something and playing the record. The second phase would consist of a game where a piano would be shown and notes would be falling. The user would have to press (or play) the piano according to the falling notes. However, unfortunately, the first phase of the game took me a lot of time and effort that I was unable to work on the second phase. In the end, my project consisted of a music instrument application with additional features.

Process:

Initially, I worked on creating a basic platform for the piano. First I decided to solely focus on the processing part and therefore I started by creating a piano where keys from the keyboard represented the tiles from the piano. After, I focused on the design of the piano (basic design) implementing the shape functions in Processing.

Then one of the main components of my project was the sound files uploaded. I found files online and downloaded them. However, while doing my project, I realized the files needed to be edited to improve the sound and functionality of my instrument. Then I focused on learning about the minim library within Processing which helped me with making the sound files work.

After finishing with the files, I started working on the additional features. These features took me more time than I expected. The features I included were recording a note, stop recording, deleting notes from the record, adding notes to different parts of the record, and playing the record. The recording part was complicated because I wanted to implement a way to allow for various notes to be recorded in one part and played at the same time. I was able to do this by using a 2-D array where one dimension would represent the part being played and the other would represent the note being played. After implementing these various features, I focused on possible areas of improvement. One example was a self-updating array where an array would be resized onc

e the number of notes passed the capacity of the array.

Then, I started focusing on connecting the Arduino and implementing the Arduino board. I used simple features from the Arduino but they certainly did add great ways to make the application interactive. The parts I used were the photocells, buttons, and the distance measuring device.  I was just able to get a total of 4 photocells. These represented the tiles in the piano. Because I wanted to create a full piano and using only 4 notes did not help, I decided to use the distance measuring device. According to the distance measured, the piano would be changing location and therefore this would affect the notes being played by the tiles. Finally, the buttons were used to access the recording and playing part. One of the buttons recorded and stopped recording while the other played the recorded file. I could not add another button to represent the deleting part but this part can be performed with a keyboard key.

Finally, I focused on the design of the piano because I did not want to keep the application with basic rectangles and circles. I decided to draw and create all the designs including the tiles and backgrounds. This was done with several applications including Procreate, Affinity Designer, and Adobe Illustrator. 

Challenges

One of the main challenges encountered was in the sound files part. The files were really large in size and included parts that were not entirely necessary. Therefore I had to crop and edit some of the files because some of them were around a minute long. For notes on a piano, I did not need an entire minute. Since I wanted to shorten the file size and duration, I had to edit a total of 88 sound files in GarageBand. While doing the project I also realized that the beginning parts also needed to be edited. This was because, for the sound files, the actual sound did not start right away at the beginning of the file. This made several of my notes sound as if the application was lagging since the sound would be performed seconds after the tile was played.

Another challenge I encountered was with the part for the tiles. Since the piano consists of black and white tiles positioned differently across the piano. I had to create a way to recognize if a tile was black or white and according to this change the position of the tiles. This was challenging because although the piano tiles follow a pattern, the pattern is really complicated. For example, a pattern consists of 7 white tiles with 5 black tiles with black tiles separated as 2 and 3 tiles. 

Finally, another main challenge I encountered was in the part where I had to connect the Arduino and Processing. I was not completely confident in using both together and therefore it took me some time to figure out some of the parts to connect this. 

Areas of Improvement

There were several parts that I wanted to improve in my project. First, I want to possibly add new features to the free play piano. Some of the features in the piano are not functioning perfectly as I wanted although they are functioning. For example, for the selection of a specific part to add a note, this has to be selected AND deselected to be able to select another part. Also, I wanted to create a part where I could print or pop up a message for certain errors. For example, each “part” in the piano cannot contain more than 6 notes that are played at the same time. I wanted to print a message to represent this part when a user would try to record more than 6 notes in the same “part”. Another area of improvement was implementing the deleting button in the Arduino as mentioned in the Process part. 

How the App works

• The first part includes the initial part where the user must press the start button to begin the application
• The 4 photocells represent 4 tiles in the piano. The keys to be played are represented by colored lines around the tile. 
• The red button is used to Start Recording. Once it is pushed, it must be pushed again to stop recording.
• The Green button is used to play the record.
• The distance measuring device represents the position of the keys represented by the photocells. When this device is altered, the position will be changed on the screen accordingly.
• A user can select a specific part within a record to add a note in the part. To select another part, the user must deselect the previous part
• A user can delete several notes at the same time by selecting the notes and pressing the ‘l’ key on the keyboard.
• A user can also travel through the recording by using the ‘q’ and ‘w’ keys. The ‘q’ key is used to move the note part to the left while the ‘w’ key is used to move the key part to the right. 

Conclusion

I was very proud to see my project completed because I dedicated a lot of time and effort to this. The code of the application resulted to be much longer and challenging than I expected. I really liked this project because I think it really represents my work. Starting from the ideas, the sound files, and the design, all components were created or edited by me. Although I could not implement my second idea, I really liked how the project turned out to be. I think it also includes an interactive part through the use of photocells and the distance measuring device. Overall, I really enjoyed working on this project and although it took me much more time and effort than I expected I think I was able to learn new skills and improve certain techniques within coding in general and applications such as Arduino and Processing.

Code Files:

IM_Final_Arduino

For my final project idea i was unsure for a quite long time. Initially, it was an “art therapy app” as you can read in the previous post, but implementing some approaches in this app, i realised that in this particular program you are not creating new art pieces, but interact with existing ones. So I switched to this name.

Description

• Welcome screen

App has first welcome screen with the button to start. Circle design is based on functions with some random statements, that is why each time program executes user gets different, a little bit unexpected design

• Modes screen

In this part user gets a question “What do you feel?” and 3 options of the answer. There are different …. Inside each mode. Design is kind of similar to the previous tab to support the app style, but not copy the first idea. Based on the user’s choice screen moves to the next slide

• Anxiety mode

For this mode I picked up the sea as an image after tones of research of methods of art therapy/ help with mental problems and ….. . A lot of peoplefeel much better after some time spending near the churning sea, because sea is infinitely big and strong. Image of the sea gives people feeling of the happiness, satisfaction and peaceful state  of mind. That it why I implemented some kind of meditation process here. You can imagine that you are the sea. And all your emotions proportional to the speed of the waves. You can use your hand to change the intensity of the power of the sea.

In terms of code i used rectangles and noise function (inspired by Ham

• Anger mode

I was thinking about the visuals of this mode for a while. Expression of state of anger is different for each person, thus the image of the anger is different. I asked several people about this and got bunch of answers:

a) kettle emits a lot of steam

b)hammer breaks glass

c)active volcano

I stopped on the last idea of the volcano. 

•  Happiness mode

I used similar example as we did in class with camera recognition and self-text representation with the text

Arduino part:

I used distance sensor and motors to make communication between processing and arduino. For distance i mapped the value and sent it to processing (to manage the sea). For motors i have a specific variable in processing, which depends on pressing the space(anger mode).  This variable increases each time user presses space and go to arduino to make motor work

You can find code here

💫Reminder

My final project is about creating art works and relieving stress simultaneously!

📌Links to go through my previous posts:

Week 13: Final Project Progress

Final Project Progress [week 14]

🔧Setup

I wanted to beat something soft, so I decided on sponges. The only ones in the immediate vicinity were kitchen yellow sponges, and I used them. In order to attach them to the table, I needed to thread a thread through them and attach them with tape. To make the distance sensor at the same height with them, I had to spend another sponge and a lot of tape.

✅Changes

• I moved the buttons in the Processing on the Arduino so that I wouldn’t move my hands from the top of the table while I was working. I also added an LED as an identifier of when the session started and when it ended.
• I added a song to the background, as well as a final screen that suggests starting the session again.
• Generative art spots don’t appear until the user selects a color, so I was able to make them transparent.

⚒ Code

Full code and all additional data are on my gitHub.

• In my handshake from Arduino side I sent 5 values and received 1

  Serial.println("0,0,0,0,0");
  

• I added screenshot feature

void keyReleased() {
  if (keyCode == TAB) saveFrame(timestamp()+"_##.png");
}

• There is a separate class for spots
• Generative art letters code is partially taken from this website (thank you, Professor!)

💡Testing

Now my project is much more user-friendly, especially after previous week testing. During testing we were choosing the song that in our opinion suits this purpose and decided to use Linkin Park – Numb.

🐋Result

Description

SpaceTrail is a game in which the user plays as a Space Artist by piloting a spacecraft along the screen and paint in a myriad of colors (different shades of blue, yellow, green red and a combination of all). The color of that the player is using is displayed on the Arduino spaceship through the LED lights, each lighting up in accordance the user input (except for the multicolor, which lights up all the lights).

But there’s a trick to this game though: the player have their eyes on the spacecraft and the spacecraft only since if you were actually piloting a ship, you wouldn’t be able to see the trail you leave behind.

The spacecraft and menu are maneuvered through a PSP Controller which gets the user input and passes it to Processing and Arduino in order to move the spacecraft on the physical world and at the same time create the artwork on the screen.

Once the player feels like they are done with their work, they can save it by pressing the SAVE button on the controller, which will locate the sketch on a folder within the program called “drawings”.

Process

Idea:

We had originally planned to make a car track simulator, however, we wanted to make something more creative and, whilst thinking of how we could incorporate the progress we had made previously (been able to get inputs from a controller using two special libraries in Processing and a Windows computer) to make something a little more spicy.

Armaan came up with the idea of using the car we created as a means to create art. Meanwhile, I wanted to leave the spatial intelligence test element in the project and alas, we came up with SpaceTrail.

We are extremely proud of our work. However, we did have to make a deal with the devil and sacrifice various campus cats (and our own sanity) to have this project ready.

Arduino and Serial Communication

Besides time consuming (to say the least) this project was also extremely painful and required a lot of trial and error, debugging and scratching out ideas to come up with new ones that actually worked. Though we had already made the controls for the car, which would be translated into the spaceship, we had a lot of problems with the serial communication, to the point were we had to re-think the type of input we wanted to get from the controller. We originally wanted to use the joystick as the input for the car movement (left, right, forward and backwards). Nonetheless, since this value had to be mapped, it decreased the speed of the car significantly as it took some time for the new values to reach the Arduino. Furthermore, the button states gave integers of 8 and 0 instead of the designated 1 and 0 booleans that configuration stated which affected the serial communication. Armaan was able to fix this by  using the integers are the constrains for the conditions and the serial communication;. He spent two whole days (and most of the night) figuring out how to fix the problems we were having with the serial communication, I can attest.

A great part of our time was concentrated around the serial communication between processing and Arduino (specially after the LED light were incorporated in our model). It was very frustrating because one day the spaceship would be working perfectly, just for it to state (the next day) that it was having issues with variables sent by processing. Additionally, placing in the motors with the wheels was very difficult as the tape would sometimes fail and the wheel would go to the side, thus making movement harder. We solved it by adding an insane amount of 3M tape. It worked out at the end.

This is a video of Armaan placing the inputs of the controller to control the lights.

This is a video of how the car was working after adding the LED’s

Armaan’s hard work paid off because afterwards, we never had a problem with the serial communication (unless it was something like forgetting to plug in the Arduino Port).

Processing

For the processing, we created a stroke object that would be part of an array list that conformed the brush/trail of the spacecraft. Making the controls for the red, blue, green and multicolored options was not that difficult, however the yellow required more thinking since it is a combination of rgb colors.

This is a video of how they looked at the beginning.

We played around some more with the shapes, and once we were happy, we started working on a background that would represent the space theme of our project.

This is how it looked like:

We also changed things from this, like the color and the amount of stars. This became the background of our whole game.

We incorporated this background with the paint brush sketch to make this:

We also included commands to save the drawing and clear the sketch to start a new one. 

To save the sketches without erasing the previous one, we create a table that save the number of the sketch, reloads it into the program when the game starts and adds one, to keep count of each sketch and be able to save them in the drawing folder.

We then added the commands for the spaceship to move around leaving a trail. This was very hard, specially because the movement of the spacecraft had to have the same logic as the one that Arduino used to command the motors of the spacecraft. It took Thais a lot of time. Indeed, too much time one could say. But alas, we made it. Thais was so tired however, because she didn’t had suhoor and it was already 5am, that she forgot to record this part. Shame on her.

We then combined both the Arduino and the Processing parts to create our functioning game!

We were so proud of our little guy. He was already drawing. They grow up so fast ;-;

That day we even changed its first wheels. 😭

The Menu

The menu was another very time consuming, but extremely satisfying element of our project. Thais designed the UI for the menu while Armaan place everything and created the logic for the transitions.

We also created a character named O.L.B.A.P, a space artist that would introduce the user to how the game works and what a space artist does.

This is some of the UI used:

Play UI Button

Exit Button HoverCharacter Information Button HoverController Instruction Button Hover

Controller UI for instructions page

We worked on various versions of the UI and their location in order to make the interface as aesthetic and welcoming as possible. For this, we tried multiple fonts, colors, hover elements and text frames.

The main menu ended up looking like this:

This is the character information page:

Controller Instructions Page

Exit Page

This is the test for the main menu:

Thais made this attempt to dress our little spacecraft with cardboard. However, we ended up discarding this design because the tape tampered with the motors, but we’ll put the image here so that you can laugh at this sad attempt of a spaceship. We even put stickers to it!

Final Product

Here we have a demo of our project, explained by my amazing partner Armaan.

This is a link to our Notion page were we have the references we used, the minutes of some of our meetings, a few images that we used as inspiration, as well as the several versions of our code.

On the other hand this is a link to our final version of the code in GitHub.

Arduino Circuits

Motor:

LED’s

Thank you so much!

So, I would like to start by saying, Cole and I have a working bomb! xD

Despite some external circumstances, we both have been able to finish this project successfully; we have a bomb, which works, explodes, and has a full mechanism to be diffused, along with full game aesthetics and UI. Honestly, it was a really fun project to work on. Kudos to Cole for being an amazing partner 🙂

Please go through our previous documentations to get an idea of how our project came along:

1. Final Project Preliminary Idea – by Cole and Shreya
2. Final Project Progress – by Cole and Shreya
3. Final Project Update 2

Things added as a result of debugging:

Since we were dealing with both Arduino and Processing, if something went wrong, or wasn’t working the way we expected it, it was really hard to find out where the problem was. It could have been in the circuit, or the Arduino code, or the handshake (arising from the difference in speeds of Arduino and Processing), or the Processing code/logic. So, we used a lot of println() statements to check if the input/output was being read from the Arduino and being correctly passed on to Processing. One problem we faced was on stage 3 when you have to replicate the LED patterns. It would set the bomb off sometimes even if the pattern was correct. After racking our brain on what was incorrect in the logic of our code, we realized that the problem was with the discrepancy in the speed of Arduino and Processing (even after establishing the serial communication) and not our code. Processing wouldn’t read all buttons pressed on the Arduino every time. When we printed these out, we realized where the problem was. This resulted in the addition of a small flashing light on the Processing screen on the bottom left corner to indicate which button color was pressed and read by Processing. If that light doesn’t flash, that means the button input wasn’t read. This proved to be a very important signifier.

Making it a bit harder

So, we coded our program to wait for specific inputs at each stage. But, what if a user by mistake, presses something Arduino/Processing is not waiting for at a specific stage such as a button when submit was expected? Since it is a bomb, ideally it should go off if something incorrect is done. So, to make the game harder and interesting, but not letting the bomb go off at every wrong input, giving the user a little flexibility to play around and have fun, we decided to have 2 checks. 1) Make the bomb explode if an incorrect input is given from one of the input methods the particular stage is waiting for, 2) reduce the time left by 10 seconds if an incorrect input is given from a wrong input method. In our program, you can record an input by either pressing one of the four buttons on the Arduino screen or the submit button on the Processing screen for the distance and potentiometer. If an input is expected from the Arduino button and submit button is mistakenly clicked, it reduces the time. However, if a yellow Arduino button was expected, but a red one was pressed, then the bomb goes off.

Additional Touches and User Experience

We decided to add sounds for user experience and specific feedback to the user. For example, it was essential to indicate to the user when their time was reduced. So we added a sound for this to the Processing program. Then, of course, you want to hear an explosion right when the bomb goes off? Should look real. So, we decided to add a sound to both our end screens – if you win or lose. Finally, we also have the best sound of all, the bomb being defused and the game being won!

Game Design and UI

We made a total of 6 screens for the whole game (the last one having 2 parts) – 1) A start/menu screen, 2) a ‘how to play’ screen, 3) a player mode choosing screen which would navigate to the game screen or bomb instructions screen, 4) bomb instructions screen, 5) the game screen, 6) end screen: win screen and lose screen.

Honestly, UI takes quite some time to implement because of all the design, color choices, etc. We even added back buttons for easy navigation and a ‘play again’ button if the user wishes to go back and replay the game, switch roles with their partner, etc.

Our game is also designed in a way that it can be 2 players or 1 player. For one player, all you would have to do is keep a screenshot/printout of the bomb instructions printed out and handy.

See a video for each in the end!

User Testing

Shoutout to all our friends who played the game. Several nights we had friends trying out the game which helped us make several changes. We found that the instructions needed to be written in a more straightforward manner about the flow of the game, the non-technical language used to describe the Arduino, as well as some limitations in what Processing was handling.

Final Game Presentation

The video for Team Cole and Shreya:

The video for Cole v/s Shreya:

Code

The zip file for the complete code can be found here: https://drive.google.com/file/d/1tCewHvnM6zKLZ8ocNCzcTAer62c9MZxg/view?usp=sharing

It can also found on the Github Cole and I used regularly to merge our code together: https://github.com/crb596/IntroToIMFinal

Interesting Parts of the Code:

Processing/Arduino communication: As Processing is more powerful running on a laptop than on a computer, all the logic about which lights are on is done via Processing. Every handshake involves sending 4 boolean variables, one for each light, as well as two bomb state variables (exploded or defused) to the Arduino. Arduino receives this information, lights up the passed LEDs or goes into an exploded/defused state. Arduino then returns the state of each button, the distance meter reading, and the potentiometer back to Processing. 

Random LED: Each time the bomb goes into a new phase, the variables for each light are randomized within the constraints of that round. For example, in the round where a random amount of LEDs flash, first a random number 1-4 is chosen and then that many random LEDs are flashed.

Button Press/Used: Since Processing is running at 60fps(if we are lucky) and Arduino is running at a much faster rate, we had to add a variable that would determine if a button is being pushed, similar to the function mouseClicked(){}. We created a boolean variable that would be set to true whenever a button was recorded being down and back to false when released so we could keep track between rounds and extra inputs were not recorded.

Bomb timer: Processing gives current time in milliseconds using the function millis(). We used milliseconds as a unit to do all time calculations – recording start time of game, getting the current time, calculating time elapsed and time left, periodically updating those and also noting the final time taken to diffuse the bomb for the end screen. However, we wanted the time to be displayed in the usual format of minutes and seconds, so we made a convert time function to display it in the appropriate format.

Our Final Project:

Overall I am really happy with how our project came out. We went from a rough idea inspired by a VR game to a fully formulated and functioning team building game that really utilizes both the strengths of Arduino and Processing. Shreya was a great partner and we made this come together through great teamwork.

Here is our previous documentation that can show how we have come along

1. Final Project Preliminary Idea – by Cole and Shreya
2. Final Project Progress – by Cole and Shreya
3. Final Project Update 2

Things added as a result of debugging:

Since we were dealing with both Arduino and Processing, if something went wrong, or wasn’t working the way we expected it, it was really hard to find out where the problem was. It could have been in the circuit, or the Arduino code, or the handshake (arising from the difference in speeds of Arduino and Processing), or the Processing code/logic. So, we used a lot of println() statements to check if the input/output was being read from the Arduino and being correctly passed on to Processing. One problem we faced was on stage 3 when you have to replicate the LED patterns. It would set the bomb off sometimes even if the pattern was correct. After racking our brain on what was incorrect in the logic of our code, we realized that the problem was with the discrepancy in the speed of Arduino and Processing (even after establishing the serial communication) and not our code. Processing wouldn’t read all buttons pressed on the Arduino every time. When we printed these out, we realized where the problem was. This resulted in the addition of a small flashing light on the Processing screen on the bottom left corner to indicate which button color was pressed and read by Processing. If that light doesn’t flash, that means the button input wasn’t read. This proved to be a very important signifier.

Making it a bit harder

So, we coded our program to wait for specific inputs at each stage. But, what if a user by mistake, presses something Arduino/Processing is not waiting for at a specific stage such as a button when submit was expected? Since it is a bomb, ideally it should go off if something incorrect is done. So, to make the game harder and interesting, but not letting the bomb go off at every wrong input, giving the user a little flexibility to play around and have fun, we decided to have 2 checks. 1) Make the bomb explode if an incorrect input is given from one of the input methods the particular stage is waiting for, 2) reduce the time left by 10 seconds if an incorrect input is given from a wrong input method. In our program, you can record an input by either pressing one of the four buttons on the Arduino screen or the submit button on the Processing screen for the distance and potentiometer. If an input is expected from the Arduino button and submit button is mistakenly clicked, it reduces the time. However, if a yellow Arduino button was expected, but a red one was pressed, then the bomb goes off.

Additional Touches and User Experience

We decided to add sounds for user experience and specific feedback to the user. For example, it was essential to indicate to the user when their time was reduced. So we added a sound for this to the Processing program. Then, of course, you want to hear an explosion right when the bomb goes off? Should look real. So, we decided to add a sound to both our end screens – if you win or lose. Finally, we also have the best sound of all, the bomb being defused and the game being won!

Game Design and UI

We made a total of 6 screens for the whole game (the last one having 2 parts) – 1) A start/menu screen, 2) a ‘how to play’ screen, 3) a player mode choosing screen which would navigate to the game screen or bomb instructions screen, 4) bomb instructions screen, 5) the game screen, 6) end screen: win screen and lose screen.

Honestly, UI takes quite some time to implement because of all the design, color choices, etc. We even added back buttons for easy navigation and a ‘play again’ button if the user wishes to go back and replay the game, switch roles with their partner, etc.

Our game is also designed in a way that it can be 2 players or 1 player. For one player, all you would have to do is keep a screenshot/printout of the bomb instructions printed out and handy.

See a video for each in the end!

User Testing

Shoutout to all our friends who played the game. Several nights we had friends trying out the game which helped us make several changes. We found that the instructions needed to be written in a more straightforward manner about the flow of the game, non-technical language used to describe the Arduino, as well as some limitations in what Processing was handling.

Final Game Presentation

The video for Team Cole and Shreya:

The video for Cole v/s Shreya

Code

The complete code can be found here. 

https://drive.google.com/file/d/1tCewHvnM6zKLZ8ocNCzcTAer62c9MZxg/view?usp=sharing

It can also found on the Github Shreya and I used regularly to merge our code together

https://github.com/crb596/IntroToIMFinal

Interesting Parts of the Code:

Processing/Arduino communication: As Processing is more powerful running on a laptop than on a computer, all the logic about which lights are on is done via Processing. Every handshake involves sending 4 boolean variables, one for each light, as well as two bomb state variables (exploded or defused) to the Arduino. Arduino receives this information, lights up the passed LEDs or goes into an exploded/defused state. Arduino then returns the state of each button, the distance meter reading, and the potentiometer back to Processing. 

Random LED: Each time the bomb goes into a new phase, the variables for each light are randomized within the constraints of that round. For example, on the round where a random amount of LEDs flash, first a random number 1-4 is chosen and then that many random LEDs are flashed.

Button Press/Used: Since Processing is running at 60fps(if we are lucky) and Arduino is running at a much faster rate, we had to add a variable that would determine if a button is being pushed, similar to the function mouseClicked(){}. We created a boolean variable that would be set to true whenever a button was recorded being down and back to false when released so we could keep track between rounds and extra inputs were not recorded.

Bomb timer: Processing gives current time in milliseconds using the function millis(). We used milliseconds as a unit to do all timer calculations – recording start time of game, getting current time, calculating time elapsed and time left, periodically updating those and also noting the final time taken to diffuse the bomb for the end screen. However, we wanted the time to be displayed in the usual format of minutes and seconds, so we made a convert time function to display it in the appropriate format.

3 weeks (and a few changes to our preliminary idea) later, we finally have a working version of our game.

User testing videos, some screen recording demos, and a diagram of the Arduino setup can be found at the end of the post. The code for the project can be found on the github repository. A zipfile containing all the code and files can also be found at the end.

TLDR: the Arduino works better if it’s suspended in the air.

Game Concept:

This game is an NYUAD-themed game (yes we’re mentioning this for the third time over the past 3 weeks). The concept is very simple. The game consists of four levels which correspond to the four academic years of life at NYUAD, and each level consists of customized objects – both good and bad (for instance, you might see study away as an object but this only appears Sophomore year onwards. Similarly, capstone appears Junior year onwards). The player is represented as Faiza the Falcon at the bottom of the screen, and is controlled by moving the Arduino set in the horizontal direction. There are 2 metrics of performance, Academics and Health, both of which are set to 50 initially. Catching the good objects in each category will help improve the player’s performance in each. Similarly, catching the bad objects worsens the player’s performance in that particular category. The player has to take both performances to 100 to proceed to the next level and to eventually graduate!

Arduino-Processing Communication

Our Arduino consists of 3 sensors – a potentiometer, a button and an ultrasonic sensor. It also consists of a tricolored LED. The functions of all of these components are described below.

• The ultrasonic sensor measures the distance between the sensor and a frame (a solid vertical surface) and sends relevant information to Processing, which then maps this distance to Faiza’s position on the screen. In this manner, using the Arduino set in the horizontal direction allows us to move Faiza on the screen.
• The potentiometer is used to traverse between different options on the screen (where applicable). The use of the potentiometer can be found on the welcome screen and when the player loses a certain level (fails an academic year).
• The button is used to make selections wherever needed, at different points in the game.
• The tricolored LED shows how well the player is doing overall in the game. Processing uses a combination of both metrics of performance (Academic and Health) and sends a signal to Arduino to set the LED to the relevant color. Red, of course, indicates danger, green indicates that the player is safe and yellow indicates that the player should do better.

Code & Some features

• Item

This is the class which describes all the attributes and methods of items which fall down the screen. These attributes include the items’ x-coordinates within a range on the screen (75% of the screen), the y-coordinate, the sprite image, the number of frames which each sprite image consists of, and the value associated with each (good or bad) item. The update() function in the Item class ensures that each item falls down the screen with a random vertical velocity, which again, is within a certain range. It also contains a display() function which determines how each sprite has to be displayed on the screen.

• Player

This is the class representing the user and displaying the player’s avatar (Faiza the Falcon) on the screen. The main function of this class is the display() method which serves to display the correct frame of the sprite as well as flip it appropriately to illustrate movements to the right or to the left. This class also takes the mapped value of the distance sensor from the main program as the current position of the player on the screen.

• Level

This is the class that coordinates the different game functions and the interactions between Player and Items. Since different levels have different sets of available objects, and said objects are further divided into categories, the Level class has a number of lists using different Java data structures to store necessary.

item_names is an ArrayList containing the names of all the objects available for the current level. We put these names in four .csv files, which are used to populate the item_names list. ArrayList is helpful for this list because it supports functions like contains(), size(), add().

frames and values are two HashMap with String keys and Integer values. Since different items have different numbers of sprite frames and values, these can be read (also from .csv files) and later called upon. We want to index these values using item names, which is why we decide to use HashMaps.

posAcademic, negAcademic, posHealth, and negHealth are four StringList storing the names of items that belong to four categories (as the names suggest, positive and negative for academics and health). The StringList structure is mostly identical to an ArrayList, only that it’s limited to storing String data. We recently discovered this data type and wanted to try it out. StringList supports the function hasValue() that can quickly check if a list contains a certain value, which comes in handy for our program.

We add two more StringLists positives and negatives which store the names of all the positive items and negative items respectively, regardless of which metrics they involve. These lists are mainly used for randomizing new items.

rare_items_count and rare_items_catch are two HashMaps using the names of rare items (described in details below) as keys to store how many times an object has appeared in a level, and how many times it has been caught by the player, respectively. These lists are used to put a limit on the frequencies of rare items appearing in a level.

The last list of this class is items, an ArrayList that stores Item objects. New items are created and added to this list, and later picked out to display on the screen.

Some other attributes of the Level class includes levelNum (0-3 corresponding to four academic years), academic and health to store Academics and Health metrics (default 50 each), a number of background images for different screens, away (boolean value to keep track of whether the player has caught a Study away item), etc.

The Level class also contains a number of methods, as will be explained below.

loadData() is called upon construction of an object. This method reads from the various .csv files we prepared beforehand to populate relevant lists as mentioned above. We first started out filling the lists manually, but then cleaned up the data into .csvs files instead to reduce confusion should we want to add new items, change frames, values, etc.

displayBackground() checks which level is currently played to display the correct background image. It also checks if the player is currently ‘studying away’.

addItems() randomizes new items for the game. We settle with the rate of two new items per second, one positive and one negative (we use positives and negatives lists for this). This method also takes into account rare_items_count and rare_items_catch to limit these items. displayItems() then picks items out of the list and display them.

checkCollision() iterates through the list of items to check whether Faiza the Falcon is colliding with any of them. Upon collision, the method checks the category to which the item belongs to adjust the performance metrics accordingly.

There are also a number of helper methods like colliding() (returning a boolean value of whether the player is ‘catching’ the item), lost() and won(), increase/decrease Health/Academic().

• Level_test

This is the main game program that coordinates different levels and different screens, and it also processes data from Arduino program and sends data back.

Most of the screens of the game either has one option (selected through pressing the button) or two options. For those with two options, we use a threshold on the reading of the potentiometer to display which option is being selected (signified with >> <<). Pressing the button will select the highlighted option.

The main program reads from the Arduino programs three sensor values: button state, potentiometer values, and distance sensor value.

As described in the documentation of a previous week, we use a formula from an earlier lecture to add smoothing to the value reading from the distance measuring sensor. In our program, we use a smoothing factor of 0.2, which eases the movement of the player. We also add a threshold in case sudden jumps in distance are detected. The player will only move if the new position mapped from the distance measuring sensor is within 1/3 of screen width from the position of the player recorded in the previous frame.

The main program also inspects the current performance metrics and sends appropriate values to the Arduino program, which will control the color of the LED.

Features

• The game consists of three rare items – Study away, J-term and Capstone. Since these are objects which aren’t as common as others (both in the context of the game and in reality), they function in a slightly different manner. For instance, a person can only get one J-term, one study away and one capstone in an academic year (in reality), hence this is what we implemented in our game too. The player is only presented with a maximum of three of each of these rare items within a certain level (this was implemented using certain boolean values). However, once the player catches one item of each of these types, that particular item won’t reappear in the same academic year in the game. To put this into perspective, if the player catches an item corresponding to a J-term, then a J-term will never appear again in the same academic year (level) and it will only appear a maximum of three times in case the player never manages to catch it.
• In order to make the game a little more interactive and NYUAD-themed, we used several different images of the NYUAD campus as backgrounds for the different levels. However, there’s something even nicer to add to this! If the player catches an item which corresponds to a study away, the background changes to an image of the New York campus (yes this is the only way we can get a semester in New York during Covid times).
• The sprites used to represent each of the items were created using https://www.piskelapp.com/. This was a very time-consuming but interesting process as we had to spend a lot of time paying attention to detail. It was also necessary to make all sprites ourselves because all items are NYUAD-themed and it was impossible to find anything on the internet.

Problems

• We first planned to use the buzzer to play a sound upon collision with an item, but we ran into some problems with it. At one point, we got it to work as we wanted, but we noticed that some items triggered a sound when colliding with the player, while some others didn’t. We didn’t really understand what caused the problem, and the item choices seemed quite arbitrary. When we got back to working on it a few days later, for some reason no sound was playing. We suspected it might had something to do with the value we were sending from Processing to Arduino, but when we tried printing the value out in Processing while sending it to Arduino at the same time, it showed the correct value we wanted. We consulted Professor Aaron and worked on it some more, but we were not able to pinpoint what the problem actually was. Since we already tested the Processing program, and the other value sent from Processing to Arduino (LED value) worked without trouble, it might have been due to the buzzer itself. But since we only have one buzzer between the two of us, we were not able to test this out. In the end, we decided to not use the buzzer and replace it with the sound library of Processing.
• When we first tried controlling the position of Faiza on the screen using the Arduino, we had to smoothen the values which were sent by the distance sensor and that gave us pretty good results. The description of that process can be found in one of our earlier posts, the link to which is attached. https://intro.nyuadim.com/2021/04/19/final-project-progress-ngoc-and-maaz/ For the most part, this allowed us to swiftly control Faiza using the Arduino but on the very last day, it would sometimes give weird results. However, lifting the Arduino set slightly above the solid surface allowed us to have results which were a lot more stable than what we got when we placed it on a table. The solution we found was completely coincidental but it improved the game play more than we expected.
• At times, we faced difficulties trying to figure out how to store all items which were used in the game, particularly because each level had a separate list of good and bad items and each item had a different number of frames for its sprite, as well as a different value associated to it. With a lot of research and experimentation, we figured out these issues using some data structures which are described in the Code and some features section above.

Reflection

Ngoc: For the second half of the semester and during working on the project, I learned a lot about using the Arduino (which was something I’d been looking forward to since high school, I was not disappointed!). Our project also includes components from earlier lectures, so I think it was a nice wrap-up for what we have learned from the class. Working in a group also made me a lot more responsible, in that I could not afford to wait until the last week to start seriously working. I also came to realize I’m really lacking on the creative side, so working in a group really balanced this out (kudos to Maaz and his collection of sprites).

Maaz: When I started working with the Arduino at the beginning of Spring 2, I wasn’t comfortable with handling the components without supervision, particularly because I have never been a very big fan of working with hardware. Looking at this project and making my friends play the game, I am super proud of what we have created. This process involved a lot of problems, followed by a lot of research and problem-solving. Two days ago, Ngoc and I spent around 20 minutes solving a problem which was caused by using something called shapeMode() instead of rectMode() (sorry Professor, I know you won’t be proud). I have learnt a lot from working with Ngoc; from having a blank face when she first suggested we make a HashMap to making one on my own, this was an extremely nice process of learning and creating something which I am extremely proud of.

Arduino Circuit

User-testing

Demo

Zipfile: not-so-creative-final-project-master (2)

I’ve been working on the final project for the last few days, and I’ve run into a few problems along the way. Now I will share the progress at the moment, as well as the main challenges. I finished the main part of the code, where I established serial communication between Arduino and Processing. You can read the whole code here. Some challenges I encountered: it took time to establish serial communication for two variables for several reasons. One example is how I forgot to change “println” to “print” when sending the information from the distance sensor. Little mistake caused 30 minutes of checking every single line of code. Also, making generative art of different colors was challenging because the order of lines could change everything.

Here you can see the progress since the last week:

>>>

>>>>>>

>>>

In order to demonstrate how my program works, I asked my friend to take a video of me, you can watch it here.

Testing was done by my friend, who helped to identify several shortcomings. First, the interface is not clear enough, so I had to explain some of the details myself, for example, where to click to take a screenshot. Now I know where to add any signifiers. Secondly, clicking on the buttons in the processing was not very convenient, and I’m thinking of moving them to the Arduino. Third, there is a lack of a final screen where you can go back and start again or end the session.

I need to work with the interface base on the feedback I received during testing, add music, work with the representation of Arduino (make it more convenient to use and intuitive), maybe add LEDs as a signifier.

For me, the hardest part (other than the coding *starts crying*) is not to change my idea. My initial idea was to recreate WHACK-A-MOLE game, but since I’m recreating it to be my own I’m going to make it WHACK-A-VAMP (whacking vampires instead of moles hahaha so creative). And so, to insure that I won’t change my idea, I started drawing the background *I’m drawing the characters and the setup of the game on my own*

// variable declaration
const int rLED = 2;
const int bLED = 3;
const int gLED = 4;
const int yLED = 5;

const int rButton = 9;
const int bButton = 8;
const int gButton = 7;
const int yButton = 6;

void setup()
{
  //Set LED pins, displays light
  pinMode(rLED, OUTPUT);
  pinMode(bLED, OUTPUT);
  pinMode(gLED, OUTPUT);
  pinMode(yLED, OUTPUT);
 
  //Set Buttons as inputs. controls if the user whacked the vamp
  pinMode(rButton, INPUT);
  pinMode(bButton, INPUT);
  pinMode(gButton, INPUT);
  pinMode(yButton, INPUT);
}

void loop()
{
// here is the code for when the LED lights are represented as the vamps so that once i'm done with the processing animation part i would just add another condition which is for the vamp to be shown on screen
for(int i = 0; i < 10; i++)
  {
    //Randomly turns on an LED
    digitalWrite(random(2, 6), HIGH);
   
    //Checks if any of the LEDs are on
    while(digitalRead(rLED) == HIGH || digitalRead(bLED) == HIGH || digitalRead(gLED) == HIGH ||digitalRead(yLED) == HIGH)
    {
      //Functions that turns off the randomly lit LED
      if (digitalRead(rLED) == HIGH)
      {
        // Makes sure button is pressed
        if (digitalRead(rButton) == HIGH)
        {
          //Turns off LED
          digitalWrite(rLED, LOW);
        }
      }

      if (digitalRead(bLED) == HIGH)
      {
        // Makes sure button is pressed
        if(digitalRead(bButton) == HIGH)
        {  
          //Turns off LED
          digitalWrite(bLED, LOW);
        }
      }

      if (digitalRead(gLED) == HIGH)
      {
        //Makes sure button is pressed
        if(digitalRead(gButton) == HIGH)
        {
          //Turns off LED
          digitalWrite(gLED, LOW);
        }
      }

      if (digitalRead(yLED) == HIGH)
      {
        // Makes sure button is pressed
        if(digitalRead(yButton) == HIGH)
        {
          //Turns off LED
          digitalWrite(yLED, LOW);
        }
      }
      delay(100);
    }
  }
 
// ends the game
for (int i = 0; i < 3; i++)
  {
    digitalWrite(rLED, HIGH);
    digitalWrite(bLED, HIGH);
    digitalWrite(gLED, HIGH);
    digitalWrite(yLED, HIGH);

    delay(250);

    digitalWrite(rLED, LOW);
    digitalWrite(bLED, LOW);
    digitalWrite(gLED, LOW);
    digitalWrite(yLED, LOW);

    delay(250);
  }
}