Author: Mariam Alameri

Concept:

For my final project, I created an interactive simulation called Brew Your Coffee, which the title itself reflects clearly. Inspired by my love of coffee, and specifically the process of making a cup of coffee, I wanted to create an experience where the user could go through a fun and interactive coffee-brewing simulation using both a p5 sketch and Arduino physical components.

The project aims to create a realistic and engaging experience through a self-made coffee machine model that includes actual coffee-making objects and a sensor that detects the user’s physical movements. By following the provided guide and completing each step, the user gets to simulate brewing their own freshly made cup of coffee!

Visual Documentation:

For the main model, I mainly used cardboard. I measured and cut the pieces using a cutter, then assembled and attached them together using a hot glue gun.

I first created openings for the visible components, including the button, joystick, LED module, and ultrasonic sensor, then started setting everything up using the Arduino, breadboard, jumper wires, and female-to-male jumper wires for the components that needed longer connections.

Here is the back of my model showing the wiring setup. The lower opening contains the joystick and button wires extending downward into the popped-out tilted section to make the controls more comfortable and easier for the user to interact with.

After securing the wiring and making sure the Arduino setup was functioning correctly, I assembled the structured pieces together into the coffee machine model I had envisioned. I then added the title of the simulation as the name of the machine, along with labels for the different components such as the controls, sensor, and LEDs to make the interaction clearer for the user. I also decided to include actual coffee-making objects that matched the visuals shown on the p5 sketch in order to create a more realistic and immersive experience.

Setup:

Fully aligned with the actual arduino:

Schematic:

User Testing: 

Interaction design:

The interaction design of my project was created to make the experience feel simple, clear, and realistic for the user. I wanted the interaction to imitate the actual process of making coffee while still being easy and enjoyable to understand.

The user first moves through the different screens using the physical button and joystick attached to the Arduino setup. The joystick allows them to navigate between the coffee-making steps, while the button is used to confirm selections and move forward in the experience.

Once an action begins, the interaction changes from button-based navigation to physical movement. The ultrasonic sensor detects the user’s hand movements in front of the coffee machine model, which then progresses the animations on the p5 sketch with audio. This was designed to make the user feel more involved in the process rather than only pressing buttons on a screen.

The LED lights were also part of the interaction design and acted as visual feedback for the user. Red indicates that the action is ready to begin, yellow shows that movement is currently being detected, and green indicates that the step has been completed successfully. Along with the sprite animations and sound effects, these interactions helped make the experience feel more responsive and immersive.

Arduino code:

The Arduino code handles all the physical interactions of the project, including the joystick, button, ultrasonic sensor, and RGB LED module. It detects the user’s inputs, sends simple serial signals to the p5 sketch for navigation and movement interactions, and receives signals back from p5 to control the LED feedback colors during the experience.

Link to GitHub Code

One of the key parts of the code was the joystick navigation between the coffee-making options. The joystick continuously reads analog values from the X-axis, and depending on the direction the user moves it, the Arduino sends either L for left or R for right to the p5 sketch through serial communication. I also added threshold values and timing delays so the navigation would feel smoother and avoid rapidly repeating the same signal multiple times from one movement. This made the interaction more controlled and easier for the user to navigate.

// Joystick left & right navigation
  int joystickX = analogRead(joystickXPin);

  // Send "L" or "R" when joystick passes threshold, using delay to avoid rapid repeated signals
  if (millis() - lastJoystickTime > joystickDelay) {
    if (joystickX < joystickLeftThreshold) {
      Serial.println("L");
      lastJoystickTime = millis();
    } else if (joystickX > joystickRightThreshold) {
      Serial.println("R");
      lastJoystickTime = millis();
    }

Another important part of the code was the ultrasonic sensor interaction. The sensor measures the distance between the user’s hand and the coffee machine setup. To make the interaction feel realistic, I did not only check the distance itself, but also checked whether the distance was changing enough to count as actual movement. This avoids possible triggering and helped make the animations progress only when the user was actively moving their hand in front of the sensor. I also used threshold ranges and delays to make the motion detection more stable and responsive.

// Movement detection using ultrasonic sensor
  int distance = detectDistance();
  int difference = abs(distance - lastDistance);

  // Send "M" when an object is within range and its distance changes enough to indicate movement
  if (millis() - lastMotionTime > motionDelay) {
    if (distance > minMotionDistance && distance < maxMotionDistance && difference > motionChangeThreshold) {
      Serial.println("M");
      lastMotionTime = millis();
    }

P5.js code:

The p5 code controls the visual and interactive side of the project, including the screens, sprite sheet animations, audio, and overall experience flow. It receives signals from Arduino to activate the interface and progress the actions, while also sending signals back to Arduino to align the LED feedback with the current interaction state.

One of the major parts of the code was creating and controlling the sprite sheet animations for each coffee-making action. I created a class to organize the actions and make each animation easier to manage. The animations work by increasing a progress value whenever movement is detected from the ultrasonic sensor. That progress is then mapped to different frames of the sprite sheet using the map() function. I also used constrain() to make sure the frame number never goes outside the sprite sheet range. Once the action reaches its maximum progress, the animation locks onto the final frame and marks the action as completed.

// Map interaction progress to sprite sheet frames
   this.frame = floor(map(this.progress, 0, maxProgress, 0, this.totalFrames));
   this.frame = constrain(this.frame, 0, this.totalFrames - 1);

   // Lock animation on the final frame once complete
   if (this.progress >= maxProgress) {
     this.frame = this.totalFrames - 1;
     this.done = true;
   }
 }

Another important part of the code was combining audio and LED feedback with the movement interactions. Whenever the user performs movement detected by the ultrasonic sensor, the correct sound effect begins playing depending on the current action. At the same time, the p5 sketch sends signals back to Arduino to control the LED colors. Once the action is completed, the audio stops and the LED changes to green to visually indicate completion to the user.

// Progresses the current action when movement is detected
function handleMovement() {
  let action = getCurrentAction();

  if (action.done) return;

  // Different actions trigger different audio feedback
  if (currentAction === "grinder") {
    grinderAudio.loop();
  }

  if (currentAction === "steamer" && !steamerAudio.isPlaying()) {
    steamerAudio.loop();
  }

  if (currentAction === "coffee" && !coffeeAudio.isPlaying()) {
    coffeeAudio.play();
  }

  action.update();

  // Once the step is completed, stop audio and switch LED to green
  if (action.done) {
    stopAudios();
    sendLed("G");
  }
}

Note: For the audio if statements, the grinder action has slightly different feedback behavior compared to the other two actions. The steamer and coffee audio only begin once the ultrasonic sensor detects movement, while the grinder audio starts when the action screen appears. I made this decision based on the grinder audio file I used, since its first few seconds are mostly silent before the grinding sound begins, which ended up aligning naturally with the user’s interaction timing.

Communication between Arduino and p5.js:

The communication between Arduino and p5.js in my project works through serial communication. The Arduino is responsible for handling all the physical interactions, such as the joystick, button, ultrasonic sensor, and LED lights, while the p5 sketch controls the visuals, animations, states, and audio on the screen.

The Arduino continuously sends simple letter signals to p5 depending on the user’s interaction. For example, when the button is pressed, Arduino sends a signal to move between screens or confirm a step. The joystick sends left and right signals to navigate between the coffee-making options, while the ultrasonic sensor sends movement signals to progress the animations during the actions.

At the same time, p5 also sends signals back to Arduino to control the LED lights. Different letters are sent depending on the state of the interaction, such as red when an action starts, yellow while the user is actively moving, and green once the action is completed.

This communication allowed both the physical Arduino setup and the digital p5 sketch to work together as one interactive system.

Aspects of the project I’m proud of:

One of the aspects I am most proud of is being able to successfully combine the visuals, interaction, and physical setup into one engaging experience. I am especially proud of finalizing the images and coding the interaction between the p5 sketch and Arduino components in a way that felt smooth, interactive, and enjoyable for the user.

I am also particularly proud of using sprite sheets and audio together to visualize movement and actions throughout the coffee-making process. Since this was completely new to me, it required a lot of experimenting, troubleshooting, and problem-solving, but I was very happy to eventually make the animations and interactions work together in a realistic and immersive way.

In addition, I am proud of my physical coffee machine model and how it turned out visually. At first, building the structure felt very challenging and almost impossible, especially because I wanted it to resemble a real coffee machine. However, through patience, planning, and problem-solving, I was able to successfully complete the model and add visual details and labels that improved both the aesthetic and the user experience.

Challenges faced and how I tried to overcome them:

Honestly, working on this project was a rollercoaster ride. Some parts went smoothly, while others were much more challenging than I expected. Most of the challenges I faced were related to the visuals and animations of the p5 sketch, especially because I created all of the images and sprite sheets from scratch using AI while still trying to achieve a very specific vision and aesthetic.

At first, the visuals were not working well and did not look as appealing or consistent as I wanted, especially the sprite sheet animations. To overcome this, I became extremely specific with the prompts and references I provided. I used exact images of the real coffee-making items I had and carefully described the animation style and movement I was aiming for. Even after that, creating the sprite sheets required a lot of patience, since many results came out incorrect or inconsistent. Eventually, after multiple attempts and feedback from the professor, I was able to achieve a result that matched my vision much better.

Here are some images from my p5 sketch. All of these screens and interactions can also be fully experienced directly through the embedded p5 sketch:

Another major challenge was aligning and displaying the sprite sheets correctly inside the p5 sketch. Some frames would shift position, go off screen, or get cropped incorrectly during the animations. At some points, multiple sprites would appear at once, while other frames looked unaligned or partially cropped. To solve this, I watched tutorials, reviewed references, and experimented with frame sizing, cropping, and scaling until I fully understood how sprite sheets and frame positioning worked within my code. This process helped me improve the consistency and quality of the animations.

// Frame widths based on each exported sprite sheet
  getFrameWidth() {
    if (this.name === "tamper") return 1040 / 7;
    if (this.name === "grinder") return 989 / 7;
    if (this.name === "coffee") return 612 / 6;
    if (this.name === "steamer") return 774 / 8;
  }

// Select the current frame from the sprite sheet
    let cropX = round(this.frame * frameW);
    let cropY = 0;
    let cropW = round(frameW);
    let cropH = frameH;

    let spriteW = cropW * scaleFactor;
    let spriteH = frameH * scaleFactor;

    // Center sprite on screen
    let spriteX = width / 2 - spriteW / 2;
    let spriteY = height / 2 - spriteH / 2;

    // Tamper sprite sheet needed slight position adjustment
    if (this.name === "tamper") {
      spriteX += 80;
      spriteY += 20;
    }

Note: For the frame widths, I used another website called Photopea to calculate how many pixels wide each sprite sheet was, then divided that value by the number of frames to calculate the width of each individual frame. I then continued experimenting and adjusting the positioning and scaling until the animations appeared more consistent on screen.

The tamper action also has an individual adjustment at the end to slightly change its X and Y position, since that specific sprite sheet frames were more visually unaligned compared to the others.

Implementing Feedback:

During the first user testing drafts, I had a user try my project and noticed two important improvements that I should make, which I then implemented into the final version.

First, I added the ultrasonic sensor to the instructions page so the user would clearly understand where to place their hand and how to interact with the movement actions. Here is the before and after of the instructions page:

Second, I added a short step guide on the main setup screen so the user could quickly remember the coffee-making order in case they forgot the steps during the interaction. I created the mini guide visually, then coded it to only appear on the main setup screen and adjusted its position to fit naturally within the interface. Here is the mini guide and its code:

// Display mini guide only on the main setup screen
    image(miniGuide, width * 0, height * 0, width * 0.2, height * 0.35);
  } else if (state === "action") {
    getCurrentAction().display(bg);
  } else if (state === "end") {
    endScreen.display();
  }
}

Areas for future improvement:

Overall, I feel very satisfied with my final result, and I am proud of myself for making my idea work, especially because it was much more challenging than I initially expected. However, there are still areas that could be improved.

I think I could make the experience feel even more realistic by creating longer and more detailed sprite sheets with smoother movement to better represent the live coffee-making actions. I would also like to improve the physical setup by creating a more visually appealing coffee machine model and refining the overall presentation of the interaction.

Since I struggled a lot with the sprite sheets during this project, I think learning how to manually create and align them myself in the future would make it much easier to keep the animations more stable and consistent. It would also give me more control over the positioning, scaling, and overall quality of the animations.

For the future, I would love to create more simulations and interactive ideas using communication between p5 sketches and Arduino. I would also like to explore something more game-based to make the experience even more exciting, such as a café challenge game where the user has to prepare and serve orders for customers. I could even expand the idea further into a full restaurant simulation with multiple menu options and interactions.

Resources:

Arduino:

For my Arduino code, I mostly used the templates we were given in class to understand the structure, along with the official Arduino references to understand specific functions and actions I needed:

https://docs.arduino.cc/language-reference/en/functions/digital-io/pinMode/ 

https://docs.arduino.cc/language-reference/en/variables/constants/inputOutputPullup/ 

https://docs.arduino.cc/language-reference/en/variables/data-types/unsignedLong/ 

https://docs.arduino.cc/language-reference/en/variables/data-types/bool/

I also watched some tutorials for additional understanding:
https://youtu.be/KGwtit2bFyo?si=Fyh10tn7at7zFYyo https://youtu.be/vo7SbVhW3pE?si=PoRPErpxfsdc1cs5 

P5:

For the p5 code, I mostly looked back at our lecture slides and what we covered in class, and I also referred to some of my previous sketches. I used the official p5 references to review and better understand specific commands:

https://p5js.org/reference/p5/image/

https://p5js.org/reference/p5/floor/

http://p5js.org/reference/p5/constrain/

I also watched multiple tutorials to help me achieve the visual actions and sprite animations I wanted:
https://youtu.be/lT_q-ylhML0?si=pfqHTvgWGA_ONRQn https://youtu.be/i2C1hrJMwz0?si=QiOL9T3fHeHro-4e
https://youtu.be/Pn1g1wjxl_0?si=YPGGnEIVbr6oa3yt

Connections:

For the Arduino and p5 communication, I mainly referred to the serial communication slides we covered in class, along with these videos for additional understanding:
https://youtu.be/MtO1nDoM41Y?si=KiDzo6fA5sIav8xj https://youtu.be/MHJ6KpgE7j4?si=lm94nLPvr4QUqhRO 

Referencing of use of AI tools:

I also used AI for support once I faced issues with my code. One major example was when I had trouble with the sprite sheets, as mentioned above, since they would completely go off screen or get cropped during some frames no matter what I tried. After reviewing references and tutorials, AI specifically explained how sprite sheets worked in my context, and that the main issue was related to cropping and frame sizing within the sprite sheets. It guided me through the process of fixing and improving them.

After experimenting with frame calculations and sprite sizing myself, AI further helped explain how sprite sheet cropping, scaling, and frame positioning worked in my specific setup. AI further helped explain how sprite sheets function using total pixels divided by frames, and how functions such as getScaleFactor() could help visually adjust and balance each animation more consistently on screen.

Most of the actions improved after that except for the tamp action. I explained the issue I was still facing, and it walked me through it, leading me to use a specific if statement for the tamping action to adjust its X and Y position separately from the other animations. Here:

// Tamper sprite sheet needed slight position adjustment
    if (this.name === "tamper") {
      spriteX += 80;
      spriteY += 20;
    }

I also used AI assistance while working with the RGB LED module, since I could not find tutorials that matched the exact LED model and interaction style I wanted. It helped me understand how the 4-pin RGB LED worked in my specific setup, how to wire it correctly with Arduino output pins, and how to match the LED colors with the p5 states through serial communication. This is the one I used:

I also used AI to help generate the visuals for my project. I described the exact theme, colors, layout, and style I wanted, then kept refining each image until it matched my vision. This was mainly used for the instructions page, setup screen, and the detailed images for each step. I specifically directed what elements and style I wanted included in each visual.

In addition, I used AI to help create the sprite sheets because I could not find ready-made ones that matched the specific aesthetic and coffee-making actions I wanted for my project.

Documentation:

Reflection:

My project was generally easy for users to figure out, mainly because instructions are provided at each step. One point of confusion was that the experience begins with the tamper selected, which made users unsure whether to start there. However, after referring to the steps, they were able to correct themselves and continue. One user also initially tried to use the joystick instead of the sensor for movement, but quickly understood the correct interaction. Overall, users were able to grasp the mapping between the controls and the actions and complete the process smoothly.

Most parts of the experience are working well, especially since the instructions are clearly displayed within the sketch. However, this also depends on the user taking the time to read them. Areas that could be improved include the clarity of the sprite sheet animations and making the interaction with the ultrasonic sensor more intuitive. This could be addressed by refining the instructions to more clearly indicate that movement in front of the sensor is required.

The main aspects I needed to explain were that users should use the physical button rather than the keyboard, and that the animations are controlled through hand movement using the ultrasonic sensor instead of controls. I believe this will become more intuitive once the physical setup is finalized and the wiring is concealed, allowing users to focus entirely on the interaction without distractions.

Concept:
For my final project, I am creating an interactive coffee simulation in which users engage between a digital interface and physical inputs using Arduino. The project allows users to navigate a coffee-making simulation through a p5 sketch displayed on screen, while using tangible tools connected to the Arduino to perform actions.
The goal is to move beyond simple selection-based interaction by allowing users to actively participate in the process of making a cup of coffee. As users perform each step, they receive real-time visual feedback on the screen, along with physical and auditory responses that reflect their actions.
By combining digital visuals with hands-on interaction and realistic sound elements, the project aims to create an engaging and immersive experience that simulates the process of preparing coffee in an intuitive and enjoyable way.

Design and Description of Arduino:
The Arduino program will use inputs and outputs to control the physical interaction. The main input will be the ultrasonic sensor, which reads distance and is used to detect the user’s movements. This data will be interpreted differently depending on the current step, allowing the same sensor to represent actions such as grinding, pressing, steaming, and pouring.
I will also use a push button for confirmation actions, such as starting the system, selecting steps, and moving to the next stage. A joystick module will be used to navigate between tools, where the Arduino reads its values to determine direction. For outputs, I will use three LEDs (red, yellow, and green) to provide feedback. Red indicates a step has not started, yellow shows the process is in progress, and green indicates completion.
The Arduino will send sensor values, joystick input, and button states to p5.js through serial communication. It will also receive information from p5.js about the current step, allowing it to correctly interpret inputs and control the LEDs.

Design and Description of P5:
The p5.js program will manage all visuals and the overall flow of the interaction. It will control the different screens, including the start screen, instructions page, and the main coffee-making interface, as well as display the navigation system using the joystick.
During each step, p5.js will use sprite sheets to display animations that represent the action being performed. These visuals will update in real time based on data received from the Arduino, especially the ultrasonic sensor, creating a direct connection between the user’s physical actions and the digital response.
The p5 program will receive sensor data, joystick movement, and button inputs from Arduino to update visuals and control progression. It will also send the current step information back to Arduino so it can adjust how inputs are interpreted and control the LED feedback.

Materials:
For my project, I will be using components from my Arduino kit, including the Arduino board, breadboard, and jumper wires to build the system. I will use an ultrasonic sensor as the main input to detect the user’s movements and gestures during the coffee-making process. A push button will be used for confirmation actions, such as starting the steps, and a joystick will allow the user to navigate between the different tools or stages. I will also include three LEDs (red, yellow, and green) to provide clear feedback, indicating when the system has not started, is in progress, and when a step is completed. I will be 3D printing or creating physical coffee-making tools such as a grinder, tamper, and cup to make the interaction more realistic. Using the stipend, I will purchase the joystick and build a simple coffee machine-style enclosure using materials like cardboard or foam board to organize the components and hide the wiring. This will improve the overall presentation and create a more immersive and polished experience.

Process:
So far, I have finalized my concept and organized all my ideas, along with doing some research to make sure everything can work the way I imagine it, especially using the ultrasonic sensor with movement and sprite sheet animations.
I have also started working on the visual side by creating images for the layout I plan to follow in p5.js. This includes the start page, the instruction guide, and the main setup with the first step of the process. In addition, I have begun creating the first sprite sheet for the initial movement so that the animation can respond to the user’s interaction.
I have also started preparing the structure of my p5 sketch to integrate these elements. My next step is to import these visuals into the p5 sketch and start coding them to work properly, making sure the interaction functions as expected. After that, I will continue adding the remaining steps using the same structure and process.

Images:

Concept:

For my final project, I want to create an interactive game that smoothly processes between the p5 sketch and Arduino and provides a realistic experience for users. I chose to create an interactive restaurant game/experience where the user is set in a kitchen with multiple sections, such as a food station, desert station, and coffee machine, and gets to choose a section and actually experience creating it as if it were real.

Inspiration:

I actually had multiple inspirations that helped me come up with this idea. Firstly, one of my options for my midterm project was an Emirati kitchen on p5, but I ended up choosing another one, so I thought it would be a good idea to use it for my final project. I also thought of many cooking games I used to play, such as Cooking Mama from my childhood, and Cooking Fever, which is more recent. Then, because I wanted it to feel more realistic, I came up with more interactive ideas.

Vision:

I have a specific vision for my final project that I wish to create if possible. Starting with the p5 sketch, I will have an aesthetic but colorful kitchen setup with around 3 different sections, and in each section there will be animations of the actions the user can perform. On the Arduino side, I would like to create a simple controller, such as arrow buttons or a joystick, to move between selections, and a push button for confirmation. I also want to include an ultrasonic sensor and create kitchen utensils where players can move in front of the sensor to finish the activity. I might also add LED lights of different colors to indicate the state of completion, such as red before starting, yellow during the process, and green when it is done.

References: 

For P5, along with recapping what we learned:

https://youtu.be/b2s8yZ06waQ?si=Su3dHMMqvrrCNXIz  
https://youtu.be/HfvTNIe2IaQ?si=EhtGjs7IjOrHAGLF

For Arduino, along with recapping what we learned:

https://youtu.be/vo7SbVhW3pE?si=ZUV6hNZY7ecwRSva
https://youtu.be/wTfSfhjhAU0?si=So-vFN7DNnjQD3hn
https://youtu.be/a37xWuNJsQI?si=uddzGXgVkkTvSW1k

This week’s reading, Design Meets Disability, made me realize how limited my understanding and awareness of design in this aspect was. I initially thought that design in functional contexts was mainly about creating something accessible for problem-solving. I also thought that disabilities were something people might feel worried or concerned about when it comes to appearance. What really stuck with me was the idea that disability can actually inspire mainstream design, not just be something that adapts to it. I was particularly interested when I read the discussion about glasses, as I never really thought of them as related to a serious disability, and that is because of how normalized they are nowadays. People who need glasses wear them openly, and they come in many different styles from different brands. Along with this, the example of the Eames leg splint showed how something designed under strict constraints for injured soldiers led to completely new techniques that later influenced furniture design. This made me understand that disabilities are not always limitations, but can actually lead to innovation.

Moreover, comparing this to hearing aids made the contrast very clear, where focusing too much on hiding them can limit both their function and how people feel about using them. This also made me think of different applications, and the one that stood out most to me was prosthetic arms and legs. They started off as very visible and mechanical, made of materials like metal or wood, but with improvements, they are becoming more realistic, and sometimes even designed in creative ways. I felt like this shows a combination of both hiding the disability and expressing design, which highlights how complex this topic can be. Overall, this reading made me realize that design is not just about fixing an issue, but also about experimenting and thinking beyond it. This connects a lot to the work we do, because I sometimes focus too much on making something work, without thinking about how I can push it further or make it more creative.

Exercise 1:

P5 code:

https://editor.p5js.org/farahshaer/sketches/K1ETNJevI 

Arduino code:

https://github.com/farahshaer/Intro-to-IM/blob/15f03b7108b0eead04ecc8fbade23e0703c774e4/sketch_apr16a.ino 

Process:

So for the first part, it was a one way communication between arduino and p5 with the idea of taking an input from a sensor on arduino and using that data to control the circle on p5. I used a potentiometer as my sensor because it gives a smooth range of values which makes it easier to control the movement on the screen. The arduino reads the values and sends it to p5 using serial.prin1n(). Then in p5 I used readUntil(“\n”) to receive the data and once I got that value I cleaned it using trim() and converted it into a number using int(). Then I map it from the sensor range to the width of the canvas so it can control the x position of the ellipse. As I turn the potentiometer, the circle moves left and right on the screen. 

Exercise 2:

P5 Code:

https://editor.p5js.org/MariamAlameri/sketches/oWfNkocIc 

Arduino Code:

https://github.com/mariam9766/Intro-to-IM/blob/736de2e50c777dff22b3545885aebfa6cce33cfd/BidirectionalCommunicationExercise2_mariam.ino 

Process:

The second exercise involved bidirectional communication between p5 and Arduino, where we were asked to control the Arduino from p5. I wanted to create a p5 sketch that allows the user to control the brightness of an LED on the Arduino board. I chose to use a slider in the sketch, as it provides an intuitive and suitable way to adjust brightness. I also referred to the exercises completed in class and adjusted them to meet the purpose of this assignment. In the Arduino code, I used two if statements to ensure that the LED responds when data is received from p5, and to constrain the brightness value within the range of 0–255 for proper PWM control. In the p5 sketch, I implemented the slider using createSlider, adjusted its position and appearance, and used port.write to send the brightness value to the Arduino, allowing accurate control of the LED.

Exercise 3:

Concept:

After completing the first two exercises and creating communication between the p5 sketch and the Arduino code and board back and forth, in this exercise we had to combine both. When the ball in the p5 sketch bounces on the ground, the LED on the board flashes on and off, and when we use the potentiometer, which is the analog sensor on our Arduino board, we control the wind in the sketch and move the ball left and right.

P5 Code:

https://editor.p5js.org/farahshaer/sketches/hAwlRZjXM 

Arduino Code:

https://github.com/mariam9766/Intro-to-IM/blob/9f561f01e2719a784d6c54e8d43e4bbc998a6608/BidirectionalCommunicationExcercise3_mariam.ino 

Demonstration:

Circuit:

Schematic:

Code we’re proud of:

One part of the code that we are proud of is the bounce detection and how it connects to the Arduino:

//the bounce detection
  if (position.y > height - mass / 2) {
    //check if the ball hits the ground
    velocity.y *= -0.9; // A little dampening when hitting the bottom (reverse the direction when it hits)
    position.y = height - mass / 2; //keep it above the floor
    bounce = 1; //to mark that the bounce happened
  }
  //send to arduino
  let sendToArduino = bounce + "\n"; //send the bounce value 1 or 0 and the "\n" tells ardunio the message is done
  port.write(sendToArduino);
  // reset bounceafter it sends
  bounce = 0;
}

We are proud of this part because it turns the ball bouncing into a signal that affects something physical (the LED). It also only sends the signal once per bounce, which makes the interaction feel more intentional instead of constant.

Process:

So in this version of the gravity and wind example, I added serial communication so the sketch can interact with an Arduino on p5. Instead of controlling the wind using the keyboard like in the original example, the wind now comes from an analog sensor (like a potentiometer) connected to the Arduino. The Arduino sends that sensor value to p5, and I map it to a wind force so it pushes the ball left or right. I also added a bounce signal that goes the other way. Every time the ball hits the bottom of the canvas, I set a variable to 1 and send it to the Arduino. This tells the Arduino to briefly turn on an LED, then it gets reset back to 0 so the signal only happens once per bounce. Most of the original physics code stayed the same, but the main changes were adding serial setup, reading sensor data to control wind, and sending a bounce message back to control the LED.

For the Arduino side, I worked on the Arduino code to align with the p5 sketch we have and create serial communication from the Arduino board to the p5 sketch and vice versa. I set it up so the Arduino sends the analog sensor value from A0 to control the wind on the ball on the sketch, and receives a bounce signal from p5 when the ball hits the ground. When this signal is received, the Arduino briefly turns on an LED to indicate the bounce, and then resets the value so the LED only activates once every time it touches the ground. I also made sure the serial communication was properly structured so both inputs and outputs work smoothly together at the same time, allowing interaction between the physical sensor and the digital sketch.

Reflection:

This exercise helped us understand how bidirectional communication actually works, instead of just sending data one way. It was interesting to see how the Arduino and p5 sketch can influence each other at the same time. One challenge we ran into was getting the potentiometer to properly control the wind. Even though the LED response worked, the sensor input was inconsistent, which made it harder to debug whether the issue was in the Arduino code or the p5 code. This made us realize how important timing and serial communication structure are, especially when both sides are sending and receiving data continuously.

If we had more time, we would focus on possibly smoothing the wind movement so it feels less jumpy. We would also experiment with adding more physical outputs, like multiple LEDs or different types of sensors, to make the interaction more dynamic but we just wanted to stick with the assignment instructions for now to get a grasp of the concept.

Concept:

In this assignment, we had to create a musical instrument using digital and analog sensors. We used push buttons as our digital sensors and a potentiometer as our analog sensor. We then decided to create a mini piano-like device that plays the four basic piano notes C, D, E, and F, and allows the user to adjust the pitch of these notes. In this project, there are four push buttons, each assigned to one note that only plays when the button is pressed, and a potentiometer that changes the pitch of the notes when it is turned.

Code:

Arduino File on GitHub

Setup:

MariamMhara

Demonstration:

Mariam

Mhara

Digital Circuit:

Schematic:

Process:

In the process of this assignment, we decided to combine four buttons (digital) with one potentiometer (analog) to control a piezo buzzer. Each button plays a different note, and the potentiometer slightly adjusts the pitch so the sound changes depending on how much it’s turned. We worked together on the idea and the wiring, but we divided the coding so each of us focused on one part. Mariam handled the digital part (the buttons and the notes), and Mhara worked on the analog part (the potentiometer and the pitch control). After both parts were working separately, we combined them into one full sketch of code.

We then tested the circuit in Tinkercad to make sure all the wiring and logic of the code were correct. This helped us confirm that the buttons were reading properly and that the potentiometer was giving smooth values. Running it in Tinkercad also made it easier to fix small mistakes before trying it on the physical Arduino board.

At first, the audio wasn’t changing when the potentiometer was turned because the mapping was happening after the tone was already being played, so we rearranged the order of the code and that finally made the pitch respond. After that, the sound became too noisy and robotic, so we added a small adjustment range (90 – 105) to each note to make the pitch change smoother and less harsh.

Code Snippets:

While building the project, there were a couple of code snippets that stood out to us because they played an important role in making the instrument work the way we wanted it to. 

tone(buzzerPin, noteC * map(sensorValue, 0, 1023, 90, 105) / 100);

This was the part we were most proud of because it solved the “robotic” and “noisy” sound problem. Instead of letting the potentiometer completely change the note, we used a small adjustment (90-150) to bend the pitch smoothly. And this showed how the digital and analog inputs can work together in one line of code. 

Another part of the code is :

pitch = map(sensorValue, 0, 1023, 200, 2000);

This line shows how the analog input (the potentiometer) controls the sound. It takes the raw value from 0-1023 and maps it into a pitch range that the buzzer can actually play. This was important because the potentiometer originally wasn’t affecting the sound at all, and fixing the order of the code made this line finally work the way we wanted it to. 

Areas of Improvement and Reflection:

After completing this assignment, we were able to learn and explore different sensors and sounds. It was easy and smooth to work as a pair, as each person focused on one part and then we combined our work together. As for areas of improvement, we could make the sound of the notes smoother and more musical, since it still sounds slightly robotic, or add more notes to make it closer to a real piano. Another idea is to implement other sensors, such as an ultrasonic sensor, to play different notes or even melodies based on motion or distance. Working with audio and sensors is a fun part of Arduino, and it allows us to create many different ideas for various purposes. Overall, we are satisfied with our final outcome and the entire process of this project.

References:

Looked back at Week 10 slides about sound to recap what we learned.

Reviewed specific code concepts using the Arduino documentation:

https://docs.arduino.cc/language-reference/en/functions/math/map/ 

• How we used it: We used this to convert the potentiometer’s range into a smaller pitch-adjustment range that works smoothly with the buzzer.

https://docs.arduino.cc/language-reference/en/functions/advanced-io/tone/ 

• How we used it: We used this page to understand how the tone() function works and how to send different frequencies to the buzzer. 

https://docs.arduino.cc/built-in-examples/digital/toneMelody/ 

• How we used it: We looked at this example to understand how notes from pitches.h are used and how tone() can be combined with different frequencies to create musical sounds.

Used ChatGPT to help navigate and resolve a major issue where the tones sounded too robotic and noisy. From this, we learned that using the map() function with a smaller range for each note helps create smoother, more controlled pitch changes.

This week’s first reading, A Brief Rant On The Future Of Interaction Design, made me realize something deeper about the things I do, and what everyone does every day. Starting by watching the video, I initially thought it was just about technology, and I found it interesting, especially with all the futuristic functions they included such as the window screen and others. What I didn’t notice or really think about was the fact that everything was done on screens, because that is our reality and something we do every day for almost everything. After reading the text, I started to realize how true this is, and how most of what I do happens through flat screens using my fingers, even for things that could be done physically, such as the reading example. I did agree with that point, that some things do not have to be done on a screen. However, I also thought about how people do not actually do everything through screens, as we still move, go outside, and use our bodies for different tasks. But since the author is talking about the future, I do agree that this should be considered so it does not reach a point where everything is done through a flat screen.

This week’s second reading, the responses to the brief rant, clarified many of the thoughts I had while reading, and also introduced aspects I had not considered, which made me go from partially agreeing to agreeing more with the author’s concern. I did think about the common idea that devices can be harmful, especially for children, but the responses helped me understand that while current technology is useful, it could become problematic if it becomes too dominant in the future. I also found the hologram example very helpful, as it made the main idea clearer, that technology should continue to develop, but in a way that is more interactive and three-dimensional, matching the environment we live in. Additionally, the quote by neuroscientist Matti Bergström about the effects of constant touchscreen use from a young age made me realize that this could become harmful if it reaches the future vision being discussed.

Both readings were interesting to read, especially because they address real situations and possible future developments. They made me think more about how we use screens in everyday life and how many things have already shifted to digital formats, such as borrowing books or using services that were once physical. I also thought about how it would be interesting and beneficial if future technologies included more physical interaction and engagement with the human body. This connects to the work we do in this class, where our Arduino projects involve physical interaction, while our p5 sketches are mostly screen-based using buttons and touchpads. Since our final project will combine both, I feel like that is a strong example of how technology can be improved by balancing physical interaction with screen-based systems.

Concept:

For this Arduino assignment, I wanted to combine an analog sensor that works according to the surrounding environment in a smooth, fading way, and a digital sensor that works with the user’s input in an immediate way. When thinking about this assignment, the idea of airplane cabin lights came to mind, where there are dim blue lights lit throughout the plane when it’s dark, along with separate small seat lights that we can turn on and off.

I created an analog system using a photoresistor, where a blue LED lights up in the dark when the surrounding lights are off, in a slow and calm fading manner. I also integrated a digital system using a pushbutton, where a yellow LED turns on and off when it is pressed. The blue light then slowly fades off again when the surrounding environment becomes bright.

Link to Code:

Arduino File on GitHub

Demonstration:

Setup:

Here is an image of my Arduino setup. I have my 5-volt wire connected to the + bus and my ground connected to the – bus. I also have the photoresistor and pushbutton connected to 5 volts, with their 10kΩ resistors connected to ground. Each LED is placed under its sensor, with one leg connected to ground and the other leg connected to a 330Ω resistor, which then connects to the assigned Arduino pin.

Digital Circuit:

Drawn Schematic:

Code:

Since I have two types of sensors, I organized my code so that under each section (components, setup, loop), the analog part is labeled first, followed by the digital part, also labeled. All explanations are included directly in the code itself.

A particular part of the code I am proud of is the use of the if-else statement for the analog sensor, which allows the blue LED to turn on and off really smoothly and slowly. I wanted to do this to give it a more realistic behavior based on my concept goal. At first, the LED would turn on and off immediately and rapidly, but using this if-else statement along with the brightness calculation allowed it to work the way I wanted by making it gradually increase and decrease. I acknowledge that this function works differently when turning the light off compared to turning it on, and that is because the light sensor values change at different rates when the environment becomes dark or bright, causing the LED to reach its target brightness at different speeds.

if (brightness < targetBrightness) { // gradually adjust brightness according to state
  brightness++;  // increase brightness slowly
} else if (brightness > targetBrightness) {
  brightness--;  // decrease brightness slowly
}

I also used an if statement for the digital sensor to make the yellow LED turn on and off with the push of the button. When I followed what we had done in class and what I already knew, the light would only stay on while holding the button. Again, I wanted to achieve a more realistic behavior for my concept. Using the if statement with the && function allowed me to achieve this toggle effect.

if (buttonState == HIGH && lastButtonState == LOW) { // detect button press and allow transitioning
    ledState = !ledState;  // toggle yellow LED
  }

Reflection:

I found this assignment really interesting, as I felt like I was learning a lot while using two different devices at once. I was initially worried about creating one code that works with two different sensors at the same time, but after planning my process, I was able to manage it. I started off by focusing only on the analog sensor since it is more complex, and then integrated the digital sensor code into each part. I also feel like the organization of my code played a big role in this success.

On the Arduino itself, the setup first looked complicated to me as I was planning, but once I really looked at each part and understood what was happening, it all made sense. However, looking at it now, I feel like I could have organized it in a better way to make the LEDs clearer, since there are so many wires connected all over. That is, to me, an area for improvement and something I will definitely focus on next time, and even try to improve on this same project.

I also feel like this analog and digital sensor assignment could have so many more different approaches, and I would like to think about and explore more creative ideas in the future.

References:

In this assignment, I mostly referred back to our previous lecture recordings and slides, particularly 8.2 and 9.2, to recap what we had learned, and then adjusted and integrated my own ideas to match the concept.

For the pushbutton functionality, I needed to use the bool function to toggle the yellow LED using an if statement, so I referred to the Arduino official documentation for a better understanding:

https://docs.arduino.cc/language-reference/en/variables/data-types/bool/ 

I also asked ChatGPT about one particular thing, when I wanted to make the brightness of the blue LED fade in and out much slower than it did by default. I knew I had to use an if-else statement, so I explained my goal, and it suggested using brightness++ and brightness– to adjust the speed of the fading effect by gradually increasing and decreasing the brightness until it reaches the target.

This week’s first reading, Physical Computing’s Greatest Hits (and Misses), made me think a lot about different physical computing themes, some that are around me, and the things that I could integrate and do with them. A sentence that stood out to me was at the start, “Sometimes when people learning about physical computing hear that a particular idea has been done before, they give up on it, because they think it’s not original,” as I related to it both during our p5 sketches work and now with Arduino, where I pressure myself to try to create something completely different. I was really interested as I read more, and some of the things that came to my mind that I could relate to were things I have already encountered, such as the floor pad games that exist in many arcades today in different styles, and the “Fields of Grass” theme that is used in places like TeamLab and museums, where you run your hand over sensors and experience an interaction. Interestingly, the dolls and pets theme reminded me of Build-A-Bear teddies I made when I was younger, where I would press a part of the bear and hear recorded audio. While reading, I found myself thinking about my Arduino and all the different sensors I have, and I think all the ideas mentioned are great inspiration and encouraged me to look deeper and find new ways to integrate them into my projects.

This week’s second reading, Making Interactive Art: Set the Stage, Then Shut Up and Listen, brought my attention to an aspect of interactive artworks that I had not considered before, and that feels different from what people usually say. I feel like it made a lot of sense to me that users who encounter something interactive should feel comfortable giving inputs based on their instincts. What I took away from the reading is that it is important to guide the user and give them context about what they are experiencing, but not to completely tell them exactly what to do, in order to maintain the purpose of “interactivity” in the artwork. The concluding example of a director working with actors really clarified this idea and made it more convincing.

All together, I feel like both readings are well aligned with each other and added to my knowledge and understanding of interactive artworks and programming. The first reading felt more like a collection of examples and structures of how a project could be, while the second reading added to it by focusing on the user experience and interaction within that same project. The knowledge I gained connects clearly to the work I have done and will continue to do in this course, and I feel like both readings gave me ideas, inspiration, and awareness of what I should consider moving forward.