Week 11: Reading Response

A Brief Rant on the Future of Interactive Design + Follow-up

The first article is essentially arguing that the dominant vision of future technology, which is everything being a flat glassy touchscreen you slide your finger across, is not actually visionary at all. It is just a timid extension of what already exists, and what already exists ignores almost everything that makes human hands remarkable. His point is that our hands do two things extraordinarily well, they feel things and they manipulate things, and touchscreens strip both of those capabilities away in exchange for a visual interface that he calls Pictures Under Glass. I reflected on his example of making a sandwich. He asks you to pay attention to how many tiny adjustments your fingers make without you even thinking about it, switching grips, sensing weight, feeling texture, and then he asks whether we are really going to accept a future interface that is less expressive than that. That question reminded of the time I tried learning drums through one of those tablet apps, and the difference between that and sitting in front of a real kit is almost laughable. On a real drum the stick bounces back after you hit it, and that rebound produces important information. Your wrist reads it and adjusts the next stroke automatically, and I could feel even as a beginner that my hands were supposed to be learning something from that response. On the app there is nothing. You tap a flat surface, it makes a sound, and that is the entire relationship. I was learning the pattern but I was not learning to actually play, and from what I can understand, that distinction is what the author is getting at.

About his response to pushback, I actually found it more interesting than the original rant. In the part when someone asked about voice interfaces and he said he has a hard time imagining a painter telling his canvas what to do. That, again, reminded  of the drums. There is no way to describe with a voice or replicate on a screen the feeling of a snare cracking back against your stick, or the way a cymbal responds differently depending on where and how hard you hit it. That knowledge is supposed to live in your hands built up over time, and I genuinely felt the absence of it every time I went back to the app and realized my fingers were learning nothing they could transfer to a real instrument. It felt like I was practicing the appearance of playing drums without any of the physical intelligence that actually makes someone a drummer.

Reading Reflection Week 11: The Touchy Touchscreen

Bret Victor makes a really good point about the so called “Pictures under glass” concept of technology. I mean, as a bit of a fun fact, in my elementary school, our books were so in need of an update that the latest technology the textbooks about computer science, mention touchscreens as the latest technology. It’s quite laughable thinking about it. But I do have to agree, in a generation and era where we’re living a touchscreen oriented life, the magic does get lost along the way.

But thing is I think right now we have at least one example of trying to make technology adapt to fit our human body, being the reMarkable paper tablet. Initially when I was looking to buy some new technology I had this one goal in mind; that it can have a touchscreen for me to draw and write on. I primarly was looking at laptops that had touchscreens when I was introduced to the reMarkable. Now I thought, I mean its nothing special when it came to what it is and what it lacked as opposed to a conventional laptop. But after testing it out, it was astounding. It felt so natural to write on it, like I was writing in a notebook. I didn’t get that artifical feel that iPads give me and genuienly is what sold me on buying the reMarkable in the end.

At some point, we do have to ask ourselves, at what point is our life simulated by screen or whether our reality is literally a headset we can’t take off? (obviously I’m joking but it is getting quite dystopian no?)

Week 11 – Arduino Music Megan

Arduino – Harry Potter Music Box

Concept

For this project, I decided to create a mini music pad that can play the Harry Potter theme. The idea came from a small Harry Potter music box that I was gifted when I was younger, since I have always loved Harry Potter and I am a big fan. I also play the piano, so I wanted to design something that felt similar to an instrument and could simulate specific notes like a simplified keyboard.

Process

For the hardware, I used four colored buttons, a potentiometer, and a buzzer to produce sound. I connected each button to a digital pin, the potentiometer to 5V, GND, and analog pin A0, and the buzzer to another digital pin. All components were connected to ground appropriately.

For the coding part, it was a bit challenging at first to find the right notes. I experimented with a basic Arduino buzzer note library and helped myself with Chat GPT to adjust different frequencies until they sounded closer to the melody I wanted. Since the Harry Potter theme has more than four notes, I used the potentiometer as an analog input to change the pitch of the notes. This allowed me to expand the range of sounds using only four buttons. Although it is not perfectly accurate, the result is still very recognizable and captures the essence of the original theme.

// ==============================
// MINI MUSIC PAD - HARRY POTTER SONG
// ==============================


// BUTTON PINS 
int b1 = 2; // E
int b2 = 3; // G
int b3 = 4; // B
int b4 = 5; // C 

// BUZZER 
int buzzer = 8;

// POTENTIOMETER 
int pot = A0;


// NOTES 
#define E4 329
#define G4 392
#define B4 494
#define C5 510


// ==============================
// SETUP
// ==============================
void setup() {

  pinMode(b1, INPUT_PULLUP);
  pinMode(b2, INPUT_PULLUP);
  pinMode(b3, INPUT_PULLUP);
  pinMode(b4, INPUT_PULLUP);

  pinMode(buzzer, OUTPUT);
}


// ==============================
// LOOP
// ==============================
void loop() {

  int potValue = analogRead(pot);

  // HIGH / LOW pitch mode
  bool highPitch = potValue > 512;

  float multiplier;

  if (highPitch) {
    multiplier = 1.3; // slightly higher pitch
  } else {
    multiplier = 0.85; // slightly lower pitch
  }

  if (digitalRead(b1) == LOW) {
    play(E4, multiplier);
  }

  else if (digitalRead(b2) == LOW) {
    play(G4, multiplier);
  }

  else if (digitalRead(b3) == LOW) {
    play(B4, multiplier);
  }

  else if (digitalRead(b4) == LOW) {
    play(C5, multiplier);
  }

  else {
    noTone(buzzer);
  }
}


// ==============================
// FUNCTION
// ==============================
void play(int note, float mult) {
  tone(buzzer, note * mult);
}

Thing That I’m Proud Of

One part of the project I am especially proud of is how I used the potentiometer to control the pitch. This idea allowed me to overcome the limitation of having only four buttons. Initially, I tried pressing multiple buttons at once to create different sounds, but it did not work as expected. Then I realized I could use the potentiometer to dynamically change the pitch, which was a much more effective solution. I am also proud of the overall design of my mini music pad, as it is intuitive to use and much cleaner than my earlier versions, where the wires were more disorganized.

// HIGH / LOW pitch mode
  bool highPitch = potValue > 512;

  float multiplier;

  if (highPitch) {
    multiplier = 1.3; // slightly higher pitch
  } else {
    multiplier = 0.85; // slightly lower pitch
  }

Overall Reflection

Overall, I really enjoyed this project. It allowed me to combine something personal, like my love for Harry Potter and music, with technical skills such as coding and circuit design. The process involved a lot of experimentation and problem-solving, especially when trying to match the melody. This project also made me feel nostalgic, since the song reminds me of my childhood. I am very happy with the final result and how I was able to turn a simple Arduino setup into an interactive musical experience.

SCHEMATIC

VIDEO

Reading Reflections

Reading A Brief Rant on the Future of Interaction Design and the Response honestly changed the way I see technology in a way I did not expect at all. I agree with the author, but what surprised me the most is that I had never even realized this was a problem in the first place. It is not that our devices are bad, but more that they are limiting us from using our full potential, especially when it comes to how we physically interact with the world.

One thing that really stuck with me was when I started thinking about the idea of using devices with your eyes closed. I asked myself what I could actually do like that, and the only thing I could come up with was typing on my laptop keyboard. And then I realized that typing is one of the only truly tactile interactions we still have, and it is not even part of the screen itself. Everything else, especially touchscreens, feels like just sliding your finger on glass with almost no physical feedback. The reading describes this as “Pictures Under Glass,” and I think that is such a perfect way to put it, because it really does feel disconnected from what you are doing.

It also made me think about why we keep simplifying technology more and more. At first it seems like progress, because everything becomes easier and more accessible, but at the same time, it feels like we are reducing the way we interact with the world to something very basic, almost like we are being treated as toddlers. And that idea really stayed with me, because I had never questioned it before. I never thought that maybe we actually deserve more complex and richer ways of interacting, instead of everything being reduced to tapping and swiping.

I also kept thinking about the comparison between the real world and digital interfaces. In real life, our hands are constantly feeling, adjusting, reacting. The reading gives examples like holding a book or a glass and understanding things like weight and position without even thinking about it. That made me realize how much information we are losing when everything becomes flat and two dimensional. It made me wonder if we are actually moving forward or backward by making everything more and more screen based.

At the same time, I thought about things like movie theaters. It took so long for us to even be able to record and display images, but now we have 3D, 4D, and experiences where you can feel water or heat from what is happening on screen. That feels like a step toward engaging more of our senses. But then I started thinking, what if we did not need glasses to see in 3D, or what if devices could actually communicate through touch in a more real way? Something closer to real life, like materials that change shape or give feedback. That is kind of what the author is pushing for with the idea of a “dynamic medium that we can see, feel, and manipulate.”

At some point, though, I also started questioning how far this can really go. If we keep trying to make technology more and more like real life, are we just trying to recreate reality itself? Because the real world is already the most advanced “interface” we have. So it made me think about where the limit is, and whether the goal should be to replicate reality or to create something entirely new.

Overall, this reading really opened my mind. It made me realize that interaction design is not just about making things look nice or easy to use, but about deeply understanding human capabilities and not ignoring them. I had never questioned touchscreens before, but now I cannot stop thinking about how much more is possible and how much we might be missing out on by staying with what we have.

Assignment 11: Dial up the Tune

“It’s a pity to shoot the pianist when the piano is out of tune.” – Rene Coty

Concept:

Soo, when it comes to instruments, I thought right, let’s make a piano. Now due to the limitations from the Arduino Breadboard, I couldn’t make it so all of it the frequencies are in tune, whether they’re lower or higher. So I needed to think of a way how to make it so all of the frequencies could be represented on the project. That’s where our lovely, potentiometer comes into play.

Check the Tinkercad out here!

Sketch:

Digital Circuit:

How it’s made:

Now for the design, I went with having the buttons in a line with each of them being connected to the digital input of the Arduino Uno. After adding to each a resistor, alongside a connection to the power, they were ready to go. Then we have a Piezo that is connected to the side and acts as our output for sound, or specifically frequencies.

But the special thing is our so called dial, represented by the Potentiometer. He is connected as an analog input and will serve the user as while it is rotated, the frequency will be increased or decreased. I will be honest, I actually had an issue intially with the Piezo constantly playing sound, but I figuered it out later after seeing that the resistors weren’t connected properly (always good to check this in case of any errors)

Highlighted bit of Code I’m proud of:

Now after that it was time to put it all together. The code itself is quite simple as we have read functions to take in our defined input. But what was interesting is trying to get it so each button having a different pitch. It was done by having the potentiometer’s value added on to a constant value which increases with each button. This gave way for the user to increase the pitch and then, always, it will be added to a specific constant which will give the sesne of a piano.

//potent variable is dependent on where the user slides the potentiometer
if (s1 == 1){
tone(buzzer, potent + 100, 100);
} 
else if (s2 == 1){
tone(buzzer, potent + 200, 100);
}
else if (s3 == 1){
tone(buzzer, potent + 300, 100);
} 
else if (s4 == 1){
tone(buzzer, potent + 400, 100);
} 
else if (s5 == 1){
tone(buzzer, potent + 500, 100);
} 
else if (s6 == 1){
tone(buzzer, potent + 600, 100);
} 
else if (s7 == 1){
tone(buzzer, potent + 700, 100); 
}

Reflection

I’m happy with this design and even using the Piezo for the first time was fun. I will be honest my ears didn’t like it as much but we got through it haha. I think an extension is to make it so you could have some sort of keyboard input or even make original songs with it. I saw some interesting ideas of how by getting specific frequencies in a loop, you could make a song which I’d love to experiement with this for my final project.

Full Code:

//Declaring every button to it's assgined digital value
int tone1 = 12;
int tone2 = 11;
int tone3 = 10;
int tone4 = 9;
int tone5 = 8;
int tone6 = 7;
int tone7 = 6;

//And here declaring the Piezo at Digital 13
int buzzer = 13;

void setup ()
{
  
//Starting up the Serial Connection on Arduino
  Serial.begin(9600);
  
//Making all of the buttons take in our Input 
  pinMode(tone1,INPUT);
  pinMode(tone2,INPUT);
  pinMode(tone3,INPUT);
  pinMode(tone4,INPUT);
  pinMode(tone5,INPUT);
  pinMode(tone6,INPUT);
  pinMode(tone7,INPUT);
  
//Making the buzzer as our output  
  pinMode(buzzer,OUTPUT);
}

void loop () 
{
  
//Defining a variable which is the value of the potentiometer
  int potent = analogRead(0);
  
//Outputing the value in the console
  Serial.println(potent); 
  
//Reading a value for each pin
  int s1 = digitalRead(tone1);
  int s2 = digitalRead(tone2);
  int s3 = digitalRead(tone3);
  int s4 = digitalRead(tone4);
  int s5 = digitalRead(tone5);
  int s6 = digitalRead(tone6);
  int s7 = digitalRead(tone7);

//If else conditions where, when the button is on, it will output a sound
//potent variable is dependent on where the user slides the potentiometer
  if (s1 == 1){
  tone(buzzer, potent + 100, 100);
  } 
  	else if (s2 == 1){
  tone(buzzer, potent + 200, 100);
  }
  	else if (s3 == 1){
  tone(buzzer, potent + 300, 100);
  } 
  	else if (s4 == 1){
  tone(buzzer, potent + 400, 100);
  } 
  	else if (s5 == 1){
  tone(buzzer, potent + 500, 100);
  } 
  	else if (s6 == 1){
  tone(buzzer, potent + 600, 100);
  } 
  	else if (s7 == 1){
  tone(buzzer, potent + 700, 100); 
  } 

//Just to give a short delay between presses
  delay(10);
  
}

 

 

Week 11 – Production Assignment

Your concept

I was a bit lost on what to do for this week’s assignment. With loads of deadlines and presentation coming up for my other classes, it was difficult to allocate a lot of time for this production assignment which is something I was very upset about. I have been seeing so many Arduino videos online and getting loads of ideas for possible productions and I hope to bring these ideas to life in my next submission. However, for today’s assignment, I referred to the SparkFun tutorials.

The idea of my circuit is that the user can play different tunes by pressing on the buttons. The sound is generated by the buzzer. The user can also adjust the volume using the potentiometer.

Schematic & Circuit

 

IMG_8039

“How this was made” section explaining how the code was made and sources of media assets

As mentioned, I followed the tutorials from the SparkFun book provided with the kit. Beyond just doing the circuit and running the code, I did read through the information provided to understand how and why everything works. The code is pretty self-explanatory and mimics things we covered in class.

Reflection and ideas for future work or improvements

For the future, I definitely should start EARLIER. Honestly, it was just a struggle with time management for this submission. Otherwise, I think I could’ve come up with something more interesting. I’m hoping to work on something else for this assignment over the weekend and hopefully submit that in addition to this!

Week 11 – Production Assignment (Air Piano)

Concept

The “Air Piano” is a musical instrument controlled using the distance from a sensor. I have always been fascinated by the distance sensors, using reflection of sound waves to calculate the distance of an object. I used the the HC-S04 sensor to detect the distance in this work as input for the instrument. I was inspired by the tone knob in electrical guitars which was discussed in class so I integrated it into my work. A potentiometer serves as my tone knob and its input is used to switch among 3 sound modes: Piano mode, Sci-Fi mode and Bass Mode. There are also different colored LEDS to show the user which mode is on. Red – Piano, Green – SciFi and Blue – Bass mode. The mode determines the scale of frequency produced by the buzzer. The coding behind this project was to map the distance of an object from the sensor to a value from 0 to 7 (8 values) and these values each correspond to a specific note.

Sketch

Image

Code

// A0 - Potentiometer
// 2 - Blue
// 4 - Green
// 7 - Red
// 12 - Buzzer 

const int trigPin = 10;
const int echoPin = 9;

// Mode 1 - Piano Scale
int mode1[8] = {262, 294, 330, 349, 392, 440, 494, 523};

// Mode 2 - Sci-Fi
int mode2[8] = {600, 750, 900, 1100, 1400, 1800, 2300, 3000};

// Mode 3 - Higher Bass / Bright Low Mode
int mode3[8] = {350, 420, 500, 600, 720, 850, 1000, 1200};

float duration;
int distance;

void setup() {
  pinMode(trigPin, OUTPUT);
  pinMode(echoPin, INPUT);
  pinMode(2, OUTPUT);
  pinMode(4, OUTPUT);
  pinMode(7, OUTPUT);
  pinMode(12, OUTPUT);

  Serial.begin(9600);
  digitalWrite(2, HIGH);
  digitalWrite(4, HIGH);
  digitalWrite(7, HIGH);
  delay(1000);

  digitalWrite(2, LOW);
  digitalWrite(4, LOW);
  digitalWrite(7, LOW);
}

void loop() {
  digitalWrite(trigPin, LOW);
  delayMicroseconds(2);
  digitalWrite(trigPin, HIGH);
  delayMicroseconds(10);
  digitalWrite(trigPin, LOW);
  int mode;

  int pm = analogRead(A0);

  duration = pulseIn(echoPin, HIGH);
  distance = (duration*0.0343)/2;

  Serial.print("Distance: ");
  Serial.println(distance);
  delay(100);

  if (pm >= 700) {
    mode = 0;
    digitalWrite(7, HIGH);
    digitalWrite(2, LOW);
    digitalWrite(4, LOW);
  } else if (pm >= 350) {
    mode = 1;
    digitalWrite(4, HIGH);
    digitalWrite(7, LOW);
    digitalWrite(2, LOW);
  } else {
    mode = 2;
    digitalWrite(2, HIGH);
    digitalWrite(4, LOW);
    digitalWrite(7, LOW);
  }

  if (distance >= 0 && distance <= 40) {

    int noteIndex = map(distance, 0, 40, 0, 7);
    int freq;

    if (mode == 0) freq = mode1[noteIndex];
    if (mode == 1) freq = mode2[noteIndex];
    if (mode == 2) freq = mode3[noteIndex];

    tone(12, freq);
  }
  else {
    noTone(12);
  }

  //delay(50);
}

 

 

How it was made

This work was made by first drawing the sketch of the circuit. The inputs are the resistance value of the potentiometer and the distance from the distance sensor. I had to watch a video and use the ardruino website to figure out how to configure the distance sensor. The output of the circuit are 3 LEDs and a buzzer. The LEDs where connected to digital pins, a resistor and ground. The buzzer was connected to a digital pin and ground. In the coding of the instrument, I used ChatGPT to generate an array of frequencies corresponding to the modes. I connected all the wires and components according to the sketch.

Reflection

Being someone with no knowledge in music, I thought this work will be very difficult for me but after I figured out the input and output of the program, it went quite smoothly. To improve the project, a more sensitive distance sensor can be used to improve the readings of the sensor and more modes with different frequencies can be added to create more musical effects.

Week 11 Assignment – Zere

Concept

The concept of my project is to create a light-controlled musical instrument using Arduino. Instead of buttons or keys, the player controls pitch by covering or exposing a photoresistor to light, like waving your hand over it. A pushbutton switches between a low octave and a high octave, making the instrument feel expressive and playful. I wanted the interaction to feel unusual and fun, more like a theremin than a traditional instrument.

const int LDR_PIN = A0;
const int BTN_PIN = 2;
const int BUZZ_PIN = 8;

void setup() {
  pinMode(BTN_PIN, INPUT_PULLUP);
}

void loop() {
  int lightVal = analogRead(LDR_PIN);
  int btnState = digitalRead(BTN_PIN);

  int freq;
  if (btnState == LOW) {
    freq = map(lightVal, 0, 1023, 500, 2000);
  } else {
    freq = map(lightVal, 0, 1023, 100, 500);
  }

  tone(BUZZ_PIN, freq);
  delay(10);
}

I started by placing a photoresistor on the breadboard with a 10kΩ resistor forming a voltage divider, which lets the Arduino read changing light levels through A0. At first the buzzer played the same tone constantly, later I discovered through the serial monitor that A0 was reading 1023 the whole time, which meant that the resistor was not properly connected to the photoresistor. Once I fixed that, the pitch started responding to light.

What I’m proud of

The part I’m most proud of is figuring out the button wiring. This was incredibly frustrating, as I spent a long time confused about which legs of the button connect internally, which pins on the Arduino were digital versus analog, and why the signal kept reading the wrong values. I watched a YouTube tutorial to better understand how the process, it was quite challenging to be honest. https://www.youtube.com/watch?v=gj-H_agfd6U

What I can do better next time

Next time, I want to map the light values to specific musical notes rather than continuous frequencies, so it sounds more like a real scale than a smooth slide between pitches. I could also add an LED that lights up when the button is pressed for visual feedback, or use multiple photoresistors to create distinct pitch-control zones. I could also arrange the wires more neatly next time.

This is the video of me testing the sound: IMG_2491

 

Week 11 Assignment

Concept

The concept of my project is to create a simple electronic musical instrument using Arduino. I wanted to turn basic components like a button and a potentiometer into something interactive and expressive. The button works like a “play key,” while the potentiometer controls the pitch of the sound. By combining these elements, the user can actively “play” the instrument instead of just hearing a fixed sound. I also added a second button to switch between low and high pitch modes, which makes the instrument feel more dynamic and closer to a real musical device.

How I Made This

int playButtonPin = 2;
int modeButtonPin = 3;
int buzzerPin = 8;
int ledPin = 6;
int potPin = A0;

int lowNotes[]  = {262, 294, 330, 349, 392, 440, 494, 523};   // C4 到 C5
int highNotes[] = {523, 587, 659, 698, 784, 880, 988, 1047};  // C5 到 C6

bool highMode = false;
int lastModeButtonState = LOW;

void setup() {
  pinMode(playButtonPin, INPUT);
  pinMode(modeButtonPin, INPUT);
  pinMode(buzzerPin, OUTPUT);
  pinMode(ledPin, OUTPUT);
  Serial.begin(9600);
}

void loop() {
  int playButtonState = digitalRead(playButtonPin);
  int modeButtonState = digitalRead(modeButtonPin);
  int potValue = analogRead(potPin);

  int noteIndex = map(potValue, 0, 1023, 0, 7);

  // detect second button:change mode
  if (modeButtonState == HIGH && lastModeButtonState == LOW) {
    highMode = !highMode;
    delay(200); // 
  }
  lastModeButtonState = modeButtonState;

  int frequency;
  if (highMode) {
    frequency = highNotes[noteIndex];
  } else {
    frequency = lowNotes[noteIndex];
  }

  if (playButtonState == HIGH) {
    tone(buzzerPin, frequency);
    digitalWrite(ledPin, HIGH);
  } else {
    noTone(buzzerPin);
    digitalWrite(ledPin, LOW);
  }

  Serial.print("Mode: ");
  if (highMode) {
    Serial.print("HIGH");
  } else {
    Serial.print("LOW");
  }

  Serial.print("  Note index: ");
  Serial.print(noteIndex);
  Serial.print("  Frequency: ");
  Serial.println(frequency);

  delay(10);
}

 

I started by building a very basic circuit with a buzzer, a button, and a potentiometer. At first, the buzzer only made continuous sounds, which didnot feel like music. Then I modified the code so that the potentiometer controls discrete notes instead of continuous frequencies. This made the sound more like a real scale (Do, Re, Mi).

After that, I added a second button to switch between two sets of notes (low and high pitch). This required both wiring changes and updating the code logic to detect button presses and toggle modes. I also added an LED that lights up when the instrument is being played, which gives visual feedback and makes the interaction clearer.

Throughout the process, I tested each part step by step instead of building everything at once. This helped me identify problems more easily and understand how each component works.

What I’m Proud Of

The part I’m most proud of is how I figured out the mistakes with the buttons. At the beginning, my buttons did not work correctly at all. Sometimes the sound would play randomly, and sometimes pressing the button did nothing. I realized later that I misunderstood how the button pins (1A, 2A, etc.) are connected internally, and I also forgot to use the resistor properly.

After debugging, I learned that the button needs to cross the middle gap on the breadboard and that I must connect a pull-down resistor to stabilize the signal. Fixing this problem made everything work correctly, and it was a moment where I felt I really understood the circuit instead of just following instructions. This experience helped me become more confident in troubleshooting hardware problems.

What I Can Do Better Next Time

Next time, I think I can improve both the design and the interaction of my project. Right now, the instrument is still quite simple, and the sound is limited to basic tones. I could improve this by adding more buttons to create a small keyboard or by programming simple melodies.

I also want to make the interaction more intuitive. For example, I could use a light sensor to control pitch instead of a potentiometer, which would make the instrument feel more creative and less mechanical. Another improvement would be organizing the wiring more clearly, because my current circuit is a bit messy and hard to read.

Overall, this project helped me understand both coding and circuit design better, but I think there is still a lot of space to make it more expressive and closer to a real musical instrument.

Reading reflection

After reading both the essay and the responses, one clear idea is the difference between what interaction design could be vs what it is now. Bret Victor wants computers to help people think and understand, not just click buttons. This made me reflect that many of my own projects are still very basic—more like reactions, not real thinking tools.

Another important thought is about making things visible. He argues that systems should show how they work instead of hiding everything. This connects to learning. When I can see changes directly, I understand faster. So interaction design is not only about design, but also about how people learn.

However, the responses made me question his ideas. Some people say his vision is too idealistic and hard to apply in real life. Real systems have limits, and not all users want to explore deeply. Sometimes people just want things to be fast and simple. So his ideas may work better for learning tools, not everyday apps.

It also made me think about the role of the designer. Instead of controlling everything, the designer creates a system where users explore by themselves. This is similar to interactive art, but it also means less control over the final experience.

Overall, these readings made me see interaction design as more than coding. It is about how people think, understand, and interact with systems, but also about balancing ideal ideas with real-world limits.

Week 11 – Serial Communication

Repository

Repository

Exercise 1

1. Overview

In this exercise, we explored serial communication between Arduino and p5.js. The main objective was to use a single sensor on Arduino and translate its input into visual movement in p5 — specifically controlling an ellipse moving horizontally across the screen. A potentiometer connected to pin A1 served as the analog input, and no data was sent back from p5 to Arduino, making this a one-directional communication exercise.

2. Concept

The core idea was to establish the simplest possible link between a physical input and a digital visual. By turning a potentiometer, the user directly moves an ellipse across a canvas in the browser in real time. This introduced us to the fundamental pipeline of physical interaction: sensor reads a value, Arduino maps and sends it, p5 receives and translates it into something visible on screen. The simplicity of the setup made it easy to trace the data flow end-to-end and understand what was happening at each stage.

3. Process and Methods
    • We began by following the example demonstrated in class and gradually adapted it to better understand the data flow. On the Arduino side, the potentiometer on pin A1 returns a raw analog value between 0 and 1023. We mapped this to a smaller range of 0–255 using map() before sending it through Serial.println(). This made the value easier to work with on the p5 side without losing meaningful range.
    • For p5.js, we used the p5.webserial library following the structure introduced in class. The serial connection is opened manually through a button, which triggers the browser’s port picker dialog. Inside draw(), port.readUntil(“\n”) reads each incoming line, trim() strips the newline character, and int() converts the string into a usable number. That number is then mapped to the canvas width using map(), which drives the ellipse’s horizontal position. The ellipse stays fixed on the vertical centre of the canvas at all times.
4. Technical Details
    • The Arduino maps the raw potentiometer value before sending it to reduce the range to something the p5 side can easily stretch across the canvas:
// ── READ SENSOR ──
// analogRead returns 0–1023 based on the voltage at A1.
// At 0V (GND side) → 0. At 5V → 1023.
int potentiometer = analogRead(A1);

// ── MAP TO SMALLER RANGE ──
// Compress 0–1023 to 0–255 so p5 can easily map it onto the canvas width.
int mappedPotValue = map(potentiometer, 0, 1023, 0, 255);

// ── SEND TO p5 ──
// Serial.println() sends the number as a string followed by a newline character '\n'.
// p5 uses that newline to know where one value ends and the next begins.
Serial.println(mappedPotValue);
    • On the p5 side, the incoming string is cleaned and converted before being mapped to a screen position:
let str = port.readUntil("\n");

if (str.length > 0) {
  // trim() removes the trailing '\n' (and any spaces).
  // int() converts the cleaned string into a number.
  let val = int(trim(str));

  // ── MAP VALUE TO CANVAS ──
  // Arduino sends 0–255. We stretch that range across
  // the full canvas width so the ellipse covers the
  // entire screen as the pot is turned.
  x = map(val, 0, 255, 0, width);
}
    • This two-step mapping — first on Arduino from 0-1023 to 0-255, then in p5 from 0-255 to 0-canvas width — demonstrates how data can be progressively transformed as it passes between systems.
5. Reflection

This exercise gave us a concrete understanding of how physical input can directly influence digital visuals in real time. The most valuable part was seeing the full pipeline in action: turning the potentiometer caused an immediate, visible response on screen, which made the abstract idea of serial communication feel tangible. It also highlighted the importance of consistent data formatting — the newline character that Serial.println() appends is what makes port.readUntil(“\n”) work reliably on the p5 side. If we were to continue developing this, we would explore using different sensors such as FSR or ultrasonic distance sensors, add smoothing to reduce noise, and expand the visuals to control more than one element.

Exercise 2

1. Overview

In this exercise, we reversed the direction of communication — from p5.js to Arduino. The objective was to control the brightness of a physical LED using mouse movement in the browser. p5 continuously sends a brightness value based on the mouse’s horizontal position, and Arduino uses that value to dim or brighten the LED through PWM. No data is sent back from Arduino to p5.

2. Concept

Where Exercise 1 used hardware to control software, this exercise flipped that relationship. The browser became the controller and the LED became the output. Moving the mouse across the screen is a familiar, intuitive gesture, and seeing that gesture reflected immediately in a physical light made the connection between the two systems feel direct and satisfying. This exercise also introduced us to the handshake pattern, which is necessary when Arduino needs to wait for p5 to be ready before the communication loop can begin.

3. Process and Methods
    • We kept the overall structure close to what was demonstrated in class and simplified the communication to a single value per message. On the p5 side, mouseX is mapped to a range of 0–255 using map() and constrained with constrain() to prevent out-of-range values. This number is sent to Arduino as a string followed by a newline character using port.write().
    • On the Arduino side, the sketch begins with a handshake loop that repeatedly sends “0” and blinks the built-in LED until p5 responds. Once connected, Serial.parseInt() reads the incoming integer from the serial buffer. After confirming the message ends with ‘\n’, analogWrite() applies the value to the LED on pin 5. Because analogWrite() requires a PWM-capable pin, the LED must be connected to one of the pins marked with a tilde (~) on the Arduino board — in our case, pin 5.
4. Technical Details
    • The p5 sketch maps mouse position to brightness and sends it continuously while the port is open:
// ── CALCULATE BRIGHTNESS ──
// Map the mouse's horizontal position across the canvas
// to a brightness value in the range 0–255.
// This matches the range that analogWrite() accepts on Arduino.
let brightness = int(map(mouseX, 0, width, 0, 255));

// ── CONSTRAIN ──
// Clamp the value so it never exceeds 0–255,
// even if the mouse moves outside the canvas.
brightness = constrain(brightness, 0, 255);
// ── SEND TO ARDUINO ──
// Only write if the port is open (i.e., user has connected).
// We append '\n' so Arduino's Serial.read() can detect
// the end of the message after parseInt() runs.
if (port.opened()) {
  port.write(brightness + "\n");
}
    • Arduino reads the value and applies it to the LED:
// ── READ BRIGHTNESS ──
// parseInt() reads digits from the buffer and returns them as an integer. 
// It stops at any non-digit character (including the newline).
int brightness = Serial.parseInt();

// ── CONFIRM COMPLETE MESSAGE ──
// After parseInt(), the next character should be '\n'.
// This confirms we received a full message and not a partial one, preventing corrupted values.
if (Serial.read() == '\n') {

  // ── SET LED BRIGHTNESS ──
  // analogWrite(pin, 0–255) uses PWM to control brightness.
  // 0 = fully off, 255 = fully on, values in between = dimmed.
  analogWrite(ledPin, brightness);
}
    • The ‘\n’ check after parseInt() is important — it confirms that a complete message was received before acting on the value, which prevents the LED from responding to corrupted or partial data.
5. Reflection

This exercise made the communication feel more interactive than Exercise 01, because the browser was no longer just a display — it was actively sending instructions to hardware. The main issue we encountered was that the LED did not dim smoothly at first. After checking the wiring, we found the LED was connected to a non-PWM pin, which meant analogWrite() had no effect. Moving it to pin 5 resolved this immediately. This was a useful reminder that the physical wiring must match the assumptions in the code. If we were to continue, we would replace mouse control with a more interesting interaction such as key presses or dragging, add multiple LEDs with independent brightness values, and eventually expand into bi-directional communication.

Exercise 3

1. Overview

This exercise brought together everything from the previous two by implementing full bi-directional communication between Arduino and p5.js. A joystick connected to Arduino controls the wind force in a physics simulation, while p5 sends a signal back to Arduino to light up an LED every time the ball bounces. Data flows in both directions simultaneously.

2. Concept

The gravity and wind sketch provided a compelling context for bi-directional communication because it has two clearly distinct interactions that naturally map to each side: the joystick controls something in the simulation, and something in the simulation triggers a response on the hardware. Rather than the user controlling a parameter directly, the LED reacts to an event — a bounce — which made the physical and digital feel genuinely connected rather than just linked. Replacing keyboard input with a physical joystick also made the experience more immersive, since it gave the user a tangible way to influence the simulation.

3. Process and Methods
    • We started from the gravity and wind example provided in class and kept the simulation structure mostly unchanged. The main modifications were replacing keyboard-based wind control with joystick input, and adding bounce detection that communicates back to Arduino.
    • On the Arduino side, the joystick’s horizontal axis is read from pin A0 using analogRead() and sent to p5 via Serial.println() after every incoming message. On the p5 side, the value is read with port.readUntil(“\n”), trimmed, converted to an integer, and mapped from 0-1023 to a wind force range of -1 to 1. This value is applied to the wind vector each frame, so tilting the joystick left or right pushes the ball accordingly.
    • For the return signal, we added bounce detection inside draw(). Each frame, a variable bounced is initialised to 0. If the ball hits the floor with a downward velocity greater than 2, it is counted as a real bounce and bounced is set to 1. This value is sent back to Arduino using port.write(). Arduino reads it with Serial.parseInt() and calls digitalWrite() to turn the LED on or off.
4. Technical Details
    • The bounce detection uses a velocity threshold to distinguish real impacts from the small residual movements that occur when the ball settles on the ground:
  // ── BOUNCE DETECTION ──
  let bounced = 0;                    // assume no bounce this frame
  let floorY = height - mass / 2;    // y where ball touches the floor

  if (position.y > floorY) {
    position.y = floorY;   // prevent ball from going below floor

    // ── VELOCITY THRESHOLD ──
    // Only count as a real bounce if the ball hits with enough
    // downward speed (> 2). This filters out the tiny movements when the ball is nearly at rest
    // Would otherwise cause the LED to flicker continuously.
    if (velocity.y > 2) {
      velocity.y *= -0.9;   // reverse and reduce (energy loss on impact)
      bounced = 1;           // signal a real bounce to Arduino
    } else {
      velocity.y = 0;        // ball has come to rest — stop it completely
    }
  }

  // ── SEND BOUNCE STATE TO ARDUINO ──
  // Sends 1 if a real bounce happened this frame, 0 otherwise.
  // Arduino uses this to briefly light the LED on each bounce.
  // '\n' is appended so Arduino can confirm end of message.
  if (port.opened()) {
    port.write(bounced + "\n");
  }
}
    • On the Arduino side, the received state is applied directly to the LED:
// ── READ LED STATE FROM p5 ──
// p5 sends '1' when a bounce is detected, '0' otherwise.
// parseInt() extracts the integer from the incoming string.
int ledState = Serial.parseInt();

// ── CONFIRM COMPLETE MESSAGE ──
// Check for the newline that p5 appends with port.write().
// This prevents acting on partial or corrupted values.
if (Serial.read() == '\n') {

  // ── CONTROL LED ──
  // ledState is 1 (HIGH) when p5 detects a bounce → LED on.
  // ledState is 0 (LOW) otherwise → LED off.
  digitalWrite(ledPin, ledState);
}
    • The joystick read and send happens inside the same while (Serial.available()) block, directly after the LED state is processed. This keeps the two directions of communication tightly coupled within each communication cycle.
5. Reflection

This was the most technically demanding exercise of the three, and also the most rewarding. The biggest challenge was the LED flickering continuously when the ball came to rest on the ground. Even though the ball appeared stationary, the physics simulation kept generating tiny downward movements due to gravity, which the system kept interpreting as bounces. Introducing the velocity threshold velocity.y > 2 solved this cleanly and also taught us something broader: physics simulations produce continuous, noisy output, and meaningful events often need an explicit threshold or condition to be extracted from that noise. If we were to continue, we would add a dead zone to the joystick centre to reduce drift, use both joystick axes for more complex motion, and add more LEDs to represent different simulation events.

6. Resources