Author: Darmen

Muiz is an interactive 3D artwork where the user can manipulate Kazakh ornament shapes (such as mүйіз, қосмүйіз, and қыңырмүйіз) using a sensor glove and head movement.

The project uses:
1. Arduino + flex sensors on a glove to read finger bends and hand gestures
2. A webcam to track the user’s head movement
3. p5.js with WEBGL to display the ornaments on a 3D canvas
Basically, the user controls and deforms existing ornament patterns in real time, like sculpting them in the air.

How It Works
Glove with Flex Sensors (Arduino)
– Several flex sensors are placed on the glove fingers.
– When the user bends a finger, the sensor value changes.
– Arduino reads these values and sends them to the computer (to p5.js) via serial communication.

These finger bends and hand gestures are used to control the ornament:
– change size
– twist the shape
– bend or “curl” the pattern
– blend between different ornament types

Head Movement: Camera Tracking
A webcam is used to track the user’s head position and movement (some libraries shown in class will be used)

Head movement can control the view in 3D space:
– turn head left/right → rotate around the ornament
– move head up/down → change vertical angle (look from above or below)
– lean forward/back → zoom in and out
It feels like walking around a floating 3D sculpture.

Initial finger mapping:
1. Index finger bend → scale (make ornament bigger/smaller)
2. Middle finger bend → twist/rotate the pattern
3. Ring finger bend → bend the pattern, make it more “curvy”
4. Small finger bend → thickness of the lines or 3D extrusion
5. Thumb + index pinch → switch ornament type (mүйіз → қосмүйіз → қыңырмүйіз)
6. Fist → reset shape to original form
7. Open hand → start a smooth animation (ornament slowly breathing, pulsing)

Task 1: 
Light sensor controls ellipse on p5.js
https://drive.google.com/file/d/1LG16uZcffBsWkmddW0rGVTF8lVujMUjo/view?usp=sharing

int lightPin = A0;
void setup() {
Serial.begin(9600);
pinMode(lightPin, INPUT);
}
void loop() {
int sensorValue = analogRead(A0);
Serial.println(sensorValue);
delay(5);
}

let address = 0;

function setup() {
  createCanvas(600, 600);
  noFill();
}

function draw() {
  background("purple");
  stroke("white");

  // Convert the incoming sensor reading (0–1023) into a horizontal screen position
  ellipse(map(address, 0, 1023, 0, width), height / 2, 100, 100);

  if (!serialActive) {
    // Show a connection screen while serial communication hasn’t started yet
    background("rgb(70,9,70)");
    stroke("white");
    textSize(50);
    text("Press Space Bar to select Serial Port", 20, 30, width - 30, 200);
  }
}

function keyPressed() {
  // When the space bar is pressed, begin the setup process for the serial port
  if (key == " ") setUpSerial();
}

function readSerial(data) {
  // If valid data arrives from the Arduino, save it for use in draw()
  if (data != null) {
    address = int(data);
  }
}

Task 2:
Controlling brightness of LED on p5.js using mouseX position
https://drive.google.com/file/d/1F5u96LDEvKJTtBuD6cbVIIlCcqiFUnyZ/view?usp=sharing

// Holds the brightness value we will send to the Arduino
let brightness = 0;

// Stores any data received back from Arduino (not used, but required)
let latestData = "";

function setup() {
  // Create the canvas where visual feedback will appear
  createCanvas(600, 400);
  noStroke();
}

function draw() {
  // Clear the screen each frame with a black background
  background(0);

  // Convert trackpad/mouse X position (0 → width) into brightness (0 → 255)
  brightness = int(map(mouseX, 0, width, 0, 255));

  // Draw a rectangle whose fill intensity matches the brightness value
  fill(brightness);
  rect(0, 0, width, height);

  // If a serial port is active, send the brightness value to the Arduino
  if (serialActive) {
    writeSerial(brightness + "\n"); // "\n" ensures Arduino reads full numbers
  }

  // If serial is NOT open, show instructions to the user
  if (!serialActive) {
    background("purple");
    fill("white");
    textSize(28);
    text("Press SPACE to choose Serial Port", 20, 40);
  }
}

function keyPressed() {
  // Press SPACE to open the Web Serial port selection dialog
  if (key === " ") {
    setUpSerial();
  }
}

// This function is REQUIRED by p5.webserial
// It receives data sent from Arduino (even if unused)
function readSerial(data) {
  if (data) latestData = data;
}

// Sends data to Arduino IF the writer is available
function writeSerial(value) {
  if (writer) {
    writer.write(value);
  }
}

int ledPin = 9;
int brightness = 0;

void setup() {
  Serial.begin(9600);
  pinMode(ledPin, OUTPUT);
}

void loop() {
  if (Serial.available() > 0) {
    brightness = Serial.parseInt();
    brightness = constrain(brightness, 0, 255);
  }

  analogWrite(ledPin, brightness);
}

Task 3:
Potentiometer to control the direction of the wind and LED light when when ball bounces
https://drive.google.com/file/d/1x8fewdACiGBJ0Qv7vc6tiyZ5ukW19EtK/view?usp=sharing

int LED = 9;
int POT = A0;


void setup() {
Serial.begin(9600);
pinMode(LED, OUTPUT);


// Test the LED
digitalWrite(LED, HIGH);
delay(500);
digitalWrite(LED, LOW);
}


void loop() {
int p_value = analogRead(POT); // read from the potentiometer
int move = map(p_value, 0, 1023, -1, 2); // map the value to -1, 0, and 1
Serial.println(move);


if (Serial.available() > 0) {
// read from p5.js
int touch = Serial.parseInt();
// set the LED command
if (touch == 1) {
digitalWrite(LED, HIGH);
} else {
digitalWrite(LED, LOW);
}
}
}

let velocity;
let gravity;
let position;
let acceleration;
let wind;
let drag = 0.99;
let mass = 50;
let on = 0;

function setup() {
  createCanvas(640, 360);
  //noFill();
  position = createVector(width/2, 0);
  velocity = createVector(0,0);
  acceleration = createVector(0,0);
  gravity = createVector(0, 0.5*mass);
  wind = createVector(0,0);
}

function draw() {
  background(255);
  
  if (!serialActive) {
    text("Click on the Screen to select Serial Port", 20, 30);
  } else {
    text("Connected", 20, 30);
  }
  
  applyForce(wind);
  applyForce(gravity);
  velocity.add(acceleration);
  velocity.mult(drag);
  position.add(velocity);
  acceleration.mult(0);
  ellipse(position.x,position.y,mass,mass);
  if (position.y > height-mass/2) {
      velocity.y *= -0.9;  // A little dampening when hitting the bottom
      position.y = height-mass/2;
  }
  // turn on the LED only when it's on the ground or hits the ground
  if(position.y == height-mass/2){ 
    on = 0;
  }else{
    on = 1;
  }
}
function applyForce(force){
  // Newton's 2nd law: F = M * A
  // or A = F / M
  let f = p5.Vector.div(force, mass);
  acceleration.add(f);
}
function keyPressed(){
  if (key==' '){
    mass=random(15, 80);
    position.y=-mass;
    velocity.mult(0);
  }
}
function mousePressed() {
    setUpSerial();
}
function readSerial(data) {
  if (data != null) {
    // make sure there is actually a message
    // split the message
    wind.x = data;
    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    let sendToArduino = on + "\n";
    writeSerial(sendToArduino);
  }
}

Reflection
I generally liked on working and understanding underlying concepts behind how to the Arduino and p5.js sketches work together. I already see my progress being able to understand how pins work and how to connect each sensor properly. I wanted to avoid using distance sensor because I used it for the last project. We worked together with Aizhan, and used AI a bit for the third task to helps us understand how gravity and velocity worked, and it helped us come up with a code to connect potentiometer as a wind.

Concept:

As we began planning how to present our musical instrument and the sound produced by the buzzer while using both digital and analog sensors, we decided to use a button as our digital sensor and a distance-measuring sensor as our analog sensor. The main concept is that the distance sensor detects how far an object or hand is, and based on that distance, it produces different notes and sounds through the buzzer.  When the button is pressed, the system pauses for 300 milliseconds (0.3 seconds) and temporarily stops reading distance values, effectively muting the instrument. Pressing the button again reactivates the sensor, allowing the instrument to continue playing notes according to the object’s position. This simple toggle system makes it easy to control when the instrument is active, giving users time to experiment with the sound and movement.

Arduino Setup and Demonstration: 

Schematic Diagram:
Setup:Video:

Highlight of the code:
Full-code on Github

if (sensorActive) {
    int distance = getDistance();
    Serial.println(distance);
    if (1 < distance && distance < 5) {
      tone(BUZZER, NOTE_C4);
    } else if (5 < distance && distance < 10) {
      tone(BUZZER, NOTE_D4);
    } else if (10 < distance && distance < 15) {
      tone(BUZZER, NOTE_E4);
    } else if (15 < distance && distance < 20) {
      tone(BUZZER, NOTE_F4);
    } else if (20 < distance && distance < 25) {
      tone(BUZZER, NOTE_G4);
    } else if (25 < distance && distance < 30) {
      tone(BUZZER, NOTE_A4);
    } else if (30 < distance && distance < 35) {
      tone(BUZZER, NOTE_B4);
    } else {
      noTone(BUZZER);
    }
  }

We would say the highlight of the coding would be the part where the distance-measuring sensor interacts with the buzzer to produce different musical notes based on how close or far an object is. We were excited to experiment with the distance-measuring sensor and explore how it could be programmed to produce sound through the buzzer. To get better understanding of the integrating notes we referred to Arduino tutorials. 

In the code above, the sensor continuously measures the distance of an object and converts the time it takes for the sound wave to bounce back into centimeters. Depending on the measured distance, the buzzer plays different musical notes, creating a simple melody that changes as the object moves closer or farther away.

Reflection:

We enjoyed experimenting with the distance-measuring sensor as it taught us how precise sensor readings can be transformed into meaningful outputs like sound, and combining it with the button control helped us manage the instrument’s activation smoothly. For future improvements, we would like to expand the range of notes to create more complex melodies and add LED lights that change color with the pitch to make the instrument more visually engaging. We could also experiment with different sensors, such as touch or motion sensors, to add more layers of interactivity. Finally, refining the accuracy and response speed of the sensor would make the sound transitions smoother and enhance the overall experience.

 

Concept
I wanted to combine things we learned in class with a new sensor. This time, it was an ultrasonic distance sensor and an LED. I think I just wanted to experiment with the brightness of the LED, to see how dim or how bright it would be depending on the distance information from the ultrasonic sensor. I watched a couple of YouTube videos to learn how to use and code for the HC-SR04, and it took me some time. Then I utilized code for sending a signal through the TRIG pin and receiving it back through the ECHO pin, calculating the microseconds, and converting it to centimeters using a specific function.

Demo
Video

Highlight of the code

int ECHO_PIN = 12;       //input, collects data from waves
int TRIG_PIN = 11;       //output, sends out the waves
int SWITCH_PIN = 2;
int SWITCH_STATE;
int RED_LED = 5;         // red to light up when objects close
int GREEN_LED = 6;      // yellow to fade if object getting further
float distance = -1.0;            // variable to store distance

void setup() {
  Serial.begin (9600);         // set up a serial connection with the computer

  pinMode(ECHO_PIN, INPUT);    // echo pin measure the duration of pulses coming back from the distance sensor
  pinMode(TRIG_PIN, OUTPUT);   // trigger pin output pulses of electricity
  pinMode(RED_LED, OUTPUT);
  pinMode(GREEN_LED, OUTPUT);
  pinMode(SWITCH_PIN, INPUT);

  digitalWrite(TRIG_PIN, LOW);
}

void loop() {
  long duration = getEchoDuration();
  if (duration == 0){
    distance = -1.0;
  }
  else{
    distance = microsecondsToCentimeters(duration);
  }

  int SWITCH_STATE = digitalRead(SWITCH_PIN);
  if (SWITCH_STATE == HIGH){
    digitalWrite(RED_LED, HIGH);
  }
  else{
    digitalWrite(RED_LED, LOW);
  }

// if conditional to control LED's depending on distance.
  int brightness = 0;
  if (5 <= distance && distance <= 30) {
    brightness = map(int(distance), 5, 30, 255, 0);
    brightness = constrain(brightness, 0, 255);
  } else if (distance > 0 && distance < 5) {
    brightness = 255;
  } else {
    brightness = 0;
  }
  
  analogWrite(GREEN_LED, brightness);

  Serial.println("Distance: ");
  if (distance < 0){
    Serial.println("Out of range");
  } 
  else {
    Serial.print(distance);
    Serial.print("cm");
  }

  Serial.println("");
  Serial.println("Brightness: ");
  Serial.println(brightness);

    delay(500);
}

long getEchoDuration() {
  digitalWrite(TRIG_PIN, LOW);
  delayMicroseconds(2);
  digitalWrite(TRIG_PIN, HIGH);
  delayMicroseconds(10);
  digitalWrite(TRIG_PIN, LOW);
  long duration = pulseIn(ECHO_PIN, HIGH, 30000);
  return duration;
}
  
float microsecondsToCentimeters(long microseconds) {
  return microseconds / 58.0;
  }

I think the if statement that detects if the distance is between 5 and 30 centimeters, and the else if statements, were quite important after I sent the signal through TRIG, received it through ECHO, and transformed the data into centimeters. So, basically, if the distance is more than 30cm, it will show “out of range” on the serial monitor and the green LED will be off. And if it is between 0cm and 5cm, then the brightness is at its peak, 255, and the green LED is fully bright. Also, the red LED works independently based on the sliding switch that we learned during class.

Sketch
Reflection
It took a long time to learn how the distance sensor works. From what I understood, it is similar to a Ping sensor which has 3 pins, but then I had to modify the code I got from the built-in example for TRIG and ECHO pins separately. For future improvements, I want to incorporate the red LED somehow inside the circuit, like if it is out of range or too close, I want the red light to light up. I know it is just a one-line fix, but then I also want it to work independently as well. Maybe when it is turned on, it will do what I said, but when it is off, then it is not going to do those things.

Physical Computing’s Greatest Hits (and misses)
Just simply scrolling through the list of “greatest hits,” I was struck by how it those simple concepts were utilized nowadays and how universal physical computing actually is. I think seeing these categories laid out so clearly is incredibly useful. For example, those floor dance arcade games, sensory gloves, and multi-touch sensors used in modern interactive museums are using those concepts developed in physical computing. This definitely shows how many systems nowadays are just an extension of physical computing projects. It reminds me that innovation doesn’t always mean inventing something completely new, but can also be about putting a personal or novel twist on a classic concept. I believe the real value here is as a starting point. Instead of staring at a blank slate, I can look at this list and think, “How could I reinterpret a ‘mechanical pin display’ or a ‘glove controller’ in a way that is unique to me?” in my future projects.

Making Interactive Art: Set the Stage, Then Shut Up and Listen
This article fundamentally challenges the way I thought about my role as a creator of interactive things. The instruction to “set the stage, then shut up and listen” is a powerful and difficult one. I think my instinct, like many, is to over-explain, to guide the user to the “correct” experience so they don’t miss my intended point. But this piece reminds me that the magic of interaction happens in the unscripted moments. I believe the author is right. The meaning is not something I embed and the user extracts, but something that is created in the dialogue between the person and the work. It makes me want to create pieces that are more like questions than statements, and to have the confidence to let the audience find their own answers.

Concept
I always keep my magnetic cardholder attached to the back of my phone. It can hold one to three cards comfortably without them falling out. Recently, I almost lost my NYU FAB card, which had around 800 dirhams on it for my study abroad visa appointment. I was so upset and searched my room for two days, cleaning everything. Finally, I found it under my glasses case. That was such a relief. From now on, I’ll always keep it in my cardholder.

For this Arduino project, I reused the LED code from class. The light stays on when the card is outside the holder, and it turns off when the card is inside, showing that it’s safe. I used aluminum foil to extend the wire’s conductivity and duct tape to build a working prototype.

Photo
Video
Unusual Switch on Arduino UNO

Highlight of the code
That’s actually the entirety of the code. The code itself is simple. It starts with a 5V input, and I used digitalWrite(2, LOW) to let the electricity flow since it works from high to low. The system detects when there is contact with the card. If the card touches the sensor, the LED turns off. If there’s no contact and the card is outside the holder, the LED stays on as an alert to remind the user to put the card back for safety. That’s exactly what I did with my FAB card.

void setup() {
  pinMode(13, OUTPUT); // led pin
  pinMode(2, INPUT);  // foil pin
  digitalWrite (2, LOW);
}

void loop() {
  int cardtouch = digitalRead(2);

  if (cardtouch == LOW) { 
    digitalWrite(13, HIGH); 
  } else {
    digitalWrite(13, LOW);
  }

}

Reflection
It’s clear that the prototype isn’t perfect. I can already see ways to improve it, especially since the wires limit the movement of the card. Using a motion or distance sensor would be a better option if I want to make it portable and attach it to my cardholder. The foil isn’t always reliable either. It sometimes slips off the wire, which reduces conductivity and can cause the LED to give the wrong signal. Still, I think it’s great that after just one class, I could use the same code to create something useful that I might actually use in the future.

“Emotion and Design”
Don Norman says that attractive things work better. When something looks good, it makes us feel happy. This good feeling helps us be more creative and more patient if we have a small problem using the product. A happy user is a better problem solver.

I agree with this. A nice looking design, like a pretty website or a well made phone, is more enjoyable to use. It makes you want to keep using it. Good design is not just about being pretty. It is about making things work better by making the user feel good.

“Her Code Got Humans on the Moon”
Margaret Hamilton was a computer scientist who led the team that created the software for the Apollo moon missions. She did this at a time when there were very few women in her field. Her work was vital to landing astronauts on the moon and bringing them home safely.

I am impressed by how Hamilton planned for mistakes. She wanted to add code to prevent errors that everyone else thought would never happen. She was right. An astronaut later made one of those exact mistakes. Her story shows why it is so important to think about what could go wrong, especially on a big and important project. Good planning can prevent big problems.

Overall
Both readings show that the best designs think about people. Norman teaches us that a product that makes users feel good will work better. Hamilton teaches us that a system designed for human error is stronger and safer. Whether it is a teapot or a spaceship, thinking about the person using it is the key to good design.

Concept
As a kid, I admired strikers for scoring beautiful goals, but now I understand what a difficult and beautiful job goalkeepers have in saving their team. My admiration for goalkeeping grew, especially after watching the incredible 2022 World Cup Final between Argentina and France in Qatar.

In the game I’ve made, the player moves the goalkeeper’s gloves with a mouse or trackpad to save penalties. Each player starts with three lives and loses one for every goal conceded. Players can unlock new skins for their gloves. Making 10 saves in a single game unlocks the first skin, and reaching 15 saves in one session unlocks a second one.

The game also saves each player’s personal best record, allowing multiple users to compete and track their high scores. There is also a special golden ball: saving it grants an extra life (+1), but conceding a goal from it costs two lives instead of the usual one. The controls are simple: press ‘F’ for fullscreen and ‘S’ to view and select skins. There is also a hidden ‘R’ key to reset the game and clear all data.

Highlight of the code

function saveUserData() {
  let userData = {
    highScore: highScoreSaves,
    unlockedSkins: unlockedSkins
  };
  localStorage.setItem(currentUser, JSON.stringify(userData));
}

One smart technical decision I learned in my Intro to CS class was to automatically save each player’s progress and unlocked skins. This ensures that players don’t lose their scores and achievements when they close or reload the game. It definitely adds to the experience, making it feel like a real game that saves user accounts and their rewards.

if (dist(ball.x, ball.y, keeper.x, keeper.y) < keeper.size / 2) {
  saveSound.play();
  saves++;
  ball.reset();
}

Another important feature is collision detection, which I implemented using the mathematical dist() function. The method of checking for an overlap between two circles, which was also taught in my Intro to CS class, is a much easier way to calculate collisions in a simple game. When a collision is detected, the code plays the ‘save’ sound, increments the save counter, and resets the ball to its starting position. I think this approach is very simple, short, and super-efficient.

Sketch
https://editor.p5js.org/da3490/sketches/e03ZtWqpi

Reflection
I drew on my previous experience creating a Jetpack Joyride duplicate in my Intro to CS class to build this game’s features, including the login and menu pages, score tracking, user record saving, collision detection, and skins. While developing the UI took some time, it wasn’t logically difficult.

I would say that making the game adaptive to different screen sizes took the most effort. Instead of using fixed pixel values, I had to use percentages and mathematical relationships, such as width * 0.5 and height * 0.3. This approach was tricky but ensures the game looks and works properly on any device. I also added glow and glass effects to enhance the UI aesthetics. Using drawingContext and shadowBlur, I created transparent rectangles that produced a modern ‘glass’ look. While this style may be less popular now, it was a prominent trend in 2023-2024.

Future improvements
For future improvements, I would love to replicate FIFA-style penalties with a two-player mode: one player aims and shoots while the other plays as the goalkeeper.

To achieve a more realistic, front-facing perspective instead of the current view, I would need to implement 3D-like mechanics. This would involve angle calculations and scaling the ball’s size to create a sense of depth. The goalkeeper would use keyboard buttons to move in all eight directions (left, right, up, down, and diagonals) and would have a jump mechanic to make saves. For the shooter, the mouse would control the aim, while holding down the spacebar would determine the power of the shot.