Author: Darmen

Are they able to figure it out? Where do they get confused and why?
My friends were a bit confused what to do in first place. So, I included instructions on top left corner, that by doing a fist they could transition between different 3D models and by using index finger and thumb they could manipulate the sketch.

Do they understand the mapping between the controls and what happens in the experience?
They do understand it. Controls how to use it is already given in the top left corner as instructions, so no problem with that.

What parts of the experience are working well? What areas could be improved?
I’m still working on adding new 3D models and adding explanation to each one and how it is specifically tied to Kazakh culture. I want people to learn about Kazakh culture and feel the items, observe them, and etc. I want to do a menu screen, and another instructions screen. Add 3 more Kazakh Cultural elements. Also, more features that would allow to do something interesting with the sketch line draw lines and oscillations and something interactive like this on the background of 3D model. In addition, I will working heavily on Arduino side from now on, I want to make Neopixel Adafruit not just turn a particular color, but make a beautiful pattern, so others on the exhibition will also come to take a look.

What parts of your project did you feel the need to explain? How could you make these areas more clear to someone that is experiencing your project for the first time?
I think using a fist to transition between different 3D models is pretty cool idea, and I get to explore hand gestures more. I will try to make instructions constant on the page, in case user forgot, they will always be there. Also, in main menu I will try to explain a bit about the project itself. The project teaches people about Kazakh culture through interactive 3D objects. The user moves their hand in front of the webcam and controls a 3D asyq, kiiz ui, dombyra, taqiya in real time. Hand rotation and hand openness change rotation, scale, and animation of the models. The goal is to make cultural objects feel alive and playful. Handpose in p5.js finds key points on the user hand. The code maps these points to rotation and size values. The 3D models load from external OBJ files. The models update their rotation and scale on every frame based on hand movement. The interaction is simple so anyone can explore the cultural objects. The user brings their hand into the camera. Turning the index finger rotates the model.  Bigger pinch between thumb and index distance makes the model grow. I want to implement moving the hand forward or backward changes the distance of the models. The idea is to mix physical movement with digital cultural storytelling. The Arduino sends arcade button state to. p5.js reads the values using the Web Serial API and starts the game. From p5.js to Arduino communication happens when user does a fist, then the 3D model changes and Adafruit Neopixel lights up a different color depending on which particular 3D model is on the screen. So far, asyq (blue), kiiz ui (orange), dombyra (red), taqiya (green). I am proud that the project takes Kazakh cultural objects and makes them interactive. I like that people can learn culture through movement. I am happy that the 3D models work smoothly with hand tracking. I used generative AI to help fix errors, structure the code, and write the description. All cultural objects are based on real Kazakh designs. I found 3D models online on Sketchfab, and I will reference them. I want to add more gestures for cultural actions like throwing the asyq. I want to add more Kazakh cultural objects.  I also want to create a cleaner guide so visitors can learn the meaning of each cultural object.

(Progress pictures)
The idea of the project is still the same, but renamed it to Mädenīet, which means culture in Kazakh. The game itself is focused on learning about various aspects of Kazakh culture. I found 3D models of dombyra, kiiz ui, taqiya, and asyq online through Sketchfab. They were of fbx format, so I used online converter to make them obj format, which p5 accepts. After trying to upload them to p5.js, they were too big, so I cloned everything to my VS Code, and decided that I will be running them locally instead. From Arduino side, I soldered arcade button, and added a circular Adafruit Neopixel with 24 LED. If we want to get into details, I connected Neopixel to digital pin 11, and LED side of arcade button to pin 9, and pin 6 to switch side of it. I laser printed out a box that I designed with Kazakh national ornaments in Adobe Illustrator. Made special holes from the back for Arduino cable to go through. I will be making a special holes for Neopixel cable to go from the front as well. I uploaded a background music of my favourite Soviet Kazakh ‘Vocal and Instrumental Ensemble’ – Dos Mukasan, and users will be listening through headphones and also be able to experience the game. User will press ‘Space’ to connect to Arduino port, then be required to press the arcade button (from Arduino to p5js connection). As soon as the hand is detected by the camera, the game and the music starts. With hand gestures of index finger user is able to rotate, and pinch between index and thumb allows to zoom in or zoom out. Making a fist, will transition to the next 3D model. Also, on the left side of each 3D model will be a short description of what this 3D model is and how valuable it is to the Kazakh culture. Change in different 3D models will send a signal from p5.js to Arduino by turning a neopixel into different colors (blue, orange, red, green). After Professor’s initial feedback, I decided not to pursue glove sensors and use Handpose ml5.js library to track the hand using a webcam, which made the coding much easier and so much more engaging. 

Muiz is an interactive 3D artwork where the user can manipulate Kazakh ornament shapes (such as mүйіз, қосмүйіз, and қыңырмүйіз) using a sensor glove and head movement.

The project uses:
1. Arduino + flex sensors on a glove to read finger bends and hand gestures
2. A webcam to track the user’s head movement
3. p5.js with WEBGL to display the ornaments on a 3D canvas
Basically, the user controls and deforms existing ornament patterns in real time, like sculpting them in the air.

How It Works
Glove with Flex Sensors (Arduino)
– Several flex sensors are placed on the glove fingers.
– When the user bends a finger, the sensor value changes.
– Arduino reads these values and sends them to the computer (to p5.js) via serial communication.

These finger bends and hand gestures are used to control the ornament:
– change size
– twist the shape
– bend or “curl” the pattern
– blend between different ornament types

Head Movement: Camera Tracking
A webcam is used to track the user’s head position and movement (some libraries shown in class will be used)

Head movement can control the view in 3D space:
– turn head left/right → rotate around the ornament
– move head up/down → change vertical angle (look from above or below)
– lean forward/back → zoom in and out
It feels like walking around a floating 3D sculpture.

Initial finger mapping:
1. Index finger bend → scale (make ornament bigger/smaller)
2. Middle finger bend → twist/rotate the pattern
3. Ring finger bend → bend the pattern, make it more “curvy”
4. Small finger bend → thickness of the lines or 3D extrusion
5. Thumb + index pinch → switch ornament type (mүйіз → қосмүйіз → қыңырмүйіз)
6. Fist → reset shape to original form
7. Open hand → start a smooth animation (ornament slowly breathing, pulsing)

Task 1: 
Light sensor controls ellipse on p5.js
https://drive.google.com/file/d/1LG16uZcffBsWkmddW0rGVTF8lVujMUjo/view?usp=sharing

int lightPin = A0;
void setup() {
Serial.begin(9600);
pinMode(lightPin, INPUT);
}
void loop() {
int sensorValue = analogRead(A0);
Serial.println(sensorValue);
delay(5);
}

let address = 0;

function setup() {
  createCanvas(600, 600);
  noFill();
}

function draw() {
  background("purple");
  stroke("white");

  // Convert the incoming sensor reading (0–1023) into a horizontal screen position
  ellipse(map(address, 0, 1023, 0, width), height / 2, 100, 100);

  if (!serialActive) {
    // Show a connection screen while serial communication hasn’t started yet
    background("rgb(70,9,70)");
    stroke("white");
    textSize(50);
    text("Press Space Bar to select Serial Port", 20, 30, width - 30, 200);
  }
}

function keyPressed() {
  // When the space bar is pressed, begin the setup process for the serial port
  if (key == " ") setUpSerial();
}

function readSerial(data) {
  // If valid data arrives from the Arduino, save it for use in draw()
  if (data != null) {
    address = int(data);
  }
}

Task 2:
Controlling brightness of LED on p5.js using mouseX position
https://drive.google.com/file/d/1F5u96LDEvKJTtBuD6cbVIIlCcqiFUnyZ/view?usp=sharing

// Holds the brightness value we will send to the Arduino
let brightness = 0;

// Stores any data received back from Arduino (not used, but required)
let latestData = "";

function setup() {
  // Create the canvas where visual feedback will appear
  createCanvas(600, 400);
  noStroke();
}

function draw() {
  // Clear the screen each frame with a black background
  background(0);

  // Convert trackpad/mouse X position (0 → width) into brightness (0 → 255)
  brightness = int(map(mouseX, 0, width, 0, 255));

  // Draw a rectangle whose fill intensity matches the brightness value
  fill(brightness);
  rect(0, 0, width, height);

  // If a serial port is active, send the brightness value to the Arduino
  if (serialActive) {
    writeSerial(brightness + "\n"); // "\n" ensures Arduino reads full numbers
  }

  // If serial is NOT open, show instructions to the user
  if (!serialActive) {
    background("purple");
    fill("white");
    textSize(28);
    text("Press SPACE to choose Serial Port", 20, 40);
  }
}

function keyPressed() {
  // Press SPACE to open the Web Serial port selection dialog
  if (key === " ") {
    setUpSerial();
  }
}

// This function is REQUIRED by p5.webserial
// It receives data sent from Arduino (even if unused)
function readSerial(data) {
  if (data) latestData = data;
}

// Sends data to Arduino IF the writer is available
function writeSerial(value) {
  if (writer) {
    writer.write(value);
  }
}

int ledPin = 9;
int brightness = 0;

void setup() {
  Serial.begin(9600);
  pinMode(ledPin, OUTPUT);
}

void loop() {
  if (Serial.available() > 0) {
    brightness = Serial.parseInt();
    brightness = constrain(brightness, 0, 255);
  }

  analogWrite(ledPin, brightness);
}

Task 3:
Potentiometer to control the direction of the wind and LED light when when ball bounces
https://drive.google.com/file/d/1x8fewdACiGBJ0Qv7vc6tiyZ5ukW19EtK/view?usp=sharing

int LED = 9;
int POT = A0;


void setup() {
Serial.begin(9600);
pinMode(LED, OUTPUT);


// Test the LED
digitalWrite(LED, HIGH);
delay(500);
digitalWrite(LED, LOW);
}


void loop() {
int p_value = analogRead(POT); // read from the potentiometer
int move = map(p_value, 0, 1023, -1, 2); // map the value to -1, 0, and 1
Serial.println(move);


if (Serial.available() > 0) {
// read from p5.js
int touch = Serial.parseInt();
// set the LED command
if (touch == 1) {
digitalWrite(LED, HIGH);
} else {
digitalWrite(LED, LOW);
}
}
}

let velocity;
let gravity;
let position;
let acceleration;
let wind;
let drag = 0.99;
let mass = 50;
let on = 0;

function setup() {
  createCanvas(640, 360);
  //noFill();
  position = createVector(width/2, 0);
  velocity = createVector(0,0);
  acceleration = createVector(0,0);
  gravity = createVector(0, 0.5*mass);
  wind = createVector(0,0);
}

function draw() {
  background(255);
  
  if (!serialActive) {
    text("Click on the Screen to select Serial Port", 20, 30);
  } else {
    text("Connected", 20, 30);
  }
  
  applyForce(wind);
  applyForce(gravity);
  velocity.add(acceleration);
  velocity.mult(drag);
  position.add(velocity);
  acceleration.mult(0);
  ellipse(position.x,position.y,mass,mass);
  if (position.y > height-mass/2) {
      velocity.y *= -0.9;  // A little dampening when hitting the bottom
      position.y = height-mass/2;
  }
  // turn on the LED only when it's on the ground or hits the ground
  if(position.y == height-mass/2){ 
    on = 0;
  }else{
    on = 1;
  }
}
function applyForce(force){
  // Newton's 2nd law: F = M * A
  // or A = F / M
  let f = p5.Vector.div(force, mass);
  acceleration.add(f);
}
function keyPressed(){
  if (key==' '){
    mass=random(15, 80);
    position.y=-mass;
    velocity.mult(0);
  }
}
function mousePressed() {
    setUpSerial();
}
function readSerial(data) {
  if (data != null) {
    // make sure there is actually a message
    // split the message
    wind.x = data;
    //////////////////////////////////
    //SEND TO ARDUINO HERE (handshake)
    //////////////////////////////////
    let sendToArduino = on + "\n";
    writeSerial(sendToArduino);
  }
}

Reflection
I generally liked on working and understanding underlying concepts behind how to the Arduino and p5.js sketches work together. I already see my progress being able to understand how pins work and how to connect each sensor properly. I wanted to avoid using distance sensor because I used it for the last project. We worked together with Aizhan, and used AI a bit for the third task to helps us understand how gravity and velocity worked, and it helped us come up with a code to connect potentiometer as a wind.

Concept:

As we began planning how to present our musical instrument and the sound produced by the buzzer while using both digital and analog sensors, we decided to use a button as our digital sensor and a distance-measuring sensor as our analog sensor. The main concept is that the distance sensor detects how far an object or hand is, and based on that distance, it produces different notes and sounds through the buzzer.  When the button is pressed, the system pauses for 300 milliseconds (0.3 seconds) and temporarily stops reading distance values, effectively muting the instrument. Pressing the button again reactivates the sensor, allowing the instrument to continue playing notes according to the object’s position. This simple toggle system makes it easy to control when the instrument is active, giving users time to experiment with the sound and movement.

Arduino Setup and Demonstration: 

Schematic Diagram:
Setup:Video:

Highlight of the code:
Full-code on Github

if (sensorActive) {
    int distance = getDistance();
    Serial.println(distance);
    if (1 < distance && distance < 5) {
      tone(BUZZER, NOTE_C4);
    } else if (5 < distance && distance < 10) {
      tone(BUZZER, NOTE_D4);
    } else if (10 < distance && distance < 15) {
      tone(BUZZER, NOTE_E4);
    } else if (15 < distance && distance < 20) {
      tone(BUZZER, NOTE_F4);
    } else if (20 < distance && distance < 25) {
      tone(BUZZER, NOTE_G4);
    } else if (25 < distance && distance < 30) {
      tone(BUZZER, NOTE_A4);
    } else if (30 < distance && distance < 35) {
      tone(BUZZER, NOTE_B4);
    } else {
      noTone(BUZZER);
    }
  }

We would say the highlight of the coding would be the part where the distance-measuring sensor interacts with the buzzer to produce different musical notes based on how close or far an object is. We were excited to experiment with the distance-measuring sensor and explore how it could be programmed to produce sound through the buzzer. To get better understanding of the integrating notes we referred to Arduino tutorials. 

In the code above, the sensor continuously measures the distance of an object and converts the time it takes for the sound wave to bounce back into centimeters. Depending on the measured distance, the buzzer plays different musical notes, creating a simple melody that changes as the object moves closer or farther away.

Reflection:

We enjoyed experimenting with the distance-measuring sensor as it taught us how precise sensor readings can be transformed into meaningful outputs like sound, and combining it with the button control helped us manage the instrument’s activation smoothly. For future improvements, we would like to expand the range of notes to create more complex melodies and add LED lights that change color with the pitch to make the instrument more visually engaging. We could also experiment with different sensors, such as touch or motion sensors, to add more layers of interactivity. Finally, refining the accuracy and response speed of the sensor would make the sound transitions smoother and enhance the overall experience.

 

Concept
I wanted to combine things we learned in class with a new sensor. This time, it was an ultrasonic distance sensor and an LED. I think I just wanted to experiment with the brightness of the LED, to see how dim or how bright it would be depending on the distance information from the ultrasonic sensor. I watched a couple of YouTube videos to learn how to use and code for the HC-SR04, and it took me some time. Then I utilized code for sending a signal through the TRIG pin and receiving it back through the ECHO pin, calculating the microseconds, and converting it to centimeters using a specific function.

Demo
Video

Highlight of the code

int ECHO_PIN = 12;       //input, collects data from waves
int TRIG_PIN = 11;       //output, sends out the waves
int SWITCH_PIN = 2;
int SWITCH_STATE;
int RED_LED = 5;         // red to light up when objects close
int GREEN_LED = 6;      // yellow to fade if object getting further
float distance = -1.0;            // variable to store distance

void setup() {
  Serial.begin (9600);         // set up a serial connection with the computer

  pinMode(ECHO_PIN, INPUT);    // echo pin measure the duration of pulses coming back from the distance sensor
  pinMode(TRIG_PIN, OUTPUT);   // trigger pin output pulses of electricity
  pinMode(RED_LED, OUTPUT);
  pinMode(GREEN_LED, OUTPUT);
  pinMode(SWITCH_PIN, INPUT);

  digitalWrite(TRIG_PIN, LOW);
}

void loop() {
  long duration = getEchoDuration();
  if (duration == 0){
    distance = -1.0;
  }
  else{
    distance = microsecondsToCentimeters(duration);
  }

  int SWITCH_STATE = digitalRead(SWITCH_PIN);
  if (SWITCH_STATE == HIGH){
    digitalWrite(RED_LED, HIGH);
  }
  else{
    digitalWrite(RED_LED, LOW);
  }

// if conditional to control LED's depending on distance.
  int brightness = 0;
  if (5 <= distance && distance <= 30) {
    brightness = map(int(distance), 5, 30, 255, 0);
    brightness = constrain(brightness, 0, 255);
  } else if (distance > 0 && distance < 5) {
    brightness = 255;
  } else {
    brightness = 0;
  }
  
  analogWrite(GREEN_LED, brightness);

  Serial.println("Distance: ");
  if (distance < 0){
    Serial.println("Out of range");
  } 
  else {
    Serial.print(distance);
    Serial.print("cm");
  }

  Serial.println("");
  Serial.println("Brightness: ");
  Serial.println(brightness);

    delay(500);
}

long getEchoDuration() {
  digitalWrite(TRIG_PIN, LOW);
  delayMicroseconds(2);
  digitalWrite(TRIG_PIN, HIGH);
  delayMicroseconds(10);
  digitalWrite(TRIG_PIN, LOW);
  long duration = pulseIn(ECHO_PIN, HIGH, 30000);
  return duration;
}
  
float microsecondsToCentimeters(long microseconds) {
  return microseconds / 58.0;
  }

I think the if statement that detects if the distance is between 5 and 30 centimeters, and the else if statements, were quite important after I sent the signal through TRIG, received it through ECHO, and transformed the data into centimeters. So, basically, if the distance is more than 30cm, it will show “out of range” on the serial monitor and the green LED will be off. And if it is between 0cm and 5cm, then the brightness is at its peak, 255, and the green LED is fully bright. Also, the red LED works independently based on the sliding switch that we learned during class.

Sketch
Reflection
It took a long time to learn how the distance sensor works. From what I understood, it is similar to a Ping sensor which has 3 pins, but then I had to modify the code I got from the built-in example for TRIG and ECHO pins separately. For future improvements, I want to incorporate the red LED somehow inside the circuit, like if it is out of range or too close, I want the red light to light up. I know it is just a one-line fix, but then I also want it to work independently as well. Maybe when it is turned on, it will do what I said, but when it is off, then it is not going to do those things.

Physical Computing’s Greatest Hits (and misses)
Just simply scrolling through the list of “greatest hits,” I was struck by how it those simple concepts were utilized nowadays and how universal physical computing actually is. I think seeing these categories laid out so clearly is incredibly useful. For example, those floor dance arcade games, sensory gloves, and multi-touch sensors used in modern interactive museums are using those concepts developed in physical computing. This definitely shows how many systems nowadays are just an extension of physical computing projects. It reminds me that innovation doesn’t always mean inventing something completely new, but can also be about putting a personal or novel twist on a classic concept. I believe the real value here is as a starting point. Instead of staring at a blank slate, I can look at this list and think, “How could I reinterpret a ‘mechanical pin display’ or a ‘glove controller’ in a way that is unique to me?” in my future projects.

Making Interactive Art: Set the Stage, Then Shut Up and Listen
This article fundamentally challenges the way I thought about my role as a creator of interactive things. The instruction to “set the stage, then shut up and listen” is a powerful and difficult one. I think my instinct, like many, is to over-explain, to guide the user to the “correct” experience so they don’t miss my intended point. But this piece reminds me that the magic of interaction happens in the unscripted moments. I believe the author is right. The meaning is not something I embed and the user extracts, but something that is created in the dialogue between the person and the work. It makes me want to create pieces that are more like questions than statements, and to have the confidence to let the audience find their own answers.

Concept
I always keep my magnetic cardholder attached to the back of my phone. It can hold one to three cards comfortably without them falling out. Recently, I almost lost my NYU FAB card, which had around 800 dirhams on it for my study abroad visa appointment. I was so upset and searched my room for two days, cleaning everything. Finally, I found it under my glasses case. That was such a relief. From now on, I’ll always keep it in my cardholder.

For this Arduino project, I reused the LED code from class. The light stays on when the card is outside the holder, and it turns off when the card is inside, showing that it’s safe. I used aluminum foil to extend the wire’s conductivity and duct tape to build a working prototype.

Photo
Video
Unusual Switch on Arduino UNO

Highlight of the code
That’s actually the entirety of the code. The code itself is simple. It starts with a 5V input, and I used digitalWrite(2, LOW) to let the electricity flow since it works from high to low. The system detects when there is contact with the card. If the card touches the sensor, the LED turns off. If there’s no contact and the card is outside the holder, the LED stays on as an alert to remind the user to put the card back for safety. That’s exactly what I did with my FAB card.

void setup() {
  pinMode(13, OUTPUT); // led pin
  pinMode(2, INPUT);  // foil pin
  digitalWrite (2, LOW);
}

void loop() {
  int cardtouch = digitalRead(2);

  if (cardtouch == LOW) { 
    digitalWrite(13, HIGH); 
  } else {
    digitalWrite(13, LOW);
  }

}

Reflection
It’s clear that the prototype isn’t perfect. I can already see ways to improve it, especially since the wires limit the movement of the card. Using a motion or distance sensor would be a better option if I want to make it portable and attach it to my cardholder. The foil isn’t always reliable either. It sometimes slips off the wire, which reduces conductivity and can cause the LED to give the wrong signal. Still, I think it’s great that after just one class, I could use the same code to create something useful that I might actually use in the future.