Week 12 – Finalised Concept

For my final project, I am building a 3D Music Interface that allows users to trigger both 3D visual animations and musical sounds through a physical touch-sensitive controller. The interface will be made out of cardboard and tinfoil touchpads connected to an Arduino. Each physical touch will trigger a corresponding 3D box animation in p5.js, along with a musical note generated directly in the browser using the p5.sound library. The goal is to create a playful, intuitive instrument that connects physical gestures to both sound and visual expression immediately and reliably.

Arduino Program Design

The Arduino will act as the input device, reading signals from multiple touch-sensitive pads made of tinfoil and cardboard.

  • Inputs:

    • Each touchpad will be connected to a digital input pin.

    • When the user touches a pad, it will register a HIGH signal on that pin.

  • Processing:

    • The Arduino continuously checks the state of each touchpad inside the loop() function.

    • When a touch is detected, the Arduino sends a unique identifier through Serial communication.

    • Each touchpad is mapped to a different number (e.g., 0, 1, 2, 3, up to 7).

  • Outputs:

    • The Arduino only sends data to p5.js. It does not control any external outputs like LEDs or motors.

  • Communication with p5.js:

    • Sends a single-digit number (0–7) each time a pad is touched.

    • No information is received from p5.js back to Arduino because the communication is one-way (Arduino → p5).

Example behavior:

  • Touchpad 0 is touched → Arduino sends 0 over Serial.

  • Touchpad 1 is touched → Arduino sends 1 over Serial.

    …and so on for each pad.

    The Arduino sketch will look something like this (early draft structure):

    void setup() {
  Serial.begin(9600);
  pinMode(touchPin1, INPUT);
  pinMode(touchPin2, INPUT);
  ...
}

void loop() {
  if (digitalRead(touchPin1) == HIGH) {
    Serial.println(0);
  }
  if (digitalRead(touchPin2) == HIGH) {
    Serial.println(1);
  }
  ...
}

p5.js Program Design

The p5.js program will be responsible for receiving data from Arduino and producing both sound and visual feedback based on the input.

  • Receiving data:

    • p5.js listens to the Serial port using the serialEvent() function.

    • Each time it receives a number from Arduino (0–7), it triggers two actions:

      1. Visual: Activates a specific 3D box animation (e.g., spin, color change, movement).

      2. Audio: Plays a specific musical note using the p5.sound library.

  • Processing:

    • The incoming serial value is matched to the corresponding element in an array of 3D boxes.

    • A corresponding musical note (e.g., C2, D2, E2, F2…) is also mapped and played.

  • Outputs:

    • On the screen: Real-time 3D animations using WEBGL (such as spinning boxes).

    • In the browser: Sound output through the computer speakers using p5.sound.

  • Communication with Arduino:

    • p5.js only receives data from Arduino.

    • No messages are sent back

Example behavior:

  • Receive 0 → Box 0 spins and note C2 plays

  • Receive 1 → Box 1 spins and note D2 plays

  • Receive 2 → Box 2 spins and note E2 plays

    Early pseudocode for handling serial data:

function serialEvent() {
  let inData = Number(serial.read());
  if (inData >= 0 && inData < boxes.length) {
    boxes[inData].play(); // Spin box
    playNote(inData); // Play corresponding sound
  }
}

Next Steps

  1. Build a basic cardboard and tinfoil prototype for the touch-sensitive interface

  2. Test Arduino with simple touch sensor code, confirming that touchpads reliably trigger digital input signals

  3. Set up p5.serialport and confirm that Arduino can send and p5.js can receive serial data properly

  4. Prepare an initial sound mapping using the p5.sound library

  5. Create a rough version of the 3D visualizations (boxes spinning in response to input)

Leave a Reply