Author: Jayden Akpalu

Finalised Concept

For my final project, I will design and build an interactive penalty-kick football game that combines p5.js, Arduino, and a custom physical controller. My goal is to create a fun, responsive experience where players physically control the shooter’s movement and the direction of their kick using a real-world device. At the same time, the screen will display a dynamic game environment – complete with a goalkeeper, goalpost, and shooting animations.

This project will allow me to explore hardware-software integration, serial communication, real-time input handling, and user-centered interface design.

Gameplay

Player one will use a custom-built controller, which I will design using laser-cut casing and embed an Arduino joystick. The joystick will allow the player to select the ball’s direction – left, center, or right. Player one will also have a push button to initiate the kick toward the selected direction. The player sprite will move toward the ball when the button is pressed, and then the ball will move toward the chosen direction while the shooter remains in position.

Player two will control the goalkeeper using hand movements detected via ML5.js. By moving their hands left, right, or center in front of a camera, the player will slide the goalkeeper sprite across the goal to attempt a save. The game will simulate a real penalty shootout in a simplified and engaging way.

Arduino to p5

• The joystick and push button inputs from the custom controller will be read by Arduino and sent to p5.js via serial.

• These inputs will determine the direction of the ball and when the kick is initiated.

p5 to Arduino

• After the ball reaches the goal or the goalkeeper makes a save, p5.js will send feedback to Arduino.

• If a goal is scored, a green LED connected to the Arduino will turn on.

• If the goalkeeper saves the ball, a red LED will turn on.

• This provides tactile and visual feedback, creating a more immersive experience for the players.

Progress So Far

So far, I have focused on setting up the digital environment and assets for the game. I have:

1. Prepared the visual assets

   • Loaded images for the football pitch, shooter, goalkeeper, ball, and goalpost into p5.js.

   • Scaled the images dynamically based on the window size to ensure the game will be responsive in full-screen mode.

   • Adjusted the positions of the shooter, ball, goalkeeper, and goalpost so that the gameplay layout is accurate and visually balanced.

2. Implemented basic classes in p5.js

   • Created classes for the pitch, shooter, ball, goalkeeper, and goalpost.

   • Ensured each class has a display() method to render its sprite correctly on the canvas.

   • Tested the scaling logic and made the goalpost larger for a better gameplay area.

3. Set up fullscreen functionality

   • Pressing the F key now toggles full-screen mode.

   • All sprites and background images adjust automatically to fit the screen size.

4. Planned the shooting logic

   • Decided that the shooter will move toward the ball when the push button is pressed.

   • Once the shooter reaches the ball, the ball will move toward the selected direction (left, center, or right).

   • The shooter itself will remain in position while the ball moves.

5. Laid groundwork for Arduino communication

   • Decided on bidirectional communication: joystick and push button inputs from Arduino will control the ball direction, and LEDs will provide visual feedback for goals and saves.

   • Begun planning the serial communication structure between Arduino and p5.js.

Next steps:

• Build and test the custom controller once I receive the Arduino joystick.

• Implement keyboard-controlled ball movement to simulate the shooter action before integrating the controller.

• Add hand-detection logic with ml5.js to control the goalkeeper.

• Integrate LED feedback for scoring and saves.

For my final project, I want to create a motion-responsive interactive installation. The idea is that a user’s movements will be sensed by Arduino sensors, like an ultrasonic sensor, and sent to p5.js, which will process the input and generate dynamic visuals. I haven’t yet decided on the exact type of art to display; it could be abstract particle systems, animated images, or other digital creations. However, the key is that the visuals will respond in real-time to the user’s motion, making the experience immersive and engaging.

A recent trip I made to teamLab Phenomena, where I experienced installations that react to human gestures, is the inspiration behind this project. For example, moving through a space could cause flowers to bloom digitally or change the colors of projected visuals. I want to recreate that sense of wonder and interaction in my own project, where the user feels connected to the digital environment.

This project will be bidirectional.

• Arduino → p5.js: The user’s gestures are sensed by Arduino and drive the visuals, controlling motion, shape, size, or color of the on-screen elements.

• p5.js → Arduino: When the visuals reach certain thresholds, for instance, a cluster of particles grows large, or an animated image reaches a boundary, p5.js will send meaningful signals back to Arduino. These signals could trigger LEDs or small motors to give tactile feedback, making the interaction feel alive and responsive. This way, the user sees the effect on screen and feels it physically, creating a continuous loop of action and response.

Before reading Graham Pullin’s Design Meets Disability, I thought about assistive technology in a pretty simple way: the better the design, the more it should be invisible. Like a lot of people, I assumed that the goal for things like hearing aids or prosthetic limbs was for them to blend in, so users could just “look normal” and not stand out.

Pullin completely changed the way I think. He points out that trying to hide a device can actually make people feel like disability is something to be ashamed of. Instead, designers should think about fashion, creativity, and self-expression.

The example he gives with eyeglasses really hit me. Glasses used to be seen as a weakness, but now they’re a fashion statement – people even wear frames without lenses! That shift happened because designers stopped trying to hide them and started making them bold and stylish. I realized it’s not visibility that’s the problem, but the lack of thoughtful, positive design.

I really agree with Pullin that we should stop seeing users as “patients” who need fixing, and start seeing them as “wearers” who can express themselves. Athletes like Aimee Mullins show this perfectly, treating her prosthetic legs as works of art or fashion. She’s not just adapting to a missing limb-she’s making a confident personal choice through design. This made me see that style can empower, rather than hide, disability.

Exercise 1: Arduino to p5.js  – Potentiometer

Concept

This project shows how a physical sensor can control a digital object. A potentiometer on the Arduino sends its values to p5.js . In the sketch, these values move an ellipse left or right on the screen. Turning the knob changes the ellipse’s position instantly.

Arduino Code

The Arduino reads the potentiometer value and sends it over serial:

int interval = 100; // milliseconds between readings

void setup() {
  Serial.begin(9600); // start serial communication
}

void loop() {
  int potValue = analogRead(A1); // read potentiometer
  int mappedValue = map(potValue, 0, 1023, 0, 255); // scale value
  Serial.println(mappedValue); // send value to p5.js
  delay(interval); // avoid flooding serial
}

p5.js Code

The p5.js sketch reads the serial data and moves the ellipse:

let positionX = 0;
let port;
let sensorValue = 0;

function setup() {
  createCanvas(400, 400);
  positionX = width / 2;

  port = createSerial();
  let used = usedSerialPorts();
  if (used.length > 0) port.open(used[0], 9600);
}

function draw() {
  background(220);

  let line = port.readUntil("\n");
  if (line.length > 0) {
    sensorValue = int(line.trim());
  }

  positionX = map(sensorValue, 0, 255, 0, width);

  ellipse(positionX, height / 2, 50, 50);
}

Video of the final result: Video

Exercise 2: p5.js-to-Arduino LED Control

Concept

This project demonstrates how a digital signal from p5.js can control hardware on Arduino. Using a p5.js sketch, we can send commands over serial to turn an LED on or off.

Arduino Code

The Arduino listens for serial commands from p5.js and controls the LED accordingly:

The Arduino listens for serial commands from p5.js and controls the LED accordingly:
int ledPin = 13; // onboard LED

void setup() {
  pinMode(ledPin, OUTPUT);
  Serial.begin(9600); // start serial communication
}

void loop() {
  if (Serial.available() > 0) {
    String command = Serial.readStringUntil('\n'); // read command
    command.trim(); // remove whitespace

    if (command == "ON") {
      digitalWrite(ledPin, HIGH); // turn LED on
    } 
    else if (command == "OFF") {
      digitalWrite(ledPin, LOW); // turn LED off
    }
  }
}

p5.js Code

The p5.js sketch sends commands to the Arduino based on events:

let port;
let connectBtn;

function setup() {
  createCanvas(400, 400);
  background(220);

  // initialize serial connection
  port = createSerial();
  let used = usedSerialPorts();
  if (used.length > 0) port.open(used[0], 9600);

  connectBtn = createButton("Connect");
  connectBtn.position(10, 10);
  connectBtn.mousePressed(toggleConnection);
}

function draw() {
  background(220);
}

function keyPressed() {
  if (port.opened()) {
    if (key === 'L') {         // press L to turn LED on
      port.write("ON\n");
    } else if (key === 'K') {  // press K to turn LED off
      port.write("OFF\n");
    }
  }
}

Video of the final result: Video

Exercise 3: Bidirectional: Controlling LED with Bouncing Ball and Controlling Wind with Potentiometer

Concept

This project demonstrates bidirectional communication between Arduino and p5.js, combining physics simulation with hardware interaction. In this, A ball bounces on the p5.js canvas, following simple gravity physics. The horizontal wind affecting the ball is controlled by a potentiometer connected to the Arduino. Turning the knob changes the wind force in real-time. Every time the ball hits the floor, p5.js sends a “BOUNCE” signal to Arduino, lighting up an LED. When the ball stops bouncing, p5.js sends a “STOP” signal, turning the LED off.

Arduino Code

Arduino reads the potentiometer and controls the LED based on serial commands:

int ledPin = 13;   // LED pin
int potPin = A1;   // potentiometer pin
int potValue = 0;

void setup() {
  pinMode(ledPin, OUTPUT);
  Serial.begin(9600);
}

void loop() {
  // Read potentiometer value
  potValue = analogRead(potPin);
  Serial.println(potValue);  // send to p5.js

  // Check for incoming serial commands
  if (Serial.available() > 0) {
    String command = Serial.readStringUntil('\n');
    command.trim();

    if (command == "BOUNCE") {
      digitalWrite(ledPin, HIGH);   // light LED on bounce
    } else if (command == "STOP") {
      digitalWrite(ledPin, LOW);    // turn LED off when ball stops
    }
  }

  delay(50); // small delay for stability
}

p5.js Code

The p5.js sketch simulates a bouncing ball and sends commands to Arduino:

let velocity, gravity, position, acceleration, wind;
let mass = 50;
let drag = 0.99;

// SERIAL
let port;
let sensorValue = 0;

function setup() {
  createCanvas(640, 360);
  noFill();

  // physics
  position = createVector(width/2, 0);
  velocity = createVector(0, 0);
  acceleration = createVector(0, 0);
  gravity = createVector(0, 0.5 * mass);
  wind = createVector(0, 0);

  // serial
  port = createSerial();
  let used = usedSerialPorts();
  if (used.length > 0) port.open(used[0], 9600);
}

function draw() {
  background(255);

  // read potentiometer from Arduino
  let line = port.readUntil("\n");
  if (line.length > 0) sensorValue = int(line.trim());
  wind.x = map(sensorValue, 0, 1023, -1, 1);

  // apply forces
  applyForce(wind);
  applyForce(gravity);

  velocity.add(acceleration);
  velocity.mult(drag);
  position.add(velocity);
  acceleration.mult(0);

  ellipse(position.x, position.y, mass, mass);

  // bounce detection
  if (position.y > height - mass/2) {
    position.y = height - mass/2;

    if (abs(velocity.y) > 1) {
      velocity.y *= -0.9;
      if (port.opened()) port.write("BOUNCE\n"); // LED on
    } else {
      velocity.y = 0;
      if (port.opened()) port.write("STOP\n");   // LED off
    }
  }
}

function applyForce(force) {
  let f = p5.Vector.div(force, mass);
  acceleration.add(f);
}

Video of the final result: Video

Challenges and Improvements

In all three projects, the hardest part was making sure the Arduino and p5.js talked to each other correctly. Sometimes the data from the potentiometer didn’t come through clearly, which made the ellipse or bouncing ball move strangely or the LED turn on and off at the wrong time. It was also tricky to scale the sensor values so they controlled the visuals in a smooth way. In the bouncing ball project, making the LED light only when the ball bounced was challenging because the ball slowed down naturally. To improve, I could smooth the sensor readings to make movement less jumpy, make the ball bounce more realistically, or add more sensors or LEDs for extra interaction. These changes would make the projects work better and feel more interactive.

Ref:  In the last exercise, ChatGPT helped us by explaining and guiding the use of p5.Vector, physics calculations, and serial communication, which made Implementation easier.

Concept:

This week, Richmond and I  made a small musical instrument inspired by the piano. We used three switches to play notes.  We also added a potentiometer that can change the pitch of the notes while playing. This means the player can make higher or lower sounds with the same switches. Using the switches and potentiometer together makes the instrument more interactive and fun, giving the player control over both the notes and their frequency.

Video Demonstration:

https://drive.google.com/file/d/1tpGMX4FXWm4Y2IJe1WP9MxxI7M97yWc0/view?usp=sharing

Schematic:

Code Highlight:

// Mini Piano: 3 switches, Piezo buzzer, optional pitch adjustment

const int buzzerPin = 8;      // Piezo buzzer
const int switch1 = 2;        // Key 1 (C)
const int switch2 = 3;        // Key 2 (E)
const int switch3 = 4;        // Key 3 (G)
const int potPin = A0;        // Optional: pitch adjust

// Base frequencies for the notes (Hz)
int note1 = 262;  // C4
int note2 = 330;  // E4
int note3 = 392;  // G4

void setup() {
  pinMode(buzzerPin, OUTPUT);
  pinMode(switch1, INPUT);
  pinMode(switch2, INPUT);
  pinMode(switch3, INPUT);
  Serial.begin(9600); // optional for debugging
}

void loop() {
  // Read potentiometer to adjust pitch
  int potValue = analogRead(potPin);        // 0-1023
  float multiplier = map(potValue, 0, 1023, 80, 120) / 100.0;  // 0.8x to 1.2x

  bool anyKeyPressed = false;

  // Check switches and play corresponding notes
  if (digitalRead(switch1) == HIGH) {
    tone(buzzerPin, note1 * multiplier);
    anyKeyPressed = true;
  }

  if (digitalRead(switch2) == HIGH) {
    tone(buzzerPin, note2 * multiplier);
    anyKeyPressed = true;
  }

  if (digitalRead(switch3) == HIGH) {
    tone(buzzerPin, note3 * multiplier);
    anyKeyPressed = true;
  }

  // Stop sound if no switch is pressed
  if (!anyKeyPressed) {
    noTone(buzzerPin);
  }

  delay(10); // short delay for stability
}

Github Link:

https://github.com/JaydenAkpalu/Intro-to-IM/blob/a60e0ae114d3cd07e5e053a4b028d050f3606b30/Week10_Musical_Instrument.ino

Reflections & Future Improvements:

This week’s assignment felt highly interactive, building upon previous projects while introducing multiple input elements and user-controlled parameters. We learned how to combine both digital and analog inputs to create a responsive musical instrument. For future improvements, we would like to implement a more realistic note duration system, where each note fades out naturally after being played, similar to a real piano. Additionally, adding more switches and possibly multiple buzzers could allow for more complex melodies and chords, enhancing the expressive possibilities.

 

Concept:

For this project, I decided to use one digital sensor and one analog sensor to control two LEDs. For the digital part, I connected a push button to a digital pin to control one LED — pressing the button turns the light on, and releasing it turns it off. For the analog part, I used a potentiometer connected to an analog input to control the brightness of a second LED through an analog output. As I turned the knob, the LED smoothly adjusted its brightness, demonstrating how analog signals can vary continuously instead of just turning on or off. This setup reflects the difference between digital and analog control and how both can work together in an interactive circuit.

Video Demonstration:

https://drive.google.com/file/d/1_urM0vEA__Piz7zGG_TUHDnuhy5QYUgQ/view?usp=drive_link

Circuit Illustration:

Code Highlight:

// Define pin numbers
int ledPin1 = 11;   // LED controlled by potentiometer (analog LED)
int ledPin2 = 2;    // LED controlled by button (digital LED)
int buttonPin = 7;  // push button input

void setup() {
  // Set pin modes
  pinMode(ledPin1, OUTPUT);     // pin 11 used for PWM (brightness control)
  pinMode(ledPin2, OUTPUT);     // pin 2 for digital on/off LED
  pinMode(buttonPin, INPUT);    // button as input (external resistor required)
  
  Serial.begin(9600);           // start serial monitor for debugging
}

void loop() {
  // Read the potentiometer (analog sensor)
  int sensorValue = analogRead(A1);     // reads values from 0–1023
  Serial.println(sensorValue);          // print the reading to serial monitor

  // Control the brightness of LED on pin 11
  // analogWrite expects a range 0–255, so divide by 4 to scale down
  analogWrite(ledPin1, sensorValue / 4);
  delay(30);                            // small delay to stabilize readings

  // Read the push button (digital sensor)
  int buttonState = digitalRead(buttonPin);   // reads HIGH or LOW

  // Control the digital LED based on button state
  if (buttonState == HIGH) {     // if button is pressed (connected to 5V)
    digitalWrite(ledPin2, HIGH); // turn LED on
  }
  else {                         // if button is not pressed (LOW)
    digitalWrite(ledPin2, LOW);  // turn LED off
  }
}

Github Link:

https://github.com/JaydenAkpalu/Intro-to-IM/blob/f43175106ae171c88a5c7db0410a56dc965127af/Week9_AnalogAndDigitalSensor.ino

Reflections & Future Improvements:

Working on this project helped me really understand the difference between digital and analog sensors and how they can control things like LEDs. I saw that a digital sensor, like a push button, only has two states — on or off — which makes it easy to control an LED directly. On the other hand, an analog sensor, like a potentiometer, can give a range of values, letting me gradually adjust the brightness of an LED. It was really satisfying to see how these two types of input behave so differently in a circuit.

If I were to improve the project, I’d try adding more interactive elements. For example, I could use multiple buttons or potentiometers to control more LEDs, or even combine the potentiometer with a light sensor so the LED responds to changes in the environment automatically. I’d also like to experiment with different LED effects, like fading, blinking patterns, or color changes, to make it more dynamic and fun. Overall, this project gave me a better understanding of how simple inputs can be used creatively to make interactive circuits.

Reading Tom Igoe’s “Making Interactive Art: Set the Stage, Then Shut Up and Listen” made me rethink the way I think about creative work. What stood out to me most was his idea that artists shouldn’t script how people experience their work. I found it interesting when he said that by explaining too much, artists end up telling the audience what to think and how to act. That made a lot of sense: if the whole point of interactive art is to create a conversation, then the artist needs to give space for the audience to respond. I liked how he compared it to being a director: you can guide your actors, but you can’t tell them exactly how to feel. The same goes for an audience—you set the stage, then step back.

I also appreciated how he described interactive art as something that lives through the audience’s actions. The idea that people complete the artwork through their participation feels powerful. It reminds me that design isn’t just about control; it’s about trust—trusting that others will find meaning on their own. This reading encouraged me to think less about explaining and more about creating experiences that speak for themselves.

Reading Tom Igoe’s Physical Computing’s Greatest Hits (and Misses) really opened my eyes to how creative physical computing can be. I liked how he talked about different project themes that keep coming up, like theremin-style instruments, gloves, or meditation helpers, but still manage to feel fresh every time. What stood out to me was his point that even if an idea has been done before, there’s always room for originality. It’s not about being the first person to come up with something, but about how you make it your own. That made me see creativity in a new way; it’s more about exploring, experimenting, and putting a bit of yourself into the project.

I also liked how Igoe focused on the human side of physical computing. He reminds readers that the goal isn’t just to build machines that move or light up, but to design interactions that mean something. It made me realize that technology can be emotional and expressive, not just functional. Overall, the reading made me appreciate how physical computing connects people and ideas, and how much space there is to create something personal, even within familiar themes.

Concept:

For this week’s assignment, I made a foot-activated switch that turns on an LED when you step on it. It uses two pieces of aluminum foil that touch when pressed together, completing the circuit and lighting the LED.

The idea was to make a basic, hands-free switch that works using pressure from your foot.

Video demonstration:

https://drive.google.com/file/d/1tqjnfQByBRSJJjC8D7xZhDbTFqgD4DTf/view?usp=drive_link

Code Highlight:

const int ledPin = 13;     // LED pin
const int switchPin = A2;  // Foil switch pin

void setup() {
  pinMode(ledPin, OUTPUT);
  pinMode(switchPin, INPUT);
}

void loop() {
  int switchState = digitalRead(switchPin);

  if (switchState == HIGH) {
    digitalWrite(ledPin, HIGH);   // Turn LED on
  } else {
    digitalWrite(ledPin, LOW);    // Turn LED off
  }
}

Github link:

https://github.com/JaydenAkpalu/Intro-to-IM/blob/4dfa5eeecae690e073a9bfd35f4140274d87f5f1/Week8_UnusualSwitchAssignment.ino

Reflections & Future Improvements:

Building the foot-activated switch helped me understand how a basic circuit can turn a simple physical action into an electronic signal. I learned how to use digital inputs on the Arduino and how completing a circuit with something as basic as foil can act as a switch.

If I were to improve it, I’d try to make the foil connection stronger and more reliable, since it doesn’t always press evenly. I’d also like to make the setup look cleaner and maybe add another component, like a buzzer or a second LED, to make it more interactive.