Reflecting on the author’s critique of what is dubbed “Pictures Under Glass,” I find a compelling challenge to our current acceptance of interface norms. The disparity highlighted between the tactile richness of our everyday physical interactions and the simplified, touch-based interfaces that dominate our devices urges us to question whether we are curbing our innovative potential. By adhering so strictly to touchscreen technology, we risk neglecting developments in interfaces that could better harness human physicality and sensory feedback.
This critique serves as a springboard for considering alternative interaction paradigms that prioritize enhancing our sensory and physical engagement with technology. While touchscreens undoubtedly provide accessibility and ease, there’s a significant opportunity to explore more immersive and intuitive interfaces. Such technologies would not only extend our capabilities but also deepen our interaction with the digital world, suggesting a future where technology complements rather than constrains our interactions with our environment. This shift could pave the way for more meaningful and natural user experiences that align closely with human behavior and physicality.
The project’s primary goal is to design and construct a custom joystick game controller using Arduino. This controller will enhance player interaction with a browser-based video game developed in p5.js, offering a more intuitive and engaging gameplay experience compared to standard keyboard or mouse controls.
Controller Features:
Joystick for Directional Control:
The joystick will allow players to control movement within the game with greater precision and fluidity. It will capture analog input to provide a smoother response compared to digital keys.
Tactile Feedback Buttons:
Additional buttons will be incorporated for game-specific actions (e.g., jumping, shooting). These will use tactile switches to give a satisfying click feel, ensuring immediate and comfortable response.
Vibration Feedback:
Incorporate vibration motors that activate during specific game events, such as collisions or other significant interactions, to provide physical feedback that enhances the immersion.
LED Feedback System:
LEDs will be programmed to react to different situations in the game, such as game alerts, low health, or achievement notifications, adding another layer of feedback and interaction.
Technical Components:
Arduino Board: Acts as the central processing unit for the joystick controller, reading inputs from the joystick and buttons, and managing outputs like LED indicators and vibration motors.
Analog Joystick Module: Captures detailed directional inputs from the player, translating them into game movements.
Vibration Motors: To provide immediate tactile feedback during critical game events, enhancing the physical experience of the gameplay
The concept of this project was to create a mini drum pad, or what is equivalent to one, with the hardware we have available. The device would use buttons to trigger different buzzer sounds, mimicking the functionality of a traditional drum pad. Each button on the device would correspond to a different sound, with the frequency of these sounds adjustable via a potentiometer. This allows the user to modify the pitch of the tones.
Code:
// Defining pins assignments for buttons and buzzers
const int buttonPin1 = 2;
const int buttonPin2 = 3;
const int buttonPin3 = 4;
// Coded with the Aid of ChatGPT
const int buttonPin4 = 5; // Monitoring and playbacks button
// Coded with the Aid of ChatGPT
const int buzzerPin1 = 8;
const int buzzerPin2 = 9;
const int buzzerPin3 = 10;
const int potPin = A0; // Potentiometer connected to A0 for frequency control
// Variables to manage button states and debounce timing
int buttonState1 = 0;
int lastButtonState1 = 0;
int buttonState2 = 0;
int lastButtonState2 = 0;
int buttonState3 = 0;
int lastButtonState3 = 0;
int buttonState4 = 0;
int lastButtonState4 = 0;
unsigned long lastDebounceTime1 = 0;
unsigned long lastDebounceTime2 = 0;
unsigned long lastDebounceTime3 = 0;
unsigned long lastDebounceTime4 = 0;
unsigned long debounceDelay = 50; // Debounce delay in milliseconds
// Struct to hold buzzer activation data including the pin and frequency
struct BuzzerAction {
int buzzerPin;
int frequency;
};
// Coded with the Aid of ChatGPT
BuzzerAction record[100]; // Array to store each buzzer activation
int recordIndex = 0; // Index for recording array
//Coded with the Aid of ChatGPT
void setup() {
// Initialize all button and buzzer pins
pinMode(buttonPin1, INPUT);
pinMode(buttonPin2, INPUT);
pinMode(buttonPin3, INPUT);
// Coded with the Aid of ChatGPT
pinMode(buttonPin4, INPUT);
// Coded with the Aid of ChatGPT
pinMode(buzzerPin1, OUTPUT);
pinMode(buzzerPin2, OUTPUT);
pinMode(buzzerPin3, OUTPUT);
pinMode(potPin, INPUT); // Setups potentiometer pin as input
}
void loop() {
// Reads current state of buttons
int reading1 = digitalRead(buttonPin1);
int reading2 = digitalRead(buttonPin2);
int reading3 = digitalRead(buttonPin3);
// Coded with the Aid of ChatGPT
int reading4 = digitalRead(buttonPin4);
// Coded with the Aid of ChatGPT
int potValue = analogRead(potPin); // Reads potentiometer value
int frequency = map(potValue, 0, 1023, 200, 2000); // Maps potentiometer value to frequency range
// Handle button 1 press and recording
debounceAndRecord(reading1, &lastButtonState1, &buttonState1, &lastDebounceTime1, buzzerPin1, frequency);
// Handle button 2 press and recording
debounceAndRecord(reading2, &lastButtonState2, &buttonState2, &lastDebounceTime2, buzzerPin2, frequency);
// Handle button 3 press and recording
debounceAndRecord(reading3, &lastButtonState3, &buttonState3, &lastDebounceTime3, buzzerPin3, frequency);
// Handles button 4 for playback
if (reading4 != lastButtonState4) {
lastDebounceTime4 = millis();
}
if ((millis() - lastDebounceTime4) > debounceDelay) {
if (reading4 != buttonState4) {
buttonState4 = reading4;
if (buttonState4 == HIGH) {
for (int i = 0; i < recordIndex; i++) {
// Play each recorded buzzer action with the specific frequency recorded
tone(record[i].buzzerPin, record[i].frequency, 200);
delay(250); // Short delay between each buzzer action for clarity
}
recordIndex = 0; // Resets record index after playback
}
}
}
// Update last button states for next loop iteration
lastButtonState1 = reading1;
lastButtonState2 = reading2;
lastButtonState3 = reading3;
lastButtonState4 = reading4;
}
// Coded with the Aid of ChatGPT
// Function to handle button debouncing and recording buzzer actions
void debounceAndRecord(int reading, int *lastButtonState, int *buttonState, unsigned long *lastDebounceTime, int buzzerPin, int frequency) {
if (reading != *lastButtonState) {
*lastDebounceTime = millis(); // Reset debounce timer
}
if ((millis() - *lastDebounceTime) > debounceDelay) {
if (reading != *buttonState) {
*buttonState = reading; // Updates button state
if (*buttonState == HIGH && recordIndex < sizeof(record) / sizeof(record[0])) {
record[recordIndex++] = {buzzerPin, frequency}; // Records the buzzer activation
tone(buzzerPin, frequency, 200); // Plays buzzer at recorded frequency
}
}
}
*lastButtonState = reading; // Updates last button state for debouncing
// Coded with the Aid of ChatGPT
}
Hardware Configuration: The system is designed with four button inputs and three buzzer outputs. Additionally, a potentiometer is used to control the frequency of the buzzer sounds.
Button Functionality: Buttons 1 to 3 are connected to buzzers and are responsible for triggering sounds with variable frequencies determined by the potentiometer. Button 4 is designated for playback. It plays back a sequence of sounds that have been recorded based on earlier interactions with buttons 1 to 3.
Frequency Control: The frequency of the sounds is dynamically adjusted using a potentiometer. The analog value from the potentiometer is mapped to a specified frequency range (200 Hz to 2000 Hz), which determines how the buzzers sound.
Debouncing: To ensure reliable button press detection without noise interference, the code implements debouncing logic. This involves measuring the time since the last button state change and updating the state only if this interval exceeds a predefined threshold (50 milliseconds).
Recording and Playback (Aided by Chatgpt)
Recording: When a button (1 to 3) is pressed, the action (which buzzer is activated and at what frequency) is recorded in an array. This includes storing both the pin of the buzzer and the frequency at which it was activated.
Playback: When button 4 is pressed, the system iterates over the recorded actions and plays them sequentially. Each action triggers the corresponding buzzer to sound at the recorded frequency for a short duration.
Loop and Functions : The main loop continuously checks the state of each button and the potentiometer, updating the frequency accordingly. A helper function, debounceAndRecord, is used to handle the logic for both debouncing and recording the buzzer actions associated with each button press.
Video of Project:
Reflection and ideas for future work or improvements:
Integrating a small display screen would significantly improve its functionality, further enhancing the project. This screen would provide real-time visual feedback on button presses and frequency outputs, allow users to scroll through and select different sounds or presets, and serve as a simple interface for directly programming the device. The potential for further development and refinement holds exciting prospects. The integration of a display screen and the addition of more customizable buttons are immediate steps that will enhance the device’s usability and versatility. Further innovations could include wireless connectivity for easy integration with other music production software or the addition of sensors
to enable gesture-based controls, which would offer an even more dynamic and immersive user experience. Several key insights stand out after reflecting on what this project has taught us. First, the practical challenges of hardware interfacing taught us the importance of robust design and a
solid plan for creating it. There is also a need for effective wire management and physical housing to enhance device durability and aesthetics.
The Brief Rant is a thought out argument for changing our perspective towards our current approach to interaction design and calling the need for other mediums. It focuses on the limits of our current technology and the implications it has on us.
The analogy between kid’s book depriving adults of the complexity of english vocab same goes to touchscreens to restricting interaction design and our capabilities.
In considering another real-world example, my mind goes to VR and AR. Although these technologies have advanced considerably in the past couple of years, there’s still lots of room for growth in terms of interaction design. Picture this: VR systems could undergo enhancement with haptic feedback technology, where it would enbale more immersion.
The other reading also highlights how adeptly we use our hands in everyday tasks, like opening jars and making sandwiches, contrasting this natural interaction with the detached experience of using flat screens. It questions why we settle for interfaces that don’t harness the richness of human touch and manipulation. The author is calling for a interactive design movement that I do see coming in the next couple years hopefully!
For this week’s assignment Sarah and I created a DJ set! The set is built using two potentiometers, a digital switch, a piezo buzzer, servo motors, and two LEDs (for visual feedback). When connected to power, the set produces a melody that the player (DJ) can manipulate by using the first potentiometer (which controls the pitch) and the second potentiometer (which controls the tempo). The red LED blinks in sync with the tempo set by the second potentiometer. The yellow LED’s brightness is proportional to the pitch (the higher the pitch, the higher the brightness). The servo motor resembles the turntables on an actual DJ set. You can activate it by pressing the digital switch. Upon activation, the servo motor will rotate at a speed equivalent to the beat’s tempo.
Implementation
Effectively, the DJ set plays a sequence of notes in succession, stored in a global 2D array, whose pitch class and the tempo of the beat are controlled by the potentiometers. In switching successively between tempos and pitch classes, the player generates different tunes and overall musical moods. The player can also switch on the servo motor, whose position incrementally increases at the same rate set by the tempo the player chose. We utilize two Arduinos, one controlling the sound manipulation between different tempos and pitches using the potentiometers (as well as the LED) and the other controlling the servo motor via a switch. The first Arduino sender program sends the position of the servo motor at time intervals corresponding to the current rate set by the tempo, using the Inter-Integrated Circuit (I2C) Protocol, over to the second Arduino. The receiver program on the second Arduino receives the position and updates the location of the servo, conditional on the button state being on. As the first receiver program also needed to synchronize sound with the blinking of the LEDs, it was essential to extrapolate the concept of blinking without delay to the playing of tones by the buzzer.
Code Snippets
The sender Arduino program reads the two potentiometer values, using the first value to control the pitch (by mapping the value to the range 0-3, corresponding to the range of sequences in the global notes array ) and the second to control the tempo/rate of playing a note. The first value is used to index the 2D array, notes[][], which stores the sequence of notes in different pitches. It also sets the brightness of the yellow LED. The second is used to index the duration[] array, which specifies the rate at which notes are played. The notes in the array notes[potVal1] are played in succession, with the next note in the sequence playing if the specified duration has passed. The state of the red LED is updated to blink according to the rate of the current melody being played. Finally, the position of the servo motor is also updated and sent to the second Arduino program.
note++; // cycle through the sequence
if (note >= 5) {
note = 0; // Reset note index
}
// direction goes forward if position is at 0
if (position <= 0){
servoDir = 1;
}
// direction goes backward if position is at 160
else if (position >= 160){
servoDir = -1;
}
position = (position+servoDir*20)%180;
Wire.beginTransmission(8); // begin transmission to receiver
Wire.write(position); // send over the positon of the servo motor
Wire.endTransmission(); // stop transmitting
Wire.requestFrom(8, 6); // request 6 bytes from receiver device #8
delay(1);
}
}
The receiver Arduino program receives the position from the sender and updates the position if the switch state is set to 1 (the switch has been pressed to turn on the motor). Since the transmission occurs at a rate equivalent to the rate of the music, the motor will move according to the current tempo of the beat.
#include
#include
Servo servo; // initialize servo
int x = 0; // variable storing data transmission
int position = 0; // position of the servo motor
const int switchPin = 7; // switch pin
int buttonState = 0; // current button state
int prevButtonState=0; // previous button state
bool servoMove = false; // keeps track of the switch state
void setup() {
Serial.begin(9600);
pinMode(switchPin, INPUT);
servo.attach(9);
Wire.begin(8);
Wire.onReceive(receiveEvent); // initialize event triggered on receipt of data
Wire.onRequest(requestEvent); // initialize event on being requested data
}
void receiveEvent(int bytes) {
x = Wire.read();
// validate received data
if (x >= 0 && x <= 180) {
position = x; // x directly maps to servo position
}
}
void requestEvent()
{
Wire.write("Hello ");
}
void loop() {
buttonState = digitalRead(switchPin);
// maintain the state of the switch and use to determine if the motor should move
if (buttonState == HIGH && prevButtonState == LOW){
servoMove = !servoMove;
};
// smoothly move the servo towards the desired position
if (servoMove){
if (position != servo.read()) {
if (position > servo.read()) {
servo.write(servo.read() + 1);
} else {
servo.write(servo.read() - 1);
}
delay(1);
};
}
prevButtonState = buttonState;
}
Circuit Schematic
Here’s a schematic of our circuitry: Have a look:
Reflections and Challenges
One of the things we struggled with, largely due to both of us lacking knowledge of musical compositions, is choosing the right sequence of tones that generate a coherent melody. With a combination of trial and error and some research, we found a suitable sequence. We also faced the challenge of one of our LED lights not turning on when we wired the servo motor to the same circuit. Instead of adding a second power source to connect the servo motor to, we opted to utilize I2C since we had an additional Arduino, which proved to be a useful exercise. Overall, we were happy with the final product, but I think it would be nice to extend this project further and give the player a little more creative control as the current setup is quite restrictive in terms of what final melodies the player can generate. For instance, we can have additional buzzers, each producing a different tone, controlled by switches that the users can use to make their own tunes from scratch over a set beat (something like a MIDI controller? ) .
For my final project, I am thinking of making a game where players are presented with a series of electronic puzzles that increase in complexity.
Each puzzle requires players to assemble components on a breadboard within a set time limit. The game provides immediate feedback on their solutions, and players earn points based on speed and accuracy.
Arduino
Breadboard where players will plug in various electronic components like resistors, wires, buzzers, or other sensors.
I would require some kind of sensors that would verify whether components are correctly placed and circuits are properly completed on the breadboard.
If Possible
Optionally, include buttons or switches on the board for starting the timer, resetting the puzzle, or requesting hints.
Processing
Processing will display each puzzle diagram on a screen, track the time remaining, and handle the transition between different puzzles.
Immediately inform players whether the circuit is correct. If incorrect, highlight errors or provide hints, depending on the game mode.
A scoring algorithm based on the complexity of the puzzle, the accuracy of the completed circuit, and the speed of completion.
Replicating the piano playing mat that we all used to play as children was the idea behind this assignment. Even those who are not musically inclined are drawn to play with the piano mat again because of how logically it was created. Why not make something that makes us all feel nostalgic, rather than just one of us? You’ll be overwhelmed with happiness as you play about with our design and hopefully be able to relive all your childhood memories in a way. This initiative will facilitate audience engagement and provide a means of connection to one another.
The code is for an Arduino setup that uses an ultrasonic sensor to measure distance and a force-sensitive resistor to detect pressure. It plays different musical notes based on the distance measured by the sensor, but only if a certain pressure threshold is exceeded on the resistor. If the measured distance is out of a specified range or the pressure is too low, no sound is played. The system uses this setup to create an interactive musical device where sound changes with distance and pressure.
int trig = 10;
int echo = 11;
long duration;
long distance;
int force;
void setup() {
pinMode(echo, INPUT);
pinMode(trig, OUTPUT);
Serial.begin(9600);
}
void loop() {
digitalWrite(trig, LOW); //triggers on/off and then reads data
delayMicroseconds(2);
digitalWrite(trig, HIGH);
delayMicroseconds(10);
digitalWrite(trig, LOW);
duration = pulseIn(echo, HIGH);
distance = (duration / 2) * .0344; //344 m/s = speed of sound. We're converting into cm
int notes[7] = {261, 294, 329, 349, 392, 440, 494}; //Putting several notes in an array
// mid C D E F G A B
force = analogRead(A0); //defining force as FSR data
if (distance < 0 || distance > 50 || force < 100) { //if not presed and not in front
noTone(12); //dont play music
}
else if ((force > 100)) { //if pressed
int sound = map(distance, 0, 50, 0, 6); //map distance to the array of notes
tone(12, notes[sound]); //call a certain note depending on distance
}
}
Reflections & Challenges:
One of the challenges we had while working on this project was the weakness of the sensors; if we had an ultrasonic sensor that measured the distance longer and a larger force detecting resistor, we would not have had any trouble playing notes. Therefore, we were forced to limit the user to playing notes (C-D-E-F-G-A) as a result of this.
Replicating the piano playing mat that we all used to play as children was the idea behind this assignment. Even those who are not musically inclined are drawn to play with the piano mat again because of how logically it was created. Why not make something that makes us all feel nostalgic, rather than just one of us? You’ll be overwhelmed with happiness as you play about with our design and hopefully be able to relive all your childhood memories in a way. This initiative will facilitate audience engagement and provide a means of connection to one another.
Code:
The code is for an Arduino setup that uses an ultrasonic sensor to measure distance and a force-sensitive resistor to detect pressure. It plays different musical notes based on the distance measured by the sensor, but only if a certain pressure threshold is exceeded on the resistor. If the measured distance is out of a specified range or the pressure is too low, no sound is played. The system uses this setup to create an interactive musical device where sound changes with distance and pressure.
int trig = 10;
int echo = 11;
long duration;
long distance;
int force;
void setup() {
pinMode(echo, INPUT);
pinMode(trig, OUTPUT);
Serial.begin(9600);
}
void loop() {
digitalWrite(trig, LOW); //triggers on/off and then reads data
delayMicroseconds(2);
digitalWrite(trig, HIGH);
delayMicroseconds(10);
digitalWrite(trig, LOW);
duration = pulseIn(echo, HIGH);
distance = (duration / 2) * .0344; //344 m/s = speed of sound. We're converting into cm
int notes[7] = {261, 294, 329, 349, 392, 440, 494}; //Putting several notes in an array
// mid C D E F G A B
force = analogRead(A0); //defining force as FSR data
if (distance < 0 || distance > 50 || force < 100) { //if not presed and not in front
noTone(12); //dont play music
}
else if ((force > 100)) { //if pressed
int sound = map(distance, 0, 50, 0, 6); //map distance to the array of notes
tone(12, notes[sound]); //call a certain note depending on distance
}
}
Reflections & Challenges:
One of the challenges we had while working on this project was the weakness of the sensors; if we had an ultrasonic sensor that measured the distance longer and a larger force detecting resistor, we would not have had any trouble playing notes. Therefore, we were forced to limit the user to playing notes (C-D-E-F-G-A) as a result of this.
Replicating the piano playing mat that we all used to play as children was the idea behind this assignment. Even those who are not musically inclined are drawn to play with the piano mat again because of how logically it was created. Why not make something that makes us all feel nostalgic, rather than just one of us? You’ll be overwhelmed with happiness as you play about with our design and hopefully be able to relive all your childhood memories in a way. This initiative will facilitate audience engagement and provide a means of connection to one another.
The code is for an Arduino setup that uses an ultrasonic sensor to measure distance and a force-sensitive resistor to detect pressure. It plays different musical notes based on the distance measured by the sensor, but only if a certain pressure threshold is exceeded on the resistor. If the measured distance is out of a specified range or the pressure is too low, no sound is played. The system uses this setup to create an interactive musical device where sound changes with distance and pressure.
int trig = 10;
int echo = 11;
long duration;
long distance;
int force;
void setup() {
pinMode(echo, INPUT);
pinMode(trig, OUTPUT);
Serial.begin(9600);
}
void loop() {
digitalWrite(trig, LOW); //triggers on/off and then reads data
delayMicroseconds(2);
digitalWrite(trig, HIGH);
delayMicroseconds(10);
digitalWrite(trig, LOW);
duration = pulseIn(echo, HIGH);
distance = (duration / 2) * .0344; //344 m/s = speed of sound. We're converting into cm
int notes[7] = {261, 294, 329, 349, 392, 440, 494}; //Putting several notes in an array
// mid C D E F G A B
force = analogRead(A0); //defining force as FSR data
if (distance < 0 || distance > 50 || force < 100) { //if not presed and not in front
noTone(12); //dont play music
}
else if ((force > 100)) { //if pressed
int sound = map(distance, 0, 50, 0, 6); //map distance to the array of notes
tone(12, notes[sound]); //call a certain note depending on distance
}
}
Reflections & Challenges:
One of the challenges we had while working on this project was the weakness of the sensors; if we had an ultrasonic sensor that measured the distance longer and a larger force detecting resistor, we would not have had any trouble playing notes. Therefore, we were forced to limit the user to playing notes (C-D-E-F-G-A) as a result of this.
Bret Victor’s “A Brief Rant on the Future of Interaction Design” emphasizes touchscreen technology. His critique resonates deeply with those of us concerned about the narrow trajectory of technological innovation where tactile and kinesthetic interaction is marginalized in favor of visually dominated interfaces. Victor’s call for broader sensory involvement in technological interfaces is a plea for innovation and an argument rooted in the natural human interaction with the world. Evidence supports his viewpoint from various fields, including educational psychology, which suggests that multi-sensory learning environments enhance understanding and retention (Wolfe and Nevills). This aligns with Victor’s advocacy for interfaces that engage more of our bodily senses, not less.
Victor’s reflections and the subsequent responses to his original piece stimulate a broader conversation about the potential biases in technology design. His critique may seem biased to those who champion digital minimalism and current devices’ sleek, streamlined aesthetics. However, it raises an essential question about whom technology is truly serving. Has reading his arguments changed my beliefs? Absolutely. It’s led me to reconsider the role of physicality in digital interaction and consider the untapped possibilities of interfaces that could mimic more complex human behaviors and interactions. This reflection opens up questions about the potential for future technologies: How far can we push the boundaries of interaction design to make digital experiences more immersive and intuitive without sacrificing functionality? How can designers balance the need for advanced functionality with intuitive physical interactions?
Citation:
Wolfe, P., & Nevills, P. (2004). Building the reading brain, PreK-3. Corwin Press.
