Week 13: Final Project – HandiSynth

Hey everyone! 👋

This is it, my final project!
(wellll… actually, my backup project :(, but that’s a story for another time)

 

Description: It’s a glove that synthesizes music, controlled by your hand (specifically, the curling of the fingers), with feedback being shown on both the glove itself (through the neopixels, aka addressable LED strips), and also on the screen (with a p5 interface).

 

So, how does it work?

As stated previously, the music is controlled by the curl of the fingers. This is detected with flex sensors attached on the glove, which is then fed into the Arduino, mapped from an auto-calibrated range to the control values, which then modify the sound in different ways (the 4 fingers control the base frequency (can be interpreted as pitch), tempo (speed), mix ratio, and multiplier, respectively). These values are also mapped and shown on the neopixels attached on top of the flex sensors, providing immediate visual feedback (and ngl, they also just look cool, but that shouldn’t be overdone). For example, the neopixel for the speed, actually changes its blink rate based on the speed. The auto-calibrated range values are also mapped into the range 0-1000 and sent to the p5 sketch, allowing it to alter the height of the bars representing each control, and also modify the center circle based on the values.

 

Arduino code:

 
// Configuring Mozzi's options
#include 
#define MOZZI_ANALOG_READ_RESOLUTION 10 // Not strictly necessary, as Mozzi will automatically use the default resolution of the hardware (eg. 10 for the Arduino Uno), but they recommend setting it (either here globally, or on each call)

#include 
#include  // oscillator
#include <tables/cos2048_int8.h> // table for Oscils to play
#include 

#include  // For the neopixels


// Flex sensor stuff

// Define flex sensor pins (these have to be analog)
const int FREQ_SENSOR_PIN = A0;
const int SPEED_SENSOR_PIN = A1;
const int MOD_SENSOR_PIN = A2;
const int RATIO_SENSOR_PIN = A3;

// Smoothening for each pin
Smooth smoothFreq(0.8f);
Smooth smoothSpeed(0.8f);
Smooth smoothSpeedLED(0.8f);
Smooth smoothMod(0.8f);
Smooth smoothRatio(0.8f);

// Input ranges for flex sensors (will be calibrated)
unsigned int freqInputMin = 1000; // Just FYI, the flex sensors in our setup roughly output in the range of ~ 200 - 650
unsigned int freqInputMax = 0;
unsigned int modInputMin = 1000;
unsigned int modInputMax = 0;
unsigned int speedInputMin = 1000;
unsigned int speedInputMax = 0;
unsigned int ratioInputMin = 1000;
unsigned int ratioInputMax = 0;


// Neopixel (addressable LED strip) stuff

// Define neopixel pins
const int FREQ_NEOPIXEL_PIN = 2;
const int SPEED_NEOPIXEL_PIN = 3;
const int MOD_NEOPIXEL_PIN = 4;
const int RATIO_NEOPIXEL_PIN = 5;

// Number of LEDs in each strip
const int NEOPIXEL_NUM_LEDS = 11;

// Define the array of leds
CRGB freqLEDs[NEOPIXEL_NUM_LEDS];
CRGB modLEDs[NEOPIXEL_NUM_LEDS];
CRGB speedLEDs[NEOPIXEL_NUM_LEDS];
CRGB ratioLEDs[NEOPIXEL_NUM_LEDS];


// Sound stuff

// desired carrier frequency max and min
const int MIN_CARRIER_FREQ = 22;
const int MAX_CARRIER_FREQ = 440;

// desired intensity max and min, inverted for reverse dynamics
const int MIN_INTENSITY = 10;
const int MAX_INTENSITY = 1000;

// desired modulation ratio max and min
const int MIN_MOD_RATIO = 5;
const int MAX_MOD_RATIO = 2;

// desired mod speed max and min, note they're inverted for reverse dynamics
const int MIN_MOD_SPEED = 10000;
const int MAX_MOD_SPEED = 1;

Oscil<COS2048_NUM_CELLS, MOZZI_AUDIO_RATE> aCarrier(COS2048_DATA);
Oscil<COS2048_NUM_CELLS, MOZZI_CONTROL_RATE> kIntensityMod(COS2048_DATA);
Oscil<COS2048_NUM_CELLS, MOZZI_AUDIO_RATE> aModulator(COS2048_DATA);

int mod_ratio; // harmonics
long fm_intensity; // carries control info from updateControl() to updateAudio()

// smoothing for intensity to remove clicks on transitions
float smoothness = 0.95f;
Smooth aSmoothIntensity(smoothness);

// To keep track of last time Serial data was sent, to only send it every x millis
int lastTimeSerialSent = 0;


void setup(){
  Serial.begin(9600); // For communicating with p5

  // Set the flex sensor pins
  pinMode( FREQ_SENSOR_PIN, INPUT_PULLUP);
  pinMode(  MOD_SENSOR_PIN, INPUT_PULLUP);
  pinMode(SPEED_SENSOR_PIN, INPUT_PULLUP);
  pinMode(RATIO_SENSOR_PIN, INPUT_PULLUP);

  // Setup the neopixels
	FastLED.addLeds<NEOPIXEL, FREQ_NEOPIXEL_PIN>(freqLEDs, NEOPIXEL_NUM_LEDS);
	FastLED.addLeds<NEOPIXEL, MOD_NEOPIXEL_PIN>(modLEDs, NEOPIXEL_NUM_LEDS);
	FastLED.addLeds<NEOPIXEL, SPEED_NEOPIXEL_PIN>(speedLEDs, NEOPIXEL_NUM_LEDS);
	FastLED.addLeds<NEOPIXEL, RATIO_NEOPIXEL_PIN>(ratioLEDs, NEOPIXEL_NUM_LEDS);
	FastLED.setBrightness(32); // 0 - 255

  // Feed/prime/initialise the smoothing function to get a stable output from the first read (to ensure the calibration isn't messed up). A value of 1630 was chosen by trial and error (divide and conquer), and seems to work best (at least for our setup)
  smoothFreq.next(1630);
  smoothMod.next(1630);
  smoothSpeed.next(1630);
  smoothRatio.next(1630);

  startMozzi();

  // Start the serial handshake
  while (Serial.available() <= 0) {
    digitalWrite(LED_BUILTIN, HIGH); // on/blink while waiting for serial data
    Serial.println("0"); // send a starting message
    delay(250); 
    digitalWrite(LED_BUILTIN, LOW);
    delay(250);
  }
}


// Basically our actual traditional loop in Mozzi (but still needs to kept reasonably lean and fast)
void updateControl(){

  // Read the smoothened freq
  int freqValue = smoothFreq.next(mozziAnalogRead(FREQ_SENSOR_PIN - 14)); // value is 0-1023, -14 since mozzi just takes a number (eg. 0 instead of A0), and the analog ones are 14 onwards

  // Calibrate the mapping if needed
  if (freqValue < freqInputMin) freqInputMin = freqInputMin * 0.5 + freqValue * 0.5; if (freqValue > freqInputMax) freqInputMax = freqInputMax * 0.5 + freqValue * 0.5;

  // Map the input to the carrier frequency
  int carrier_freq = map(freqValue, freqInputMin, freqInputMax, MIN_CARRIER_FREQ, MAX_CARRIER_FREQ);


  // Read the smoothened ratio
  int ratioValue = smoothRatio.next(mozziAnalogRead(RATIO_SENSOR_PIN - 14));

  // Calibrate the mapping if needed
  if (ratioValue < ratioInputMin) ratioInputMin = ratioInputMin * 0.5 + ratioValue * 0.5; if (ratioValue > ratioInputMax) ratioInputMax = ratioInputMax * 0.5 + ratioValue * 0.5;

  // Map the input to the ratio
  mod_ratio = map(ratioValue, ratioInputMin, ratioInputMax, MIN_MOD_RATIO, MAX_MOD_RATIO);


  // calculate the modulation frequency to stay in ratio
  int mod_freq = carrier_freq * mod_ratio;
  
  // set the FM oscillator frequencies to the calculated values
  aCarrier.setFreq(carrier_freq);
  aModulator.setFreq(mod_freq);


  // Read the smoothened mod
  int modValue = smoothMod.next(mozziAnalogRead(MOD_SENSOR_PIN - 14));

  // Calibrate the mapping if needed
  if (modValue < modInputMin) modInputMin = modInputMin * 0.5 + modValue * 0.5; if (modValue > modInputMax) modInputMax = modInputMax * 0.5 + modValue * 0.5;

  // Calculate the fm_intensity
  fm_intensity = ((long)modValue * (kIntensityMod.next()+128))>>8;


  // Read the smoothened speed
  int speedValue = smoothSpeed.next(mozziAnalogRead(SPEED_SENSOR_PIN - 14));

  // Calibrate the mapping if needed
  if (speedValue < speedInputMin) speedInputMin = speedInputMin * 0.5 + speedValue * 0.5; if (speedValue > speedInputMax) speedInputMax = speedInputMax * 0.5 + speedValue * 0.5;

  // use a float here for low frequencies
  float mod_speed = (float)map(speedValue, speedInputMin, speedInputMax, MIN_MOD_SPEED, MAX_MOD_SPEED) / 1000;
  kIntensityMod.setFreq(mod_speed); 


  // Set the leds

  FastLED.clear(); // Resets them

  // The frequency controls how many of the LEDs are light up (in a rainbow colour)
  int freqLEDAmount = map(freqValue, freqInputMin, freqInputMax, 0, NEOPIXEL_NUM_LEDS);
  fill_rainbow(&freqLEDs[NEOPIXEL_NUM_LEDS - freqLEDAmount], freqLEDAmount, CRGB::White, 25); // &...LEDs[i] to start lighting from there, allowing us to light them in reverse

  // The speed controls the blinking rate of its LEDs (between 1/2 to 3 seconds per blink cycle)
  int speedLEDBlinkRate = smoothSpeedLED.next(map(speedValue, speedInputMin, speedInputMax, 2000, 500));
  if (millis() % speedLEDBlinkRate < speedLEDBlinkRate/2)
    fill_rainbow(speedLEDs, NEOPIXEL_NUM_LEDS, CRGB::White, 25);

  // For the mod, show a meter (blue - deep pink) showing the mix level of the 2 sounds
  int modLEDAmount = map(modValue, modInputMin, modInputMax, 0, NEOPIXEL_NUM_LEDS);
  fill_solid(modLEDs, NEOPIXEL_NUM_LEDS, CRGB::Blue);
  fill_solid(&modLEDs[NEOPIXEL_NUM_LEDS - modLEDAmount], modLEDAmount, CRGB::DeepPink);

  // The ratio controls the hue of its LEDs
  // int ratioLEDHue = map(ratioValue, ratioInputMin, ratioInputMax, 0, 360);
  // fill_solid(ratioLEDs, NEOPIXEL_NUM_LEDS, CHSV(ratioLEDHue, 100, 50));
  // We could also blend between 2 colours based on the ratio, pick the one you prefer
  fract8 ratioLEDFraction = map(ratioValue, ratioInputMin, ratioInputMax, 0, 255);
  // fill_solid(ratioLEDs, NEOPIXEL_NUM_LEDS, blend(CRGB::Blue, CRGB::DeepPink, ratioLEDFraction));
  fill_solid(ratioLEDs, NEOPIXEL_NUM_LEDS, blend(CRGB::Blue, CRGB::Red, ratioLEDFraction));

  FastLED.show(); // Shows them


  // Communicate with p5
  
  if (Serial.available() && Serial.read() == '\n') { // Send the data once a newline character is received, indicating the end of a message/handshake
      Serial.print(map(freqValue, freqInputMin, freqInputMax, 0, 1000));
      Serial.print(',');
      Serial.print(map(speedValue, speedInputMin, speedInputMax, 0, 1000));
      Serial.print(',');
      Serial.print(map(modValue, modInputMin, modInputMax, 0, 1000));
      Serial.print(',');
      Serial.print(map(ratioValue, ratioInputMin, ratioInputMax, 0, 1000));
      Serial.print(',');
      Serial.println(speedLEDBlinkRate);
    }
  }
}


// Mozzi's function for getting the sound. Must be as light and quick as possible to ensure the sound buffer is adequently filled
AudioOutput updateAudio() {
  long modulation = aSmoothIntensity.next(fm_intensity) * aModulator.next();
  return MonoOutput::from8Bit(aCarrier.phMod(modulation)); // phMod does the FM
}


// Since we're using Mozzi, we just call its hook
void loop() {
  audioHook();
}
 

 

Schematic:

 

p5 sketch (and code):

 

Description of communication between Arduino and p5:

They communicate using a wired serial connection, with the Arduino initiating the handshake. Once p5 acknowledges it, it sends back a newline character, causing the Arduino to send over the normalised values of the flex sensors, as well as the speed neopixel’s blinkrate, and then a newline character back (delimiting the end of the message). Each part waits until the newline to send their data (in the case of p5, just an acknowledgement, while in the case of the Arduino, the sensor and 1 computed value).

 

While I’m not incredibly happy for this, and do wish I could improve things a lot, I’m still pretty glad with how some things turned out. The glove input and the glove itself proved to be quite liked, with many people commenting about the unique control scheme. I also feel the p5 sketch provided another option to view data, resulting in a quicker understanding of the mapping between the amount the fingers are curled and the value it outputs.

However (for future reference), I do wish I could provide greater variety in the synthesizing options (eg. I thought of a button that cycles through different modes, with one mode for example controlling a set of instruments), improve the design & sturdiness of the glove (one of the wires actually disconnected! But luckily it was near the end of the show), and also polish it a bit more.

Week 13: Final Project – User Testing

Hey everyone! 👋

As we’re finishing up our final projects, we also have to conduct user testing to gain valuable feedback and identity potential issues before the showcase.

Here’s one of the short user testing sessions:

 

As you can see, I did not give any prompt or instructions, and they were able to figure it out nearly instantly (that this is a music synthesizing glove), which is a good sign. They were easily able to play around with it and make music, and understood that bending the fingers was controlling the sound.

However, one piece of feedback I got was that while it was obvious which finger controlled which bar, it wasn’t clear what they were mapped to. I had initially expected that the answer would reveal itself upon the glove being played with for a while (especially with the helpful addressable LEDs even providing hints and feedback on their actions), but clearly this wasn’t the case. Most people were able to somewhat guess the action of the first finger (controlling the pitch (actually the base frequency)), but weren’t able to tell more than that. To be honest though, even I wouldn’t have been able to immediately tell the action of the last 2 fingers. So taking this into account, I added little labels at the bottom of the bars, indicating what each finger controlled.

For some curious enough, I also explained how the sound was actually being synthesized. While this does mean there was something to explain, I think this was more the theory behind the scenes, and so it isn’t vitally important to know. In fact, I think people should be able to play around and enjoy the sound, without knowing the actual technical details behind it, which was the case.

Other than that, I’ll polish up the interface, and although I would also like to improve the music generation, I doubt I’ll be able to change it much between now and the showcase.

Week 12: Final Project Proposal

Hi everyone! 👋

I have a bit of a better idea now for my final project (though I’m not ruling out the possibility of doing something completely different 😅). I’m not really sure how best to describe it, but simply put, you have the power to manipulate objects with your hands, and you use to protect yourself and fight against a villain, who you must also chase down (a bit of a reference to my midterm).

# Design Description

Arduino:
Inputs:
  • Pressure sensor (inside of a stress ball), for detecting when the player closes their hand.
  • Capacitive touch sensor strip, as a control for firing.
  • Some game state info (from p5)
Outputs:
  • Neopixel (addressable LED strip)
  • Send values of pressure sensor and touch sensor to p5.
Circuit View:

Note: I used a flex sensor in place of the touch sensor as that wasn’t available right now.
Schematic:

Note: I used a flex sensor in place of the touch sensor as that wasn’t available right now.
p5:
Inputs:
  • MoveNet pose detection, to get angle of the player’s hand in relation with the screen.
  • Pressure sensor and touch sensor values from Arduino.
Outputs:
  • The game on screen.
  • Some game state info to the Arduino.

 

Week 11: Reading Reflection of Graham Pullin’s “Design Meets Disability”

Hi there! 👋

I was reading Graham Pullin‘s “Design Meets Disability”, and found it pretty good. There were so many great examples and points showcased throughout it, in neat little sections, and I like how he treats disability in the context of design, not as something to deal with purely technically (entirely function, no form), but rather as a source of inspiration to create more personalised and human designs.

I was particularly fascinated with the example of eyeglasses. They were initially quite utilitarian objects, only serving to help correct someone’s vision. In line with this (“disability”), they were designed to not draw much attention, with the NHS using pink plastic to sort of camouflage against the skin. That didn’t work particularly well. Not only were they visible, but they also led to a stereotype and a dislike for wearing them, making those who need to wear them potentially feel ashamed and humiliated. Then, they were fashionised. By moving away from this utilitarian and functional only approach, into designing something that also looked great, the industry, and the perception of eyeglasses, were revolutionized.

“up to 20% of some brands of glasses are purchased with clear nonprescription lenses, […] wearing glasses has become an aspiration rather than a humiliation.”

While I do dislike the overly fashion focused industry that has propped up around this, I also recognise that it has made eyeglasses a lot more approachable and comfortable for those who need it, so I am genuinely curious as to why more industries and product design don’t adopt the philosophies mentioned here, particularly about design.

This also reminds me the phenomenon (which we have also previously seen), where things that look better, work better. As I’m most involved in the tech sphere, I can I’ve definitely seen and noticed this effect directly, where even though an app or website might have an amazing functionality, if it doesn’t look appealing, its likely going to be perceived as subpar, and likely isn’t going to go very far (though of course, there are exceptions). This is why in tech, there’s very often a designer on board each team as well. In some cases (such as in 37signals), there’s a 1:1 ratio, as 1 designer and 1 programmer are paired up to work on a feature, highlighting just how important design is.

I also found the story about prosthetics similarly inspiring. While I understand the motive behind not fracturing an already small market, there is something nice to know, that multiple, well designed options exist, allowing someone to be more expressive and confident (something people with disabilities might have a harder time with).

Week 11: Preliminary Final Project Concept

Hey everyone! 👋

For my final project, I’m thinking of doing another glove based project. I’m picturing a gauntlet (black glove with metal attached on top, for that overlapping gauntlet effect), that can detect when a user closes their hand (instead of using flex sensors, I’ll use a stress ball with a pressure sensor inside), and the orientation of their hand (using an IMU). With these inputs, I can allow the player to “grab” and manipulate the objects on screen (sort of like a gravity gun), which they will use to solve puzzles and combat enemies.

However, this is a completely new project (with WEBGL in p5, which isn’t great to work with), so I’m not sure that’s a wise (or even feasible) idea so close to the finals. So, I’m also thinking about expanding upon my midterm, using either the same gauntlet or a sleeve (with a copper pad for the touch sensor, and still an IMU), to point and shoot at enemies or blast through a wall (enlarging the allowable area, as a limited special ability to survive a particularly difficult pose). Or, I might just expand upon HandiSynth.

Week 11: In-class Activity (by Taskin & Zavier)

# Jump To:


# Introduction

Hi there! 👋

These are a few in-class activities Taskin and I did this week (and had to post), resolving around serial communication between p5 and Arduino.

# Exercise 1: Arduino Affecting p5

 

Task:

“Make something that uses only one sensor on Arduino and makes the ellipse in p5 move on the horizontal axis, in the middle of the screen, and nothing on arduino is controlled by p5.”

 

Code:
Arduino:
 
void setup() {
	Serial.begin(9600);
	pinMode(A0, INPUT);
}

void loop() {
	Serial.println(analogRead(A0));
	delay(10);
}
 
p5:
 
let xPos = 0;

function setup() {
	createCanvas(600, 600);
	noFill();
}

function draw() {
	background(32, 64);
	
	stroke("white")

	ellipse(map(xPos, 0, 1023, 0, width), height/2, 100, 100);
	
	// Turn the screen red to make it very clear that we aren't connected to the Arduino
	if (!serialActive)
		background(128, 0, 0);
}

function keyPressed() {
	if (key == " ")
		setUpSerial(); // Start the serial connection
}

function readSerial(data) {
	if (data != null) // Ensure there's actually data
		xPos = int(data);
}			
 

 

# Exercise 2: p5 Affecting Arduino

 

Task:

“Make something that controls the LED brightness from p5.”

 

Code:
Arduino:
 
void setup() {
	Serial.begin(9600);
	pinMode(3, OUTPUT);
}

void loop() {
	analogWrite(3, Serial.parseInt());
	Serial.println();
	delay(10);
}
 
p5:
 
let xPos = 0;
let LEDBrightness = 0; // 0 - 255

function setup() {
	createCanvas(600, 600);
}

function draw() {
	if (keyIsDown(UP_ARROW) && LEDBrightness < 255) LEDBrightness += 1; else if (keyIsDown(DOWN_ARROW) && LEDBrightness > 0)
		LEDBrightness -= 1;
	
	// Just a visual indicator of the brightness level on p5
	background(LEDBrightness);
	fill(LEDBrightness < 128 ? 'white' : 'black')
	text(LEDBrightness, 25, 25);
	
	// Turn the screen red to make it very clear that we aren't connected to the Arduino
	if (!serialActive)
		background(128, 0, 0);
}

function keyPressed() {
	if (key == " ")
		setUpSerial(); // Start the serial connection
}

function readSerial(data) {
	writeSerial(LEDBrightness);
}
 

 

# Exercise 3: Arduino and p5 Affecting Each Other

 

Task:

“Take the gravity wind example and make it so every time the ball bounces one led lights up and then turns off, and you can control the wind from one analog sensor.”

 

Code:
Arduino:
 
const int LED_PIN = 3;
const int POT_PIN = A0;

void setup() {
	Serial.begin(9600);
	pinMode(LED_PIN, OUTPUT);
	pinMode(POT_PIN, INPUT);

	// Start the handshake
	while (Serial.available() <= 0) {
		Serial.println("0"); // Send a starting message
		delay(300); // Wait ~1/3 second
	}
}

void loop() {
	while (Serial.available()) {
		int LEDState = Serial.parseInt(); // 0 or 1
		
		if (Serial.read() == '\n') {
			digitalWrite(LED_PIN, LEDState);
			Serial.println(analogRead(POT_PIN));
		}
	}
}
 
p5:
 
let position, velocity, acceleration, gravity, wind; // vectors
let drag = 0.99, mass = 50, hasBounced = false;

function setup() {
	createCanvas(640, 360);
	noFill();
	position = createVector(width/2, 0);
	velocity = createVector(0,0);
	acceleration = createVector(0,0);
	gravity = createVector(0, 0.5*mass);
	wind = createVector(0,0);
	
	textAlign(CENTER);
	textSize(18);
}

function draw() {
	background(255);
	applyForce(wind);
	applyForce(gravity);
	velocity.add(acceleration);
	velocity.mult(drag);
	position.add(velocity);
	acceleration.mult(0);
	fill(hasBounced ? 'green' : 'white')
	ellipse(position.x,position.y,mass,mass);
	if (position.y > height-mass/2) {
		velocity.y *= -0.9;  // A little dampening when hitting the bottom
		position.y = height-mass/2;
		
		if (!hasBounced && abs(velocity.y) > 1) {
			hasBounced = true;
			setTimeout(() => hasBounced = false, 100); // Set hasBounced to false after 0.1 s
		}
	}  
	
	if (!serialActive) {
		background(8);
		fill('white');
		text("Press c to connect to the Arduino", width/2, height/2)
	}
}

function applyForce(force){
	// Newton's 2nd law: F = M * A
	// or A = F / M
	let f = p5.Vector.div(force, mass);
	acceleration.add(f);
}

function keyPressed(){
	if (key == ' '){
		mass = random(15,80);
		position.set(width/2, -mass);
		velocity.mult(0);
		gravity.y = 0.5*mass;
		wind.mult(0);
		
	} else if (key == 'c') {
		setUpSerial(); // Start the serial connection
	}
}

function readSerial(data) {
	if (data != null) { // Ensure there's actually data
		wind.x = map(int(data), 0, 1023, -2, 2);
		writeSerial((hasBounced ? 1 : 0) + '\n');
	}
}
 

 

Circuit:

 

Demo:

Week 10: Musical Instrument – Handisyn (Group project by Zavier & Xiaotian)

INTRO

What makes something an instrument? I’d like to borrow a concept from Rehding 2016, Three Music-Theory Lessons: ‘epistemic things,’ things that we use to practice knowledge while themselves are the knowledge in terms of how knowledge is generated. Aside from the extent to which a person can easily make acoustically pleasant sound waves out of an installation, what constitutes an instrument is that itself is an epistemic thing—having connections with and making it feasible to practice our common musical knowledge on while embodying a distinct system that generates/contributes to musical knowledge/practice.

That being said, how we divided our work is basically mapping the two sides (and the two stage of the development) to the two of us—the music and the interface, the backend and the frontend, the soft and the hard, etc. On my side, I had the Arduino and the speaker to start up a very basic synthesizer consisting of two audio oscillators and two control oscillators.

Process

In a nutshell, what I tried to realize is a simple cosine wave oscillator frequency modulated according to the four input parameters that control root frequency: intensity (the extent to which the root signal is modulated to the target frequency), vibrato speed (the rate at which the modulation is automatically carried on), and modulation ratio (the multiple to determine the target frequency from the root).

#include <Mozzi.h>
#include <Oscil.h> // oscillator
#include <tables/cos2048_int8.h> // table for Oscils to play
#include <Smooth.h>
#include <AutoMap.h> 

// desired carrier frequency max and min, for AutoMap
const int MIN_CARRIER_FREQ = 22;
const int MAX_CARRIER_FREQ = 440;

// desired intensity max and min, for AutoMap, inverted for reverse dynamics
const int MIN_INTENSITY = 10;
const int MAX_INTENSITY = 1000;

// desired modulation ratio max and min
const int MIN_MODRATIO = 5;
const int MAX_MODRATIO = 2;

// desired mod speed max and min, for AutoMap, note they're inverted for reverse dynamics
const int MIN_MOD_SPEED = 10000;
const int MAX_MOD_SPEED = 1;

AutoMap kMapCarrierFreq(400,700,MIN_CARRIER_FREQ,MAX_CARRIER_FREQ);
AutoMap kMapIntensity(400,700,MIN_INTENSITY,MAX_INTENSITY);
AutoMap kMapModRatio(400,700,MIN_MODRATIO,MAX_MODRATIO);
AutoMap kMapModSpeed(400,700,MIN_MOD_SPEED,MAX_MOD_SPEED);

const int FREQ_PIN = 0; 
const int MOD_PIN = 1; 
const int RATIO_PIN = 3;
const int SPEED_PIN = 2;

Oscil<COS2048_NUM_CELLS, MOZZI_AUDIO_RATE> aCarrier(COS2048_DATA);
Oscil<COS2048_NUM_CELLS, MOZZI_CONTROL_RATE> kIntensityMod(COS2048_DATA);
Oscil<COS2048_NUM_CELLS, MOZZI_AUDIO_RATE> aModulator(COS2048_DATA);

int mod_ratio; // harmonics
long fm_intensity; // carries control info from updateControl() to updateAudio()

// smoothing for intensity to remove clicks on transitions
float smoothness = 0.95f;
Smooth <long> aSmoothIntensity(smoothness);

void setup(){
  Serial.begin(115200); // set up the Serial output for debugging
  startMozzi(); 
}

void updateControl(){
  // read the freq
  int freq_value = mozziAnalogRead<10>(FREQ_PIN); // value is 0-1023

  // map the input to carrier frequency
  int carrier_freq = kMapCarrierFreq(freq_value);

  // read the ratio
  int ratio_value = mozziAnalogRead<10>(RATIO_PIN);

  // map the input to ratio
  mod_ratio = kMapModRatio(ratio_value);

  //calculate the modulation frequency to stay in ratio
  int mod_freq = carrier_freq * mod_ratio;
  
  // set the FM oscillator frequencies to the calculated values
  aCarrier.setFreq(carrier_freq);
  aModulator.setFreq(mod_freq);

  // calculate the fm_intensity
  int mod_level= mozziAnalogRead<10>(MOD_PIN); // value is 0-1023
  fm_intensity = ((long)mod_level * (kIntensityMod.next()+128))>>8;

  // use a float here for low frequencies
  int speed_value = mozziAnalogRead<10>(SPEED_PIN);
  float mod_speed = (float)kMapModSpeed(speed_value)/1000;
  kIntensityMod.setFreq(mod_speed);
}

AudioOutput updateAudio(){
  long modulation = aSmoothIntensity.next(fm_intensity) * aModulator.next();
  return MonoOutput::from8Bit(aCarrier.phMod(modulation)); // phMod does the FM
}

void loop(){
  audioHook();
}

THE GLOVE

Zavier here, now it’s my turn! So after testing the circuit and code (huge thanks to Xiaotian for the awesome sound control!), I began working on attaching it to the glove, which was… painful. I first just measured out and positioned things, then I started attaching them with my small transparent tape (which isn’t the ideal way (sowing it would be better), and besides my tape wasn’t strong at all). After asking a friend for help, I got the sensors attached. Ok, the easy part done. Now the troubles begin. You see, I obviously had to connect the flex sensors to the Arduino. I thought I could I just use the female-male wires, but nope! The pins on the flex sensors were too small, so it was far too lose :(. I tried thinking of a few other options, but in the end, I had to do what I was trying to avoid, soldering. To be honest, I didn’t even mind it that much before (past me talking), and thought it would take a few fun minutes, but boy oh boy. I don’t know what it is about these solders (maybe they’re dirty?), but only a very tiny bit of the tip is actually going to melt the solder, so a lot of time was spent just moving and rotating the tip. Also, I shouldn’t have attached the flex sensors already. It was a huge pain to get them soldered. Now admittedly, (probably a large) part of that is because I’ve hardly soldered before, but also (in addition to the tip issue), I was making these 3 point connections (such as connecting the ground of one the flex sensors, to the ground of the adjacent ones), so whenever I tried soldering, it would just release the other 2!

Anyways, after some work, I finally got the flex sensors wired up, and it was finally working. Great! Ok, we’re done… we’re done?…. right?. Haha, nope. I thought it would be a good idea to add neopixels (addressable LED strips) to the gloves too. After testing a strip, I didn’t repeat my mistake, and this time I soldered together the strips first, before attaching them. This went a lot smoother (also thanks to just having some experience doing it for the flex sensors), but it still took sooo long. Unfortunately, since I soldered it first, the connections weren’t the right length 🙂. Luckily, I had expected them not to be precisely correct (and besides the distance between the strips would change a bit as the hand was flexed and relaxed), and so kept it a bit longer, so that the length could be adjusted as needed. This unintentionally also ended up creating a nice pattern 😅.

While it was a lot of work, it definitely made things a LOT cooler, and also provided a way to give some info to the user visually.

 

Final result:

 

 

 

Demo (note: the sound is very distorted in the recording)

 

Code:

// Configuring Mozzi's options
#include <MozziConfigValues.h>
#define MOZZI_ANALOG_READ_RESOLUTION 10 // Not strictly necessary, as Mozzi will automatically use the default resolution of the hardware (eg. 10 for the Arduino Uno), but they recommend setting it (either here globally, or on each call)

#include <Mozzi.h>
#include <Oscil.h> // oscillator
#include <tables/cos2048_int8.h> // table for Oscils to play
#include <Smooth.h>
#include <AutoMap.h>

#include <FastLED.h>


// Flex sensor stuff

// Define flex sensor pins (these have to be analog)
const int FREQ_SENSOR_PIN = A0;
const int MOD_SENSOR_PIN = A1;
const int SPEED_SENSOR_PIN = A2;
const int RATIO_SENSOR_PIN = A3;

// Smoothening for each pin (was previously using rolling averages)
Smooth<unsigned int> smoothFreq(0.8f);
Smooth<unsigned int> smoothMod(0.5f);
Smooth<unsigned int> smoothSpeed(0.75f);
Smooth<unsigned int> smoothRatio(0.5f);

// Input ranges for flex sensors (will be calibrated)
unsigned int freqInputMin = 1000; // Just FYI, the flex sensors in our setup roughly output in the range of ~ 200 - 650
unsigned int freqInputMax = 0;
unsigned int modInputMin = 1000;
unsigned int modInputMax = 0;
unsigned int speedInputMin = 1000;
unsigned int speedInputMax = 0;
unsigned int ratioInputMin = 1000;
unsigned int ratioInputMax = 0;


// Neopixel (addressable LED strip) stuff

// Define neopixel pins
const int FREQ_NEOPIXEL_PIN = 2;
const int MOD_NEOPIXEL_PIN = 3;
const int SPEED_NEOPIXEL_PIN = 4;
const int RATIO_NEOPIXEL_PIN = 5;

// Number of LEDs in each strip
const int NEOPIXEL_NUM_LEDS = 11;

// Define the array of leds
CRGB freqLEDs[NEOPIXEL_NUM_LEDS];
CRGB modLEDs[NEOPIXEL_NUM_LEDS];
CRGB speedLEDs[NEOPIXEL_NUM_LEDS];
CRGB ratioLEDs[NEOPIXEL_NUM_LEDS];


// Sound stuff

// desired carrier frequency max and min, for AutoMap
const int MIN_CARRIER_FREQ = 22;
const int MAX_CARRIER_FREQ = 440;

// desired intensity max and min, for AutoMap, inverted for reverse dynamics
const int MIN_INTENSITY = 10;
const int MAX_INTENSITY = 1000;

// desired modulation ratio max and min
const int MIN_MOD_RATIO = 5;
const int MAX_MOD_RATIO = 2;

// desired mod speed max and min, for AutoMap, note they're inverted for reverse dynamics
const int MIN_MOD_SPEED = 10000;
const int MAX_MOD_SPEED = 1;

Oscil<COS2048_NUM_CELLS, MOZZI_AUDIO_RATE> aCarrier(COS2048_DATA);
Oscil<COS2048_NUM_CELLS, MOZZI_CONTROL_RATE> kIntensityMod(COS2048_DATA);
Oscil<COS2048_NUM_CELLS, MOZZI_AUDIO_RATE> aModulator(COS2048_DATA);

int mod_ratio; // harmonics
long fm_intensity; // carries control info from updateControl() to updateAudio()

// smoothing for intensity to remove clicks on transitions
float smoothness = 0.95f;
Smooth<long> aSmoothIntensity(smoothness);


void setup(){
  Serial.begin(9600); // set up the Serial output for debugging

  // Set the flex sensor pins
  pinMode( FREQ_SENSOR_PIN, INPUT_PULLUP);
  pinMode(  MOD_SENSOR_PIN, INPUT_PULLUP);
  pinMode(SPEED_SENSOR_PIN, INPUT_PULLUP);
  pinMode(RATIO_SENSOR_PIN, INPUT_PULLUP);

  // Setup the neopixels
    FastLED.addLeds<NEOPIXEL, FREQ_NEOPIXEL_PIN>(freqLEDs, NEOPIXEL_NUM_LEDS);
    FastLED.addLeds<NEOPIXEL, MOD_NEOPIXEL_PIN>(modLEDs, NEOPIXEL_NUM_LEDS);
    FastLED.addLeds<NEOPIXEL, SPEED_NEOPIXEL_PIN>(speedLEDs, NEOPIXEL_NUM_LEDS);
    FastLED.addLeds<NEOPIXEL, RATIO_NEOPIXEL_PIN>(ratioLEDs, NEOPIXEL_NUM_LEDS);
    FastLED.setBrightness(32); // 0 - 255

  // Feed/prime/initialise the smoothing function to get a stable output from the first read (to ensure the calibration isn't messed up). A value of 1630 was chosen by trial and error (divide and conquer), and seems to work best (at least for our setup)
  smoothFreq.next(1630);
  smoothMod.next(1630);
  smoothSpeed.next(1630);
  smoothRatio.next(1630);

  startMozzi(); 
}


// Basically our actual traditional loop in Mozzi (but still needs to kept reasonably lean and fast)
void updateControl(){

  // Read the smoothened freq
  int freqValue = smoothFreq.next(mozziAnalogRead(FREQ_SENSOR_PIN - 14)); // value is 0-1023, -14 since mozzi just takes a number (eg. 0 instead of A0), and the analog ones are 14 onwards

  // Calibrate the mapping if needed
  if (freqValue < freqInputMin) freqInputMin = freqValue;
  if (freqValue > freqInputMax) freqInputMax = freqValue;

  // Map the input to the carrier frequency
  int carrier_freq = map(freqValue, freqInputMin, freqInputMax, MIN_CARRIER_FREQ, MAX_CARRIER_FREQ);


  // Read the smoothened ratio
  int ratioValue = smoothRatio.next(mozziAnalogRead(RATIO_SENSOR_PIN - 14));

  // Calibrate the mapping if needed
  if (ratioValue < ratioInputMin) ratioInputMin = ratioValue;
  if (ratioValue > ratioInputMax) ratioInputMax = ratioValue;

  // Map the input to the ratio
  mod_ratio = map(ratioValue, ratioInputMin, ratioInputMax, MIN_MOD_RATIO, MAX_MOD_RATIO);


  // calculate the modulation frequency to stay in ratio
  int mod_freq = carrier_freq * mod_ratio;
  
  // set the FM oscillator frequencies to the calculated values
  aCarrier.setFreq(carrier_freq);
  aModulator.setFreq(mod_freq);


  // Read the smoothened mod
  int modValue = smoothMod.next(mozziAnalogRead(MOD_SENSOR_PIN - 14));

  // Calibrate the mapping if needed
  if (modValue < modInputMin) modInputMin = modValue;
  if (modValue > modInputMax) modInputMax = modValue;

  // Calculate the fm_intensity
  fm_intensity = ((long)modValue * (kIntensityMod.next()+128))>>8;


  // Read the smoothened speed
  int speedValue = smoothSpeed.next(mozziAnalogRead(SPEED_SENSOR_PIN - 14));

  // Calibrate the mapping if needed
  if (speedValue < speedInputMin) speedInputMin = speedValue;
  if (speedValue > speedInputMax) speedInputMax = speedValue;

  // use a float here for low frequencies
  float mod_speed = (float)map(speedValue, speedInputMin, speedInputMax, MIN_MOD_SPEED, MAX_MOD_SPEED) / 1000;
  kIntensityMod.setFreq(mod_speed);


  // Set the leds

  FastLED.clear(); // Resets them

  // The frequency controls how many of the LEDs are light up (in a rainbow colour)
  int freqLEDAmount = map(freqValue, freqInputMin, freqInputMax, 0, NEOPIXEL_NUM_LEDS);
  fill_rainbow(&freqLEDs[NEOPIXEL_NUM_LEDS - freqLEDAmount], freqLEDAmount, CRGB::White, 25); // &...LEDs[i] to start lighting from there, allowing us to light them in reverse

  // For the mod, show a meter (blue - deep pink) showing the mix level of the 2 sounds
  int modLEDAmount = map(modValue, modInputMin, modInputMax, 0, NEOPIXEL_NUM_LEDS);
  fill_solid(modLEDs, NEOPIXEL_NUM_LEDS, CRGB::Blue);
  fill_solid(&modLEDs[NEOPIXEL_NUM_LEDS - modLEDAmount], modLEDAmount, CRGB::DeepPink);

  // The speed controls the blinking rate of its LEDs (between 1/2 to 3 seconds per blink cycle)
  int speedLEDBlinkRate = map(speedValue, speedInputMin, speedInputMax, 500, 3000);
  if (millis() % speedLEDBlinkRate < speedLEDBlinkRate/2)
    fill_rainbow(speedLEDs, NEOPIXEL_NUM_LEDS, CRGB::White, 25);

  // The ratio controls the hue of its LEDs
  int ratioLEDHue = map(ratioValue, ratioInputMin, ratioInputMax, 0, 360);
  fill_solid(ratioLEDs, NEOPIXEL_NUM_LEDS, CHSV(ratioLEDHue, 100, 50));
  // We could also blend between 2 colours based on the ratio, pick the one you prefer
  // fract8 ratioLEDFraction = map(ratioValue, ratioInputMin, ratioInputMax, 0, 255);
  // fill_solid(ratioLEDs, NEOPIXEL_NUM_LEDS, blend(CRGB::Blue, CRGB::DeepPink, ratioLEDFraction));

  FastLED.show(); // Shows them
}


// Mozzi's function for getting the sound. Must be as light and quick as possible to ensure the sound buffer is adequently filled
AudioOutput updateAudio() {
  long modulation = aSmoothIntensity.next(fm_intensity) * aModulator.next();
  return MonoOutput::from8Bit(aCarrier.phMod(modulation)); // phMod does the FM
}


// Since we're using Mozzi, we just call its hook
void loop() {
  audioHook();
}

 

Week 9: Reading Reflection of “Physical Computing’s Greatest Hits…” and “Making Interactive Art: …”

Hey there! 👋

Tom Tigeo’s article “Physical Computing’s Greatest Hits (and misses)“, mentions many of the physical interaction systems commonly found in interactive projects. While this was published in mid 2008, and there have been numerous advances since then, I don’t think we’ve unlocked any revolutionarily new (or sufficiently different) input interfaces (with the exception of VR & AR), so this is still a pretty good list of possible interaction elements someone could include in their project (though obviously heed his warning and suggestions). I also really liked the 2nd paragraph. Though all of these have clearly already been done (and likely several more ideas, even ones that you might think are totally brand new), it’s still worth pursuing them, as there is still a lot of room for originality and creativity, with your unique variations on them.

 

Moving on to another one of his articles, “Making Interactive Art: Set the Stage, Then Shut Up and Listen“, I find that it’s a good reminder of the difference between classic artwork and interactive art. While traditional art is your expression or a statement, interactive art is a conversation, between (you through) the artwork, and those interacting with it, so don’t prescript or prime people into one way of thinking, but rather design the system well, so they can explore it themselves, and also come up with creatives uses of it. Luckily this is something we’ve been reminded quite often in this class, especially during user testing, so I’m now more often thinking about the affordances and signifiers, and the system of play and interactive between people and my work.

Week 8: Reading Reflection of McMillian’s “Her Code Got Humans on the Moon…”

Hi there! 👋

Today, I also wanna reflect on McMillan’s article, “Her Code Got Humans on the Moon — And Invented Software Itself” (on WIRED), which was an interesting article, talking primarily about the history and nature of Margaret Hamilton’s work with NASA on the Apollo missions.

However, one thing I didn’t really like about the article was that it hypes her up a lot, often unjustly and by greatly exaggerating. Make no mistake, I’m not going after her. The work she did was invaluable, and the fact that she worked in a time when women often didn’t get these roles is noteworthy, and I applaud her for that. It’s the article I have an issue with.

For example, the title literally says “Her Code … And Invented Software Itself”, which to anyone familiar with the history of computing would know, is a huge stretch, to say the least. Ada Lovelace is widely believed to have written the very first computer program (software), and hence is the first computer programmer (though some believe Charles Babbage should have this title, as he had some writings written earlier that could be considered programs). So anyways, the title rests between them, not Hamilton, who came over a hundred years later (no knock on her, it’s simply the facts).

Ok, you might be thinking “So what, a news outlet used a clickbaity and incorrect headline, what’s new?” However, it also mentions “Software engineering, a concept Hamilton pioneered” in the article itself, which is also a bit of a stretch, as neither was she the first, nor did she invent the term, which was already in use by her time (although for some credit, maybe not very widely).

Also, while I get that the article focuses on her, it makes it seem like she was single-handedly responsible for saving the mission and some other important aspects, which leaves out the huge collaborative effort and teamwork that went into those missions, as well as a space to call out the other brilliant experts who worked on it as well.

In the end, the article offers an engaging look at Margaret Hamilton’s vital contributions to the Apollo missions (with some feel-good vibes), and while it definitely might’ve overstated her accomplishments, she is highly accomplished regardless, and her, along with the rest of the entire team’s achievements, are something we can all be inspired by and appreciate.

Week 8: Reading Reflection of Norman’s “Emotion & Design: Attractive things work better”

Hey there! 👋

Today, I wanna reflect upon Noman’s essay, “Emotion & Design: Attractive things work better”.

It was a pretty nice and interesting read, that spoke about how design, behaviour, and (a new term for me,) affect, interact. I feel like it takes pieces which we individually can easily accept as true, and ties them together to create more interesting ideas. Overall, he says that how an object is perceived is also dependent on the emotion of the user, as well as the beauty/design (obviously including its workings, but not sorely dependent on it). For example, happy, relaxed users interacting with a beautifully designed product are much more likely to forgiving and overlook some of the faults and issues, compared to focused users in stressful situtations. He ends the main text with title text “Attractive things work better” (an arguably controversial but true statement).

However, while reading this, I couldn’t help but think of some of my previous research (for a different course), where a similar thing could be said for people. Now obviously, this is much more controversial and offensive, but bare with me, research has shown that the “Pretty Privilege” or “Attractiveness Advantage”, where people who are more conventionally attractive have a noticeable advantage in many aspects of life, is unfortunately, very real.

For example, people are slightly more forgiving to those conventionally attractive, not too dissimilar to how people were more forgiving of issues in beautifully designed objects. Also, the halo effect (where a someone associates other positive traits after seeing just a few / one) comes into play too, and since one of the first thing people see when they meet someone, is how they look, that often ends up being the basis (or initial positive trait) for the halo effect, and so those people are usually also considered smart, trustworthy, and kind (among other things).

However, the opposite is also true, and something people who are more conventionally attractive are also considered to be rude, snobbish, arrogant, and undeserving of their success (sometimes for good reason, but not always, and certainly not everyone). Similarly (going back to objects), I think that people also hold more beautiful products to higher standards, and although they might end up being more forgiving sometimes, they could also easily become upset due to expecting it to support their desired functionality (and quite well too), as simply the appearance of a better looking product often leads people to believe it is also better made.

In conclusion, I found this essay to be an insightful reading about how design, affect, and behaviour interact, and how the experience of someone using a product could differ not only on how it works, and not also only about its design, but also the emotional and mental state the user is in, and how the design needs to take that into account. I then compared and contrasted this with how people interact with others (specifically those considered more conventionally attractive), and while drawing similarities between people and objects may seem inhumane, I think it could help us further understand and research these effects, as there’s now another situation that exhibits the same properties, allowing us to look at this problem from another angle.