I made a simple “switch” which completes the circuit when there’s a conductive fluid in between the ground and live wire. It’s a simple circuit, and there’s not much to it really. Water is not supposed to be conductive, but the tap water on campus has added minerals, so that works for this purpose. I just used the LEDs and jumper wires in the Arduino kit, and borrowed some solid core wires from the IM Lab to put in the liquid container.
I had the most fun making the demo film with my friend, though. Here’s the demo:
When my friend pees in the container with the two exposed wires which were initially not touching, the circuit is completed, and the LED lights up.
The piece by Dan Norman aptly captures my response to reading The Design of Everyday Things, which I had brought up in class discussions as well, and this story of the three teapots in Emotion & Design is a perfect follow up to his story about the artistic but inconvenient doors in his previous work. “Pleasurable designs are not necessarily usable. But need these attributes be in conflict? Why not beauty and brains, pleasure and usability?” I would consider myself a proponent of beautiful design, but I am also a regular user of everyday things, and I care about their usability. What Dan Norman says, and I agree, is that “design matters, but which design is preferable depends on the occasion, the context, and above all, my mood.” There need not be a race between aesthetics and utility. In fact, I like nice design so much that I occasionally go on the Museum of Modern Art online store website just to look at pretty things. This $315 lamp or this $140 side table are probably not setting utility standards for any other lamps or side tables, but they’re beautiful, and I am glad someone designed them.
Also, Norman’s discussion of the system of affective thinking was agreeable, and quite therapeutic to be honest. I can appreciate how it can be taken as a lens for looking at life in general. If the affective system works independently of thought, as he claims, then I can attribute unstructured or neurotic thoughts to the neurochemicals that “bathe” my relevant brain centers, affecting the way I “perceive, decide, and react.” Norman also offers his solution: affective reactions can be overcome with time and practice. This, he says, requires patience, and taking agency over your thought process. This too, then, is not a race, since anything that requires patience by definition cannot be done in haste. To learn to have healthy thoughts is to learn to be patient.
For my midterm project, I created a spooky juggling game called Morbid Juggler. I thought of the concept in week 2, and for the project involving arrays and classes I made a simple game that lets users add balls using the keyboard and juggle them using the cursor. Morbid Juggler version 2 is built on top of the same motion logic, but instead of using standard input (mouse and keyboard), players have to use their webcam and their hands to interact with the game. To add a new ball, the user makes the “🤘” gesture with their left hand. The balls automatically move in a parabolic trajectory, and the goal of the game is to not drop the balls, i.e. catch them before they leave the screen. To catch a ball, users can use any hand and pinch their fingers and drag a ball across the screen. To throw the ball again, they just have to release the pinch.
How it works
The following explains the balls’ motion and is from my documentation for my week 2 project:
I save the initial time when an (eye)ball is created, and measure the time that has passed since. This allows me to use values of elapsedTime as a set of x values, which, when plugged into a quadratic equation, give a parabola. Changing the coefficients in the equation allows me to modify the shape of the parabola and thus the trajectory of the (eye)ball, such as how wide the arc created by its motion is. I played around with the coefficients and decided to use (0.4x)(2-x), which works nicely for a canvas of this size.
A more detailed explanation of how the balls are stored in memory and moved with each frame update can be found here.
For tracking hands using the webcam, I used a Javascript library called Handsfree.js. I was initially going to use PoseNet, but I realized it wasn’t the best for this use case. PoseNet’s landmark model tracks 17 points on the human body, but for my game, I didn’t need all that. I just needed to track the user’s fingers, and PoseNet only returns one keypoint for the user’s wrist. So, I looked up other libraries and found Handsfree.js, which is built on the same framework, Google’s Mediapipe, as PoseNet, but it is more geared towards hand tracking. It tracks all fingers and knuckles and even has built in gesture recognition to detect pinches, which is what my users would need to do to drag balls around the screen. Furthermore, it was very easy to train the model to recognize new gestures using the website. It lets you collect your own data and create a gesture model to plug into your code. For example, this is the code for recognizing the “🤘” gesture.
function addBall() {
const hands = handsfree.data?.hands;
if (hands?.gesture) {
if (hands.gesture[0]?.name == "addBall") {
let x = sketch.width - hands.landmarks[0][9].x * sketch.width;
let y = hands.landmarks[0][9].y * sketch.height;
console.log(x, y);
balls.push(new Ball(x, y));
canAddBall = false;
// go to sleep for a second
setTimeout(() => {
canAddBall = true;
}, 1000);
}
}
}
The hardest part of this project was working with Handsfree.js. There is criminally limited documentation available on the website, and I had to teach myself how to use it by looking at the few demo projects the author of the library had created. For hand tracking, there was a piece of code that closely approximated what I wanted to do. It loops through the hands in the object returned by handsfree.data, and for each hand, it loops through all the fingers. Each finger can be identified using its landmark, and its location and other information can be used elsewhere in the program. For example, in handsfree.data.landmarks[handIndex][fingertips[finger]], handIndex=0 and finger=8 represents the tip of the left index finger. In Morbid Juggler, when handsfree.data.pinchState for any hand or finger becomes held near a ball, the ball sticks to the tip of the pointer finger. When it becomes released, the ball restarts its parabolic motion.
To see the full code for the project, refer to the p5 web editor.
Playing the game
The game can be played here. When the project has loaded, click on ‘start game’. It takes a second for Handsfree to load. You can check if the game is fully loaded by waving your hands in front of your webcam. You should see the skeleton for your hand mapped on the canvas.
Make a “🤘” sign with your left hand (tuck in your thumb, and make your index and pinky straight!). You’ll see a new ball pop from between your fingers. Add a few more balls like this, but be quick! Don’t let them leave the screen, or the counter in the corner of the screen will tell you how badly you’re losing even after death. To keep the balls from falling off, pinch your fingers like you would pick up a real juggling ball, drag the ball you’re holding across the screen, and release your fingers somewhere near the bottom left corner of the screen, so the ball can travel again. You can add as many balls as you like — at your own risk.
Final thoughts
I think there’s still room to fine tune the hand tracking. If I were to upgrade the game, I would probably use a different library than Handsfree.js. The tracking information isn’t the most accurate, and neither is it consistent. For example, even when a hand is being held up still, the keypoints on screen are seen to jitter. Since during dragging, the balls stick to the tip of the pointer finger, the balls were jittering badly too. I later added some smoothing using the lerp() function to fix that. I also had to do a lot of trial and error when picking a gesture that would trigger the function to add a new ball. The model wasn’t very confident about a lot of the other gestures, and kept adding balls erroneously. The “🤘” sign was the final choice because it was explicit enough and did not resemble other gestures a user might inadvertently make while playing the game.
One thing worth special mention is the background music used in the game, which was produced by my friend by sampling my other friend’s vocals. I heard the track and liked the spooky yet playful feel to it and asked if I could use it in my project. My friend agreed, and now I have a bespoke, perfectly fitting music to accompany my project.
I think as a kid, for the longest time, I was obsessed with telekinesis. I had just watched Matilda, and although I didn’t know the name for it yet, I already knew how groundbreaking it was that Matilda could move things by the power of her mind, without touching anything. I would focus really hard on something like she did, and try to make it move (just a centimeter would have been enough), but it never worked.
I am happy, elated even, to announce that I think I am finally old enough to finally realize my childhood dreams. So for my midterm project, I will be building a computer vision game that lets you interact with the virtual world without using a mouse or keyboard — your “stdin”. I am looking at classic games for inspiration, such as Fruit Ninja, or maybe the games for PlayStation EyeToy mentioned in the reading.
I can use ml5.js+PoseNet for this project, and there are so many online resources for learning about them. Even Daniel Shiffman has PoseNet tutorials and code examples, so I hope this idea isn’t too ambitious for a midterm project. I still haven’t written any code, but this is how I imagine the final project structure will look like:
1. The basic structure is built with p5.js
2. Using PoseNet, I will train my model to recognize specific poses (such as throwing a punch, or dicing the fruit with a slash of the hand, etc.)
3. The data returned by the model will be used in the core p5.js code to alter the appearance of elements on screen. For example, if it is something like Fruit Ninja, I could use sprite sheets to animate the fruit being cut up.
4. Obviously, sound plays a big part in our reality, so to make the game more immersive, I would have to trigger particular sound effects based on the user’s actions.
I am a little wary of using the ML model and of having to train it. Everything else I could just look up and implement, but training an ML model seems a little daunting. I have worked with PoseNet before, but it was a much simpler task than the one I now plan to undertake.
Reading this article, what I picked up was this: “it took computer scientists a few decades to figure this out, but now you get to play around with it too.” At the beginning, the author states how computer vision used to be a domain exclusive to specialists, such as the military or law-enforcement. Then at the end, there’s the appendix with the sample processing code that even “novice programmers” could use to kick-start a computer vision project “in as little as an afternoon”. This is grounds for celebration because, by lowering the barrier to entry here, we have democratized the technologist community by a notch, and that opens up the scope for so much creativity that would not have such a limitless medium of expression otherwise. The article seems a little old, and newer technologies have entered the space since its time of writing, including as Google’s Teachable machine. It is fascinating how I can now train a ML model and make something like *Cheese* by Christian Möller (2003) in a few hours using Teachable, whereas Möller’s piece was a product of the joint effort of the artist and a whole research laboratory.
Another aspect of computer vision that made me think was the hand of humans, and all of our humane subjectivities, in the development of these algorithms. The author mentions that a computer processes a video stream differently than text, in that there are no principles for semiotic interpretation in place, and so computers need to be programmed to answer even the most rudimentary questions about the images it “sees”. Then, the algorithms are susceptible to absorbing the same implicit biases as their creators. For example, whether a computer thinks it’s looking at a “woman in traditional clothes” or “an exotic foreigner” will depend largely on how the programmers perceive the woman. Biases in AI and computer vision are now a popular topic in contemporary tech discourse, but I wonder how these concerns have evolved over the years, since the inception of computer vision in the 70s until now. Have programmers always been wary of negatively influencing their softwares, or are these 21st century fears? What can technologists do to ameliorate them?
Initially, I wanted to make a nice data visualization, but by my standards, it would only be cool if the data was live. There’s little glamour in loading a CSV file — even Excel can do that. However, I really struggled to find a free API that lets you make bulk requests, and I couldn’t get around the request limit without paying for it. I settled for a project on loading/displaying text, and came up with my alphabet soup idea.
I tried creating a bowl of p5 alphabet soup, which lets you add new letters by typing on your keyboard. The alphabets all float randomly on the canvas, but they can bunch together if you haven’t pressed space. When they hit space, the bunch is ‘complete’, and the next time the user types, a new bunch is created. Under the hood, each bunch is just an array of “Letter” objects, each of which have their own position and speed. At the end of every array, there’s an invisible space, which I’m moving across the screen, and all the letters simply follow the next letter in the array. A little Perlin noise and rotation is used to simulate some turbulence in the liquid.
Additionally, the user can also use the mouse to scramble the letters on the screen. If the pointer is too close to a certain Letter, the Letter is separated from the bunch and it will go on moving independently. Whether the Letter was in the active bunch or an already completed bunch, it will be extracted from its array and added to a different one called floatingLetters. In this array, all the Letter objects are always moving around randomly. Letters in this array can also be scrambled — the user just has to hover the pointer near a Letter.
The logic to make the letters bunch together was a little tricky to pin down, and a bunch in this context wasn’t really like a boid. They have to (loosely) maintain a left to right order, and that depends on which direction they’re moving in. I eventually took the gradient (speed differential) of the last letter in the array, and made the rest of the letters arrange themselves accordingly. For example, if the next Letter is moving down and right, the current Letter will be drawn slightly to the left and up. If the next Letter is moving up and right, the current Letter will be drawn slightly to the left and below, and so on. There are four such cases.
I used a font that resembles the letters in alphabet soup, and each letter also has a shadow. In real soup, letters are always going above and below the surface of the liquid, which makes sometimes makes look smaller, fainter. I think I could make this more realistic if I added varying opacity to each letter. If the Letter objects randomly had a lower opacity, it would look like they were drowning a little.
Dan Norman takes a strictly humanistic stance in the opening of his book, which can be summed up nicely in this one line from the chapter: “It is the duty of machines and those who design them to understand people. It is not our duty to understand the arbitrary, meaningless dictates of machines.” His ideas are well informed and drawn from experience in both engineering and cognitive science, and I do in fact agree with him on the subject of the gap between designers’ faith in users who are willing to trial and error, and the very limited size of such a user base. Most people do really just want to get on with whatever they’re doing. However, the same people also want things to look nice. For example, almost all of us care about the aesthetic qualities of appliances we put in our homes. There’s people like the English couple mentioned in the book with the fancy washing machine, who don’t bother to learn about the machine’s functions but still keep the high-tech appliance around probably because it just looks nicer.
Another interesting case study would be SMEG, a home appliance company with a focus on design. They consistently get a bad rap for the poor quality of their products, which are simply not good investments. Yet, SMEG remains immensely popular, and consumer behavior shows that people are willing to pay much more for an appliance, be it of poorer quality, for something that just looks nice. Thus, I have this lingering question for Dan Norman: “so what?” Humans were born into an unbending, mystifying nature. But people still made it work, weaved clothes from flowers of cotton and learned to bake bread from grains. Now, is a washing machine with one too many functions really going to the nemesis to human development? Are we expected to be so lazy that reading a device manual becomes too much to ask? I’m really not talking about bad design — if I have to struggle to read text on a website in an avant-garde font, maybe that’s a design choice worth reevaluating. But some pretty doors that blend into a building’s facade, which can be swung open by pushing on either one of just two ends, shouldn’t really be that much of a headache to most people, considering once you’ve taken a minute to figure out which end to push, you can forever go on enjoying the nice design of a normally functioning door.
For this project, I created a juggling game and named it Morbid Juggler. The only rule is to keep juggling the (eye)balls for as long as possible. I myself don’t really know how to juggle, so I had to think a little bit about how to create the same stressful experience with code, and the p5.js project embedded below is the culmination of that.
The player adds an (eye)ball to the screen by pressing shift on the keyboard, and the new (eye)ball is added at the cursor’s current position. The (eye)ball travels just as a juggler’s ball would in real life, with gravity acting on it, in a parabolic trajectory. The game goes on if the player can catch (click and hold) an eye(ball) before it leaves the screen, and bring it over to the side and release it again. When an (eye)ball is rereleased, it restarts its motion. The player can add as many (eye)balls as they can handle by pressing shift, but there is a counter in the corner that shows how many (eye)balls they have dropped. The game goes on forever: there’s no winning, because in the Morbid Juggler the player is already dead, and they can keep juggling forever. When they run out of (eye)balls, they can just hit shift again.
Note: when the drawing first loads, the canvas needs to be woken up with a click for it to be able to detect key presses.
Behind the scenes, the (eye)balls are objects of the Ball class. When the user hits shift, a new Ball object is instantiated. I think this is where I had the most fun, figuring out how to make the objects move like a real object in free fall. If we launch an object at an angle in a vacuum chamber (no drag), the projectile will have a constant horizontal velocity. Vertically, on the other hand, the object will accelerate due to gravity, and its vertical distance (y) would be a function of time elapsed since launch, which also dictates its horizontal distance (x).
Equipped with this very basic understanding of projectile motion, I save the initial time when an (eye)ball is created, and measure the time that has passed since. This allows me to use values of elapsedTime as a set of x values, which, when plugged into a quadratic equation, give a parabola. Changing the coefficients in the equation allows me to modify the shape of the parabola and thus the trajectory of the (eye)ball, such as how wide the arc created by its motion is. I played around with the coefficients and decided to use (0.4x)(2-x), which works nicely for a canvas of this size.
So I use elapsedTime to move the object horizontally: x=x+elapsedTime/3, and the y values come from the quadratic equation. But I also used the map function to normalize the values, which gives me a nice set of y values from 1 to 5. Both these choices ensure that the simulation runs at a speed appropriate for a playable game.
When the object is created, the time elapsed is zero, so our x value is also zero. As timeElapsed goes on increasing, the y values trace a parabola. The user can click and drag an (eye)ball, and upon release the object’s contactTime property is reset so that timeElapsed is zeroed. This is why the object restarts its parabolic motion.
The drag and drop effect was a little tricky to achieve. Since I’m updating the position of each object using a for loop, and I was trying to update the selected object’s position within the loop, my initial code kept selecting the wrong object for dragging. Although it took me a while to arrive at this, a simple solution was to use a helper function to detect when a particular object is clicked, and then sync its position with mouseX and mouseY in the draw function.
// detect when a user is dragging a ball
function mousePressed() {
for (let i = 0; i < balls.length; i++) {
let d = dist(mouseX, mouseY, balls[i].x, balls[i].y);
if (d < 20) {
draggingBall = i;
offsetX = balls[i].x - mouseX;
offsetY = balls[i].y - mouseY;
// break oout of the for loop immediately, otherwise a different ball might be selected
break;
}
}
}
For some time, I was trying to import an .obj model into my project so the ball would be 3D. The game would look nicer, but it was starting to get more complicated, and I kept getting errors from the p5 script. i settled for simply rotating the (eye)balls as they move, which is a much simpler workaround but produces a similar visual effect.
I read this chapter in one long “aha!” I can tell the author sounds like someone who has put a lot of thought into the philosophies that drive digital creation, and I would definitely be interested in looking into the later chapters of the book. I can’t think of a time when I really stopped to think along these lines: when people ask me what I want to do with my computer science degree, I blurt out a mushy something that resembles the job description of a web developer. When I work on web or mobile projects, I’m usually doing all the design and code by myself, so finding the line between these things just never came up. But, now enlightened, I can say that there is in fact, a place for a UI designer and an interactivity designer, and I think the final product is more the outcome of how these two agents (among other things) interact with each other rather than simply the sum of their individual contributions.
Definitely a lot of takeaways here (for me at least), but what hit closest to home was something he wrote towards the end: “good interactivity design integrates form with function”. Ever since I discovered Codepen and JsFiddle, I have been integrating the most random effects and interactions in my work. Sometimes a website has frills that doesn’t really add anything to it but doesn’t take anything away either, but in my work I would compromise on function for something that just “looks cool”, regardless of whether it actually complements the remaining UX or gets in the way of the user. If I can take apart my roles when I’m working alone, and think for stakeholders from both arts/humanities and math/science backgrounds, or those from the “Webby” generation or the “Personal Computer” generation, my finished work would probably come together much better.
I tried recreating The Strokes’s “Is This It” album cover for this project. Here’s the reference picture:
The album originally had a different cover, but some time after the first batch of vinyls had already been pressed, frontman Julian Casablanca called his product manager saying he found “something even cooler.” The record label obliged, and later productions of the album appeared with a picture from CERN’s Big Bang experiments in the 70s. This new image shows tracks left by subatomic particles in a bubble chamber. As each charged particle moves through superheated liquid hydrogen, it leaves a trail of “bubbles” as the liquid boils in its wake. The trajectory of their paths can then be analyzed to identify the particles. For example, two particles with opposite charges would travel in opposite directions, because of how they interact with the magnetic field in the chamber.
This is how I approached the problem of animating these particles: I use a for loop to generate some sets of random values (within defined limits) for initial coordinates, radii, angles, and stroke weights. Each set determines the initial frame for each particle. A heavier particle, such as an alpha particle, would have a smaller radius than a smaller particle, such as an electron, because the heavier particle moves slower, which is why each track needs to have a different radius. They can also curve in one direction or the other as they travel, depending on the charge they carry. So the angle needed to be randomized between negative and positive values.
// array to store trail "head" information
for (let i = 0; i < 20; i++) {
let x = random(width);
let y = random(height);
let weight = int(random(1, 5));
let angle = random(-TWO_PI, TWO_PI);
let radius = random(1, 10);
particleArray.push({ x, y, weight, angle, radius });
}
Every time the draw() function is called, I offset the values in each set by a little, to define the new position for a particle, and then draw a line joining these two coordinates. The for loop in this function goes over each “particle” in the array and gives it a new position to be at. When the frame refreshes, this happens all over again.
There’s a lot more that’s going on in the album cover, and I’m trying to make my artwork something more than just lines on a yellow background. So far I’ve only added some grain to the canvas. I can also make out some drop shadow in the original picture, but it’s a little complicated to achieve that effect with code. Since the arcs are comprised of infinite small lines, I don’t have the shape of the full arc at any point in time, so I can’t add drop shadow to the whole arc. But if I add shadows to each little line, the shadow falls not just on the background but on neighboring lines too, and that makes the arc look disjointed. I just added some noise to the coordinate calculation so that the arc look just slightly more realistic and not completely smooth.
I think the trickiest part was figuring out the logic for how to connect each coordinate. At first, I was trying to use ellipses instead of lines, and wrote some more complicated logic to connect them in an arc, but I did something wrong so it would connect points of different arcs, shown below. It looks cool in its own right, but it just wasn’t what I was trying to do.
