Category: F2023 – Mang

Concept

Something that I have always been interested in but never had the opportunity to make is visual novels. Having the option to influence and experience a story with your own choices is a concept that has always fascinated me in video games, and I believe that P5.js would be great for that. Through this form of interactive fiction, I would be able to tell a story that would not only be engaging but also intriguing, creating something that would look aesthetically pleasing while also invoking several emotions.

In regards to the story, my intention is to depict a weird and disturbing yet comical narrative, focused on making the user uncomfortable. Topics will most likely be related to dark themes, but in subtle and fun ways in order to not cheaply shock the user.

Design

My initial inspiration for the visual novel came from the game “Phucker in the Woods”, due to its bizarre storyline and visuals. What captures my attention is the uncanny design of the characters, something that I want to replicate although in a minimalistic way. The ambient and sounds will be dark, using mostly black and white colors and eerie ambient music and sound effects. The storyline will be short and simple, but with different choices that the player can make, altering the endings. Each screen will have drawn characters and objects along with dialogues.

Below is an example of the character which the player will follow:

Additionally, I would like to implement different game mechanics as opposed to just using the point-and-click aspect of a visual novel. Two of these mechanics could be scenes in which the player can control the character and walk around the screen by using sprites and the “keyIsDown” function, and a maze scene in which the player controls a cursor.

Frightening / Challenging Aspects

• Coding the logic of screen transitions after each choice
• Implementing an engaging story within a reasonable amount of resources and lines of code
• Applying different game mechanics under one project

Risk Prevention

• Utilize boolean variables to update the screen after each choice
• Write a concise story with a limited amount of choices and refactoring the code as much as possible
• Only focus on the point and click aspect or make the scenes with different mechanics as simple as possible

 Among the three sections of the paper, the one I liked the most was “Computer Mission in Interactive Art.” When the author initially introduced the idea that our entire body could interact with computers, I was a bit skeptical and didn’t fully grasp it. However, as I read more about it, I found it to be truly amazing because it represented the first step in enabling people to interact with computers in this way. It was like the initial building block for human-computer interaction. What I found particularly fascinating was how people were exposed to a new platform they had never encountered before. Just imagine the reactions of those who first learned about it; it must have been quite impressive.

Though many examples are given, One of the examples that caught my attention within this section was “Messa di Voce,” a performance where voices were transformed into images. What struck me as the coolest aspect of this artwork was that the sounds themselves generated the images. At a time when audio platforms were not as common as they are today, the concept of visuals emerging directly from audio would have been incredibly intriguing for the audience.

The next art installation, “Standards and Double Standards” from 2004, featured fifty belts suspended from the ceiling using special robotic motors. The interactive element here was particularly engaging and exciting for viewers. The belts responded to the movements of people in the room, creating a dynamic and immersive experience. It was as if the belts came to life, interacting with those present, making the overall experience dynamic and captivating.

These examples span a wide range of artistic categories, highlighting the diversity in how artists use technology to create interactive experiences. It’s remarkable to think about how these early innovations have played a significant role in shaping the technology-driven world we live in today. They served as the foundation for the kinds of interactive experiences we now take for granted.

 

“Computer Vision for Artists and Designers: Pedagogic Tools and Techniques for Novice Programmers” offers a fascinating exploration of computer vision and its applications, particularly within the realm of art and design. It shows how computer vision has traditionally been reserved for experts in fields like signal processing and artificial intelligence. What I liked the most about this paper is how it breaks down complex concepts related to computer vision into simple terms, making them accessible to readers without a technical background.

Among the diverse applications discussed, one that particularly stood out to me is the audiovisual performance “Messa di Voce.” This performance seamlessly combines whole-body vision-based interactions with speech analysis to visually enhance the expressions of abstract vocalists.

Furthermore, the paper introduces the concept of motion detection, which proved to be a valuable insight for my own project. My midterm project heavily relies on motion detection, as it involves tracking the movements of the user and objects within the video frame. Techniques like frame differencing, as discussed in the paper, offer a practical approach to dynamically adjusting colors and content in specific video areas. This knowledge aligns perfectly with my project’s objective of creating an immersive and responsive text display.

Name: Designer Chair Plunge

Concept: For my midterm project I want to implement a simple game inspired by famous designer chairs. Currently I am taking a core class named “Have a Seat” where we explore various ideas about seating, posture and chair design, and so this class is my main inspiration for my midterm. Regarding the strategy of the game, I want to create a falling designer character that the user has to catch by manipulating the chair on the bottom of the screen. The chair is responsive to the mouseX position therefore the user controls the game just by the movement of the mouse.

Moodboard: –>

 

Sketch:

I firstly started by implementing the control strategy of the game with simple figures. The ball and the rectangle are both objects, and the ball starts to move faster after each successful catch to increase the pace of the game.

Further/ complex implementations:

The nexrt steps will be to replace the simple figures with the png character and the chair. I would like the chair figure to be randomized from a number of different chairs to increase the variety. I will also need to work on the start game and end game windows to match the aesthetic. I will want to add some background music and add sound effects for each succesful catch or miss. I would love to draw the figures myself, so probably this would be the most challenging part.

 

I think as a kid, for the longest time, I was obsessed with telekinesis. I had just watched Matilda, and although I didn’t know the name for it yet, I already knew how groundbreaking it was that Matilda could move things by the power of her mind, without touching anything. I would focus really hard on something like she did, and try to make it move (just a centimeter would have been enough), but it never worked.

I am happy, elated even, to announce that I think I am finally old enough to finally realize my childhood dreams. So for my midterm project, I will be building a computer vision game that lets you interact with the virtual world without using a mouse or keyboard — your “stdin”. I am looking at classic games for inspiration, such as Fruit Ninja, or maybe the games for PlayStation EyeToy mentioned in the reading.

I can use ml5.js+PoseNet for this project, and there are so many online resources for learning about them. Even Daniel Shiffman has PoseNet tutorials and code examples, so I hope this idea isn’t too ambitious for a midterm project. I still haven’t written any code, but this is how I imagine the final project structure will look like:

1. The basic structure is built with p5.js
2. Using PoseNet, I will train my model to recognize specific poses (such as throwing a punch, or dicing the fruit with a slash of the hand, etc.)
3. The data returned by the model will be used in the core p5.js code to alter the appearance of elements on screen. For example, if it is something like Fruit Ninja, I could use sprite sheets to animate the fruit being cut up.
4. Obviously, sound plays a big part in our reality, so to make the game more immersive, I would have to trigger particular sound effects based on the user’s actions.

I am a little wary of using the ML model and of having to train it. Everything else I could just look up and implement, but training an ML model seems a little daunting. I have worked with PoseNet before, but it was a much simpler task than the one I now plan to undertake.

The concept:
For my midterm project, I was inspired to create a unique piece of interactive art that would heavily rely on the user’s live video display. This idea had been brewing in my mind for several weeks, sparked by two key moments: first, when I saw a screen outside the introduction to interactive media (IM) lab displaying users amidst a backdrop of language characters, and second, when our professor showcased a project featuring flying birds (shown in image 1). These encounters left me captivated by the potential of technology and user interactivity in the realm of art. Additionally, I’ve always had a keen interest in game development, especially when it comes to implementing various elements such as graphics, animations, and user interactivity. So, I set out to combine both concepts by creating a simple engaging game that would contribute to the dynamic text video display.

The core concept of the game revolves around collecting words. In this game, the player maneuvers a box to capture randomly falling words. These words are a fusion of adjectives that depict various moods and emotions. To add an extra layer of depth, these words are categorized, with each category corresponding to a specific display color and score. As the player collects these words, they have a direct impact on the text video display. The collected words are seamlessly integrated into the live video feed, dynamically altering the text and its color based on the specific word the player successfully captures.

Design:
when it comes to the visual layout, I’ve split it into two halves (as shown in image 2). One side is all about the game interface (right side), where you interact with the falling words and move that collecting box. The other side (left side), well, that’s where the magic happens. It’s your canvas for the live video display, where you get to experience the fusion of gaming and art.

Sketch:

Challenges:
One of the challenges I’ve encountered involves effectively implementing the text video display within the project. It requires intricate coding and synchronization to seamlessly integrate dynamic text updates with the live video feed, ensuring a cohesive and engaging user experience.
Another challenge is handling every single pixel to ensure a crisp and smooth video display. To achieve a higher level of visual quality, I’m opting to increase the pixel density on my canvas. However, this choice brings the potential for glitches and slower performance.

Risk prevention:
To implement the text video display, I’ve delved into tutorial resources that explain similar concepts. These tutorials have been invaluable in guiding me through the complexities of working with a canvas as a grid of individual pixels. I’ve gained insights into handling pixel-level manipulations.
To achieve the perfect balance between visual clarity and smooth execution, after getting done with other project parts, I will try to optimize the number of pixels which I think will be done by testing different pixel settings.

Concept:

For my midterm project I want to create a piece that resonated with a hobby of mine. I have always enjoyed baking and cooking. It’s such a calming experience, creating something from scratch. Whenever I get stressed out, I turn to my kitchen and let myself get lost in my recipe book, I let it take away my stress and heavy thoughts, therefore I decided to create a small kitchen simulation. I hope that through interacting with my piece users a can get to feel a semblance of the calmness I get to experience when I bake/cook.

I will design it in a way so that the user will get to choose one of 3 recipes, cupcakes, spaghetti and meatballs or an avocado salad, to bake/cook. 

Each recipe will be displayed as an icon on the homepage, when pressed it takes the user to the next page to start the preparations. The user will have to drag and drop the ingredients in order to prepare the food and then put it in and take it out of the oven if needed, there will be different steps for each recipe that resemble the actual recipe being followed. I will also try to attach a recipe that the user can download at the end to try the recipe in real life if they’re interested. 

Design:

I want to use a soft palette to resonate with the calming experience I am relaying to the users as well as deliver an aesthetically pleasing design. I did a quick sketch to show what I have in mind for the homepage and an example of the following stages, I will add more dynamic objects and interactivity features within my design when I am actually implementing it.

homepage:

cupcake-making page:

Expected Challenges:

I am expecting to have some troubles with the drag and drop feature and the transitions between the screens. I want the transitions to be as smooth as possible without having any awkward jumps or glitches. I also expect to have some troubles with transitioning between different music files when switching different scenes because I do not want to have any weird transitions when switching tracks.

Risk Prevention:

For the drag and drop, I plan on watching several youtube videos, and look at other people’s code to understand how to implement the drag and drop feature as easily and as efficient as possible because I am a bit conflicted on how to make without running into other bugs in the code.

for the transitions, so far I have come up with having multiple screen variables and a function for each screen that would be called at the relevant times to switch to the desired screen. I will also refer to the p5js. reference page and examples page to get some inspiration for the transitions that could be used to make my project as close to perfect as I can.

p5js Sketch:

for now this is what I have created, I am still working on class definitions, function creations and music loading. I have only created the first page, but I have implemented the icon buttons, however they only direct the user to an empty screen right now.

An issue I faced is the decrease in the resolution of the icons when I loaded them onto p5js, which I am trying to fix as well because it completely ruins the visuals.

I have also used MouseIsPressed to implement the buttons, but I realize that this makes the selection harder and not as smooth so I plan on changing it to MouseIsClicked to function better.

 

For this particular assignment I wasn’t quite set on the concept. Again, I was more worried about the coding aspect of this assignment that I guess I didn’t have the luxury to be thinking about its aesthetics and data visualization. I do think that my specific way of visualising data was ineffective in the sense that its unable to tell us anything and its unclear what part of the data I’m visualising but nonetheless I used this assignment as a meek approach to solidifying my previous knowledge on object-oriented programming whilst incorporating a new knowledge of json!

Prior to this, I had seen a poster with lines being dispersed outwards similar to that of comics – as a means to depict emphasis. So, I tried to recreate this but ended up with a slightly altered result with a befitting meaning: fireworks. …Now that I’m reevaluating my process in determining an aesthetic by writing this post, I’m currently kicking myself because I just thought of a much better design that would probably make for a more solid conceptual idea which incidentally, would have been easier to code…

I used my previous assignment on object-oriented programming as a reference to creating the code for this piece since it incorporated the use of classes. I created and formatted my own json file data which included a few picks of my favourite songs, which more specifically outlined artist names, song titles and number of replays. I thought fireworks were a good embodiment of the explosion of emotions I felt when listening to music and the only way to cement this would be to link it to the number of replays a song has. The piece of code I’m most proud of is my use of map():

this.minLength = map(replays, 0, 250, 20, 100);
this.maxLength = map(replays, 0, 250, 40, 130);

I had the hardest time linking a specific range of replay numbers to min/max lengths so I ended up spending almost an hour looking for a solution using google, youtube and reddit. Ironically the solution was in my notes the whole time and so whilst it was a frustrating process, I was just glad that it was over and done with. Now comes the improvements…
One thing that I must criticise is the design choice. It isn’t my best work if not the worst.

1) Rather than randomising the position of each firework object, it would’ve been better to give it a static position so that each firework is given enough space for easy visualisation. With randomisation, every time the sketch is refreshed, it could be that the firework objects appear to be on top of each other, making size differentiation difficult. 2) To mimic actual fireworks, each firework object could appear and fade from their designated position. This would continue on as a loop. 3) Each firework object signifies a certain song within the json file, therefore to give it more distinction and meaning, each song could have had its own colour to signify the specific emotion I would feel when listening to the song.

For the week 4 assignment I decided to give a visual representation of the five most successful A24 movies in 2023, by using a bar chart and a csv file containing the 200 most successful movies of 2023. The chart displays the rank of the movie, the total gross, and the title. For the concept I took inspiration from Jacob Rivkin on his video of working with basic data on YouTube.

After reading the paper, what I find most interesting is the methods used for computer vision and how simple they actually are. I always thought computer vision requires a significant amount of different complex algorithms, but it turns out to be as simple as comparing pixels one by one at different times. However, I still have no idea how arrays of pixels (how algorithms interpret images or videos) can be used to gain advanced information such as “tracking the orientation of a person’s gaze.”

I also find it interesting how these techniques can be applied to interactive media art. The way it makes artwork interactive is not making the viewer touch buttons whatsoever, but only move around and see what happens. This method of interactivity gives the viewer more freedom to explore what lies beyond the artwork, instead of just playing within the set of rules the artist sets. It also gives viewers an urge to explore the artwork more because they feel freedom when participating in the artwork. They would want to try out different ways of interaction and see what happens, and that’s what I find most interesting about computer vision used in interactive art pieces.