Concept & Interaction
What I love more than horror games, is psychological story-based visual novels. Games that hold you in place, extremely focused and afraid to even blink to miss something important (or something that will get you in trouble). Also, I really love when innocent and soft, childlike things are framed in the way that makes you really uncomfortable, creating a really two-sided feeling of nostalgia and comfort with unsettling disturbance.
More than that, for a very long time I wanted to try experiment with computer vision and body capturing, so I decided to combine these two things in my midterm.
What I want to do is a game that is controlled by user’s video input. In Russia, we play a clapping game called “Ладушки” (ladushki; I believe in English it’s called Patty Cake), where you need to match the rythm of the other person clapping, as well as their hands (right to right, left to left, two hands to two hands). I want to make the user to play this game with the computer. There will be a girl in a room that will welcome the player to play this game with her. Her clapping will be consequentive, and player just has to match her hands state with their hands.
The twist is that if the player fails to match the girl’s rythm and hands state, she will get angry. And as the user makes more mistakes, she will get more angry. With the anger level increasing, the whole picture and game will become distorted: video glitching (in later phases: disappearing), rythm becoming unstable and/or much faster, unfair hand detection, intentional mistakes in detecting state of the hands, sound becoming distorted, phrases girl says after mistakes turning to be aggressive, her appearance shifting as well. If the user makes it to the anger level = 100, there will be a jumscare with their video distorted (I figured that out of all jumscares it will make the most impact).
Some details about the concept might be found as comments in my code, and the general outline is planned to be like in the picture below. To create creepy atmosphere, I plan to use really soft colors and cute artstyle that won’t match the gameplay and plot.
Code Outline
Right now I decided to focus on the technical side of the project, making the algorithm work, so after this I will have the core mechanic. After that, I will focus on visual and sound design: drawing sprites, finding suitable sounds, creating glitching effects, scales, text etc.
This is the plan of how I (as of now) think the code should include:
- class Girl with one object initiated and the following methods:
- talking method (reaction when user fails to match the girl’s tempo)
- changing hands states method
- comparing Girl’s to user’s states method
- drawing girl method (sprites needed)
- anger level scale draw method
- Functions in the general code block:
- detecting user’s handpose function
- displaying user’s video
- video distortion function
- sound implementation + sound distortion methods
- final video screamer function
- Assembled game in setup() and draw() with restart option (maybe pause + exit buttons ?)
Code Made
Sketch requires camera access!
This is my sketch so far and the code I made. As I said, now I’m focusing on the technical part. Now, the code can:
- detecting user’s handpose function
- display user’s video in the corner
- talking method (reaction when user fails to match the girl’s tempo) (Girl Class)
- changing hands states method (Girl Class)
- comparing Girl’s to user’s states method (Girl Class)
Instead of drawing, the code now just outputs the anger level and the state of the girl’s hands. The code compares the video input and user’s hands position with girl’s hands state. When user makes mistake, anger level is increased by 10 and there’s text displayed on the screen (3 phrases for each sector: 4 sectors depending on anger level). However, it isn’t staying on it now (to be fixed). Also, the game loop stops once annger level reaches 100.
The base code for hand detection function implementation is copied from ml5 HandPose reference page. I also used this The Coding Train video about Hand Pose.
Complex Part
I believe that the most difficult part of my midterm is working with video input and hand detection. It’s a pretty new concept for me, as well as it’s pretty hard to implement it to use not just as a small interactive component but a core concept around which the game is built. Risk of improper detection of pose, poor video input and glitching is quite high. However, I tried to build this part first and it turned out to not be too difficult. After testing my code for some time, I defined three poses the computer should recognize: two hands open, left hand open, right hand open. Perfectly, to fit my concept and the Ladushki gameplay, I also need to have a pose for clap, but the problem is that when hands are clapped and face their edge to the camera, hand detection disappears. Since it could possisbly break the game and unfairly detect it as user’s mistake when it’s not, I decided to ignore this state fully and check only for claps to the camera, when palm is facing the computer.
Also, to avoid random poses appearing and being detected, I added a confidence level: if the confidence in detection of hands is lower than this set level, the computer won’t register it as a pose. This really helps a lot to not identify some random pose/made up movement as the user’s mistake.
Now, the most challenging part for me would be the visual design. I don’t have much experience in this type of creative work unlike in coding, so creating sprites and building environment that will serve for the goal of the game and suit its atmosphere, as well as arranging everything properly and not overloading, will be a bit hard for me. To check on my progress here and to track the result of the aesthetic impact and suitability, I would ask my friends to give me feedback and interact with AI so it can give me some more theoretical rules about how things should be arranged in final outcome according to some basic design rules.
I believe that this project is really fun and much easier to make than I expected, since the hardest part was mostly completed already!
Some of my inspirations for design, concept and aesthetic are: Needy Streamer Overload and DDLC