The Concept
For my midterm project, I want to make a walking game controlled by sounds the player records. Instead of assigning fixed voice commands, the game will let each player choose a short sound for Up, Down, Left, and Right. A player could use spoken words, clicks, whistles, or other distinct sounds. I like the idea that the controls become personal to the player, and that learning to move is part of the experience.
User Interaction
The player will first enable their microphone and record one sound for each direction. Once all four sounds are saved, they can press “Start listening” and play using the microphone. Each recognized sound moves the character for a chosen number of seconds. The player can adjust that duration with a slider and re-record a sound if it is difficult to recognize.
Movement on the two axes works independently. For example, if the player makes the Up sound and then makes the Left sound while Up is still active, the character moves diagonally. The arrow keys also work, which gives me a way to test movement separately from the microphone.
Code and Design Progress
I started with a p5.js prototype of a walking character. It loads a sprite sheet, separates it into animation frames, and displays the correct row for each direction. The character animates while moving and stays within the canvas.
I then added the microphone setup, recording buttons, sound recognition, and timed movement. The code samples a sound’s frequency profile, divides it into three parts over time, and compares it with the four recorded signatures. It rejects sounds when the match is too uncertain. I also changed the movement code to keep separate horizontal and vertical timers, which allows two directions to remain active together.
Most Frightening Part
The most uncertain part is sound recognition. A person will not make a sound exactly the same way twice, and background noise could cause a false command. Rapid commands are another challenge: the program needs to notice where one sound ends and the next begins.
To reduce this risk, I built the recognition system before adding more game features. The prototype normalizes the frequency profile so a louder or quieter version of a sound can still match. It waits for a short silence to finish recording a command and rejects ambiguous matches. This gives me a working system to test with real sounds, although I still need to find out how reliably it performs in different rooms and with different players.
Next Steps
My next step is to test several sets of four sounds and record which commands are missed or confused. I can use those results to adjust the detection thresholds and the pause between commands. After the controls feel reliable, I want to design a space for the character to navigate so that the audio movement has a clear goal.
