I have three ideas for my final project, each emerging from my interdisciplinary interests.
The first idea I would like to propose is creating a Sign Language Glove, aimed at facilitating communication and improving accessibility for individuals who are deaf or hard of hearing. I shall limit it to fingerspelling using the alphabet for now. The glove will incorporate flex sensors on each finger to detect bending movements. Arduino will process this data and send the finger configurations to a p5.js sketch, which will interpret the gestures and recognize the corresponding letters of the alphabet.
The p5.js screen will display the recognized letters visually and audibly using text-to-speech. Additionally, users will have the option to input letters via a keyboard to display the corresponding Sign for it on the screen. This interactive system enables individuals that use sign language to have two-way communication with non-sign language users effectively.
I initially thought of using American Sign Language (ASL), but the issue is a lot of the signs have the same finger positions, and it will be difficult to differentiate the signs.
An alternative is using Indian Sign Language, which uses two hands, but can overcome the above issue. However, this adds complexity of checking 10 finger configurations.
My second idea is conducting a psychology experiment utilizing p5.js for the visual presentation of stimuli and Arduino for participant response collection. I aim to design either Perception experiments, such as Visual search tasks, which involve participants searching for a target stimulus among distractors, or Cognition experiments, which may involve memory tasks, where participants memorize and recall sequences of stimuli presented, or face recognition tasks, where participants identify familiar faces. In these experiments, the p5.js sketch will display visual stimuli, while Arduino buttons will serve as response inputs.
Eg, in the visual search tasks, the p5.js screen will display each of the trials and participants will use buttons connected to the Arduino board to indicate when they have found the target stimulus. Arduino will record response times and accuracy.
At the end of the experiment session, participants will be able to view their performance metrics and compare them to group averages or previous trials. This setup allows for the seamless integration of psychological experimentation with interactive technology, facilitating data collection and analysis in a user-friendly manner.
For my third project idea, I propose creating an interactive system that generates music from art! The user will be able to draw on the p5.js canvas, creating their unique artwork. The system will then analyze this artwork pixel by pixel, extracting the RGB values of each pixel. These RGB values will be averaged to create a single value for each pixel, which will then be mapped to a musical note. Additionally, the system will detect sharp changes in color intensity between adjacent pixels, indicating transitions in the artwork. These transitions will determine the length of each note, with sharper changes resulting in shorter notes. The coordinates of each drawn point can influence the tempo or volume of the music, to make it sound better. Once the music composition is generated in p5.js, it will be sent to Arduino, where a piezo buzzer will play the music in real-time. This interactive system lets users create their own art and music.