Hello! I remixed Notwaldorf's "Piano Genie" so that you can play music by pinching together your fingers. Piano Genie is a Glitch app which lets you mash keys together but makes them sound good using machine learning through Magenta.js which allows me play multiple notes with a single finger.

It only took me 15min or so to remix this with Handsfree.js.org, my library that helps you implement face, body, and hand tracking in the front end super duper fast. If you check out the source, I've linked to each of the specific documentation pages for those methods.

Basically all I did was:

// Create a "plugin" called fingerKeys that runs on every webcam frame
handsfree.use('fingerKeys', ({hands}) => {
// Bail if no hands are detected
if (!hands.pinchState) return

// hands.pinchState[x][y]
// [x]: 0 = left hand, 1 = right hand
// Left
if (hands.pinchState[0][3] === 'start') {pinchKey(1)}
if (hands.pinchState[0][3] === 'released') {releaseKey(1)}

// etc.
})

Been working on this for 3 years but I wasn't sure how to introduce this to r/WebDev so here I am lol If you'd like to read about what inspired me to start working on this while I was homeless I wrote this Twitter Thread here: https://twitter.com/MIDIBlocks/status/1359976809924726786