This weekend, Tyler Williams, Aidan Feldman, and I formed a team for New York’s Music Hackday. Our goal was to build a musical instrument with the Kinect’s 3d camera. I bought a Kinect and we opened the box for the first time Saturday morning.
Stringer is a virtual string instrument played with a Kinect 3d camera. A person can draw his/her own instrument in the air using gestures. Multiple people can play at once, through separate hand-tracking. Strings can be set to morph as people interact, with length determining pitch, creating a constantly changing melody.
We started purely in Processing, using Daniel Shiffman’s library. We had fun working around some of its limitations, with simple, blob-like threshhold tracking. We built a prototype that might work as a sidewalk installation. Pedestrians would trigger a shifting set of strings, each playing a unique melody.
Then Tyler got the more advanced C++ libraries talking to Processing over OSC, and that was things got really interesting. We were able to track hands very accurately, so even multiple sets of people could play at once.
Later on, Aidan added drawing gestures through a simple but elegant hold-to-draw motion. The user could then draw new lines in the air.
We tackled this as a proof of concept, and definitely have many more ideas. I also have many more videos and will update this blog post soon. A few more pictures in this Flickr set.