New Technology Turns Hand Gestures into Music

Graduate Student Johnty Wang demonstrates the gesture control system. CREDIT: John Corry and Kevin Doherty, University of British Columbia View full size image

Some people are said to talk with their hands. A professor of electrical and computer engineering at the University of British Columbia decided to make it a literal statement.

Sidney Fels and his team equipped a set of gloves with position sensors that track were they are in three dimensions. Different gestures produce different kinds of sounds. For instance, a closed right hand creates consonants, and opening it creates vowels. Meanwhile the left hand controls the sounds that have "stops" in them -- a B or P (in American English).

PHOTOS: 10 New Gadgets Ready for Their Close-Up

The idea was, in part, to duplicate some of the motions of the throat and tongue when speaking. Vowels are 'open' sounds, made with an open throat and lips, so opening the hand makes those types of sounds.

For an additional level of control, the right hand makes higher-pitched sounds when held up and the horizontal position controls which vowel is sounded out.

The technology can be used to make music, too. Among Fels' co-investigators for this project are UBC School of Music Assistant Professor Robert Pritchard and Johnty Wang, an electrical and computer engineering masters student -- and concert pianist. So far, there have been several public concerts using Fels' invention.

BLOG: Device Organizes Your Music By Moods

While this technology makes for very cool instruments -- there's a kind of space-age quality to the idea of virtual instruments -- there are other applications as well. Fels noted in a statement that controlling heavy machinery could be adapted to such a system, so that a crane could be moved in a way that is more natural and intuitive.