New Technology Turns Hand Gestures into Music

Graduate Student Johnty Wang demonstrates the gesture control system.
Graduate Student Johnty Wang demonstrates the gesture control system. (Image credit: John Corry and Kevin Doherty, University of British Columbia)

Some people are said to talk with their hands. A professor of electrical and computer engineering at the University of British Columbia decided to make it a literal statement.

Sidney Fels and his team equipped a set of gloves with position sensors that track were they are in three dimensions. Different gestures produce different kinds of sounds. For instance, a closed right hand creates consonants, and opening it creates vowels. Meanwhile the left hand controls the sounds that have "stops" in them -- a B or P (in American English).

PHOTOS: 10 New Gadgets Ready for Their Close-Up

The idea was, in part, to duplicate some of the motions of the throat and tongue when speaking. Vowels are 'open' sounds, made with an open throat and lips, so opening the hand makes those types of sounds.

For an additional level of control, the right hand makes higher-pitched sounds when held up and the horizontal position controls which vowel is sounded out.

The technology can be used to make music, too. Among Fels' co-investigators for this project are UBC School of Music Assistant Professor Robert Pritchard and Johnty Wang, an electrical and computer engineering masters student -- and concert pianist. So far, there have been several public concerts using Fels' invention.

While this technology makes for very cool instruments -- there's a kind of space-age quality to the idea of virtual instruments -- there are other applications as well. Fels noted in a statement that controlling heavy machinery could be adapted to such a system, so that a crane could be moved in a way that is more natural and intuitive.

Jesse Emspak
Live Science Contributor
Jesse Emspak is a contributing writer for Live Science, and Toms Guide. He focuses on physics, human health and general science. Jesse has a Master of Arts from the University of California, Berkeley School of Journalism, and a Bachelor of Arts from the University of Rochester. Jesse spent years covering finance and cut his teeth at local newspapers, working local politics and police beats. Jesse likes to stay active and holds a third degree black belt in Karate, which just means he now knows how much he has to learn.