Transforming Any Flat Surface Into a Control Panel With Sound

New submitter brunozamborlin writes "I just published a short video that shows how a very cheap contact microphone can be used to recognize different types of fingers touch and transform any surface into an interactive board. In the video we put the microphone over different surfaces such as kitchen tables and balloons and through realtime gesture recognition we show how we can play different virtual music instruments using a technique called physical modeling . A mobile version would be definitely possible." The project's Web page shows several more examples. Update: 12/31 15:17 GMT by T : Bruno Zamborlin points out that the surfaces don't need to be flat; instead, they simply need to be rigid.

Comments

Huh?

[frnic]

I don't see that "recognition" is achieved beyond being able to tell that touches were made and were "different" from each other - what those differences were was not demonstrated.

Re:Huh?

[flyneye]

Following the links leads me to a place that says it runs on Mac. (Insert dissapointed/disapproval emoticon of your choice in this space)

Re:Huh?

[datavirtue]

This is just a novel button.One that requires a lot more processing power than even a touch screen that are available cheaply. In addition it is not as flexible as a touch screen in that with a touchscreen you can change the interface and location of the "buttons" at will.

Re:Huh?

[brunozamborlin]

In the video we see that different gestures are mapped to different sounds. For example the coin enables a certain abstract sound whist the bare fingers enables the bass sound and the nails enables the pad... this is done thanks to gesture recognition techniques (a modified version of Hidden Markov Models to work in realtime). Apologies whether it is not clear enough in the video though...

Re:Huh?

[brunozamborlin]

The code is actually standard c++, hope to be able to share it soooon....

Re:Huh?

[flyneye]

Sweet!

Re:Huh?

[ZeroExistenZ]

How do you recognize the sound ?

Pitch? volume? Or really "patterns" (sequence of sound signatures) ?

Fe. can you measure the distances from the microphone of multiple sound signals ?

Re:Huh?

[houghi]

Depending on how precise it can measure where your 'mouse' is I could see the following happening:
1) You build this into a smart phone or tablet PC
2) Calibrate by selecting the 4 corners of your surface.
3) use the whole surface you selected as an extended touch screen.
This could be a bar top, a table at your local fast food, a whiteboard, your desk, tray in a plane, ....

It would allow the screen to be smaller yet the input to be bigger without the need to drag the keyboard with you. You could use a paper with a pre-printed design on it, like a keyboard or a map or whatever. Make it in silk and it is indestructable (well almost) and yet very small. http://duo.irational.org/food_for_free/material_maps/silk_map.html

Re:Huh?

[Amyntas]

Look for Acoustic Pulse Recognition. It's pretty well exactly what you just said.
It uses a series of microphones, usually four of them on a glass pane, to locate contact.

I would personally love such a setup for a glass desk top or keyboard tray, with the letters and buttons lightly etched into the glass. I'm sure creating imperfections in the glass would require a more rigorous calibration, but I believe it would be well worth the trouble.

I can also imagine using more than four microphones would allow for vastly more detailed input, but I really have no clue.

Training?

[Mabbo]

Do you have to give the system training for the specific purpose and gesture? And could the microphone be on the opposite side of the surface? If so, I think I've got an awesome new way to unlock my door.

Re:Training?

[Anonymous Coward]

That could be pretty epic.
Add an elfish desing and a coupe of leds to freak out everyone

Re:Training?

[brunozamborlin]

Yes you can train the system with your own gestures :-)

Re:Training?

[OzPeter]

If so, I think I've got an awesome new way to unlock my door.

*knock* *knock* Mabbo?
*knock* *knock* Mabbo?
*knock* *knock* Mabbo?

Re:Training?

[newcastlejon]

Please, tell me you're not thinking about the barber's knock.

Re:Training?

[cavePrisoner]

Already been done: http://www.youtube.com/watch?v=zE5PGeh2K9k

would it work without earbuds

[Anonymous Coward]

how would you prevent sound feedback loops

Re:would it work without earbuds

[flyneye]

Looks like it just triggers some keyboard/synth program. Presumably you could get some annoying feedback this way by using a contact speaker on the sensing surface. speaker-----O o____contact mic arranged thusly. Perhaps moving the speaker over the area with a bit of pressure could produce cataclysmic glitching as well. How damn cool is that? Maybe it could be ported to *nix so all the modules of the program could be maybe linked up in Jackd. This is O0ober cool and would make a nice performance controller with midi and pd.

Flawed, or useable?

[zAPPzAPP]

I assume that things that sound the same in this demo, would be recognized as the same gesture.
It is shown that doing the same gesture produces the same sounds over and over again, so that gesture would be reliably recognized.

But, this would be extremly dependant on the type of surface and the spot on the surface, that the gesture is performed on. Changing either the spot on the surface, or to another surface entirely, would alter the outcome of the gesture, even though it is actually the same. That is also shown in the video.
How would they work around that?
Maybe by only looking at signals in relation to each other, not at the total signal?

Re:Flawed, or useable?

[Andy_R]

It depends what you want to use it for - as a replacement for a touch screen, it's probably not very useful because location information isn't going to be as easy to capture as timbral information, but as a musical instrument (or as a way of adding more sensitivity to existing electronic instruments, it's amazing.

I wonder if accurate position information could be captured by triangulating with 3 of these contact mics?

Re:Flawed, or useable?

[Andy_R]

Quick back of an envelope calculation... speed of sound 340m/s, sampling rate of most D/A converters 44,100 samples/sec... 7mm accuracy (or better if the speed of sound in the solid object is lower)? Not great, but good enough to turn any solid object into a virtual keyboard.

Re:Flawed, or useable?

[brunozamborlin]

In this system the user can define a gesture by performing it just once. So users can easily define their gestures for every surface in few seconds. The next step would be to automatically recognise that the surface has changed so to avoid to re-train the system for every surface. Thanks for the interest!

Re:Flawed, or useable?

[Sleepyhead5]

The speed of sound happens to be much higher in solids. The more rigid the material, the faster sound will travel through it.

Re:Flawed, or useable?

[CSMoran]

Quick back of an envelope calculation... speed of sound 340m/s,

In vacuum. In solid objects sound is typically an order of magnitude faster -- e.g. sound travels at approx 3400 m/s in wood.

Re:Flawed, or useable?

[Anonymous Coward]

Speed of sound in vacuum? Fail.

Re:Flawed, or useable?

[Anonymous Coward]

In vacuum.

So, in space, people can hear you scream? I guess physicists have been wrong this whole time!

Re:Flawed, or useable?

[PPH]

The speed of sound is much higher in rigid objects. But this has already been done (to a certain degree) with high end electronic drum kits that can vary their response based on the location (radially) at which the pad is struck.

They probably do some hardware DSP at much higher sample rates than 44K samples/sec. But this pushes the solution out of the realm of using a simple mic or two and a PC audio card.

Re:Flawed, or useable?

[CSMoran]

s/vacuum/air/

Yeah, I definitely need another coffee :D.

Re:Flawed, or useable?

[White Flame]

It doesn't appear that the point of this project is to deal with positional information at all, but that it's a musical project where the type of contact the person does with the arbitrary surface is simulated as perturbing some physics-based audio source.

Multiple-mic positioning has been done before in other projects.

Re:Flawed, or useable?

[Anonymous Coward]

Maybe I'm missing something because this isn't an area I've done any research in, but if you can use multiple microphones to capture position information, why would they have to be far apart, e.g. at the corners of the surface? Why couldn't they all be in the same small quarter-size device? Especially if you used echo information?

Re:Flawed, or useable?

[aaronb1138]

I'm pretty sure with just two mics a fixed or known distance apart, you could get 2d coordinates. Just have to look at the timing offset of similar waves and calculate the radial intersection. On the flip side though, small CCD elements with sufficient resolution for a Wii/Smoothboard or Kinect style setup are more than cheap enough to use and relatively compact. A single mic as demonstrated might be useful for broadening the number of unique touch actions though.

Re:Flawed, or useable?

[ResidentSourcerer]

Two mikes give you the difference in distance to the source, if you are doing it just with timing. The solution to a point that is x cm from one mike and x+y cm from another is a hyperbola. Done with radio, and the other way around, you get Loran navigation.

Three mikes can intersect two hyperbolas for a location.

Positional accuracy is limited. Human hearable frequencies top out at about 18 KHz, which in air has a wavelength of about 1.9 cm. At CD audio speed, you can get 44,000 samples per second, which corresponds to about 0.7 cm. To do better than this you have to sample fast enough to get phase information. And to work with the speed of sound in most solids, you have to have another factor of 10 in sampling speed.

However you don't necessarily need high fidelity. I suspect that even 3-4 bit sampling at a megahertz or so would be sufficient.

this is as bad as when they said a new star wars

[decora]

was coming out... then we go to the theatre and its muppets in space crossed with some kind of saturday morning cartoon reject.

cool

[Anonymous Coward]

Anytime someone does something that expands the ways humans and machines interact, I think it's pretty cool.

Re:cool

[PPH]

Mic your PC's case so it can respond appropriately when you smack it out of frustration.

So..

[Anonymous Coward]

Someone hooked up a microphone... Come on... Now 3 of these to track position may be news...

Theremin by a different technique?

[Psychofreak]

http://en.wikipedia.org/wiki/Theremin
Seems like a different input technique...

Phil

Re:Theremin by a different technique?

[brunozamborlin]

:-) A (very) old project I've done in the past was actually "augmenting" a DIY theremin using it as a controller: http://www.youtube.com/watch?v=c5nOK44Z02c

Re:Theremin by a different technique?

[kcitren]

Not even close.

cheers

[Anonymous Coward]

cheers. and thank you for showing computer power + (single) sensor input = impressive. as far as i concern. this is the best example of what 20 years of computer era has given us. and no, that is not ironic. amazing. massive. and practical implementations on the fly. i'm speechless.

Don Cherry has one of these

[cangrande]

http://youtu.be/qdJp5-g69go

[spoiler: it's an amusingly-dubbed video of eccentric Canadian hockey announcer Don Cherry, who wears really loud clothes and makes vigorous table thumping gestures]

New?

[Anonymous Coward]

I'm sure I saw someone do this years ago.

very cool

[Anonymous Coward]

It looks like mapping of sounds to actions. A very cool idea indeed. Like those "corded" keyboards that had physical buttons (you "type" like playing a guitar). I wonder how many (realistically) different cords can be defined - would it be possible to "type?" Imagine sending text messages without having to look where your fingers are on the virtual keyboard! You could type in your pocket (okay - bad idea). The "keyboard" could be the back of the device - more screen for one to view and maybe a more natural grip.

Or maybe not a keyboard, but a menu / action selection device. Video editing (rewind, step whatever) without the need for expensive touch pads. Maybe it isn't practical to turn your table into an input service - but it could be a cheaper "touch" surface.

Cool, but perhaps flawed for most uses

[wanderfowl]

Acoustical modeling to determine the point of origin of sounds is nothing new, and although it's a wonderful idea to implement it as they are here, it'll go to hell the moment there are other vibrations on the surface. A low-pass filter should stop most environmental noise bouncing off the table/surface from triggering it, but if you put down your coffee mug on your desk, or bump your leg to the table, you'll likely get false input. Not to mention, as others have pointed out, the processing costs. This could be another of those technologies which is great for allowing input on an inert and durable psuedo-sterile and wipe-clean surface in a quiet, controlled room, but likely won't be worth much outside of bizarre use cases like that. But it's still amazing research.

I was looking for a video like this

[mshenrick]

Years ago, before youtube was really popular, I saw a video of a man doing a similar thing but couldn't find it recently!

I see, its a less useful version of this

[james_van]

saw stuff like this years ago. cant find the exact tech demo that i originally saw that used 3 mics for triangulation, but this video is pretty close. even mocked up a rough version (low precision, slow response time, but pretty damn good for an afternoon worth of work and very little experience with arduino) tied to a projector in my shop. customers had a blast with it, thought it was the coolest thing in the world. http://www.youtube.com/watch?v=CxDHiwheK8w&feature=related

Source code or binary release?

[Anonymous Coward]

Very cool idea!

Was there a source code or binary release planned? I'd certainly like to play with this software!

How much of this is real?

[Animats]

I have some suspicions about this. They're getting more information out of one microphone than is usually possible. You might be able to extract some positional information by picking up the echoes off the edges of the object.

Similar student project at Cornell

[bondiblueos9]

This student project from Cornell a couple years ago uses a similar idea: http://people.ece.cornell.edu/land/courses/ece4760/FinalProjects/s2009/gh96_jbw48/gh96_jbw48/index.html

Keylogger?

[Dwonis]

Hmm. I wonder how accurate of a keylogger could be made with something like that...

Saw this before

[Anonymous Coward]

I saw this (although for a different purpose) years ago here.

piezo mic application

[Anonymous Coward]

Six months ago, available now, free.

http://cycling74.com/practical-max/practical-max-1/

video
http://www.youtube.com/watch?v=CSopUi9-pUg&feature=player_embedded

Scratch Input?

[Anonymous Coward]

Seems very similar to Harrison's Scratch Input, shown at SIGGRAPH in 2009...

http://www.chrisharrison.net/index.php/Research/ScratchInput