Slashdot Mirror
Video Chat Via Transparent Desktop Overlay
Jason0x21 writes "Wired News has an article about UNC Comp. Sci. researchers developing a transparent desktop overlay for video conferencing, allowing remote coworkers to literally point and interact with things on your screen. The researchers say that Apple's Quartz graphics engine let them go from idea to prototype in 'about 45 minutes'. Windows versions predicted in the future."
Windows has had the ability to draw transparent windows since 2000. However, there's a limit to how far they can go.
Particularlly, you can't do any blending against windows that are being drawn with DirectX/DirectDraw which is the way that any program that wants to approach full-motion video or 3D graphics has to do things. And that's what prevents Windows from handling this application.
Mac's OSX is a lot cleaner in this department because in their universe there are no exceptions to the rules... everything passes through Quartz, so there's a chance to capture and play with anything on the screen. DirectX and DirectDraw are painted onto the screen after all mortal windows are drawn in Windows, and that's why there's no chance to add an overlay to them.
ollaborating with co-workers in the same office is painful enough, but it's nigh impossible over a network.
For a couple of decades, researchers have tried to blend shared workspaces -- systems that allow two or more people to work on the same document -- with Internet video-conferencing systems, with little success.
Now researchers at the University of North Carolina at Chapel Hill have designed a new system that cleverly blends a video-conference feed with a transparent image of a computer desktop into one full-screen window.
Called Facetop, the system simultaneously transmits a video feed of users along with a shared, transparent image of the desktop. It allows two colleagues to work on the same document, Web page or graphic, while communicating face to face.
The system also tracks the position of the users' fingertips, which can control a cursor. As well as operating the shared desktop -- opening and closing files or selecting text, for instance -- the collaborators can use natural pointing gestures to communicate ideas about the document.
Developed by David Stotts, an associate professor of computer science, and graduate student Jason Smith, Facetop was conceived for collaborative tasks like programming or editing text. But the researchers say it has obvious uses in other areas such as medical imaging or remote teaching.
"So far, from the feedback we've received, it works fantastically," said Smith. "It's a very natural interaction. You can see the facial expressions and all the nuances of face-to-face communication."
"It's spectacular technology," said Robert Gotwals, associate director of Chapel Hill's Morehead Planetarium and Science Center, who saw a demonstration of an early version. "I've done lots of video-conferencing work. This is pretty cutting edge. It's a fast-moving field and the stuff David (Stotts) is doing is pretty cool."
The system can also be used for delivering lectures or PowerPoint presentations: The speaker is projected in the background of the document allowing her to point out bullet points or important passages. According to Smith, users easily switch attention between the subject and the desktop.
"The brain is really good at picking out what part of the screen the person is interested in," said Smith. "It's like being in a room full of conversations but having no trouble paying attention to only one.... People adapt to the system really naturally."
Facetop may also be used to as an alternative to the mouse, for controlling a machine simply by pointing with a finger.
The system is implemented in Mac OS X and is made possible largely by the system's Quartz rendering engine, which can make any part of the interface transparent. Thanks to Quartz, a quick prototype was whipped up in about 45 minutes, Smith said.
A PC version will likely be delayed until the release of Longhorn, the next major version of Windows, due in 2006, which will include a similar graphics subsystem.
The system is fairly inexpensive; it has been implemented on a pair of Apple PowerBooks and two $100 FireWire cameras. So far it has been tested only on Ethernet networks and not the Internet, though the researchers say there's no reason it shouldn't work just fine. They are also trying to hook it to Apple's iChat instant-message/video-conferencing software and other similar systems.
Facetop was initially developed for "pair programming," an increasingly popular form of collaborative coding that pairs programmers in teams of two: one to program, the other to suggest and correct. Stotts said programmers normally sit next to each other, and he has been interested for some time to see whether they could collaborate over the Internet.
According to Stotts, pair programming -- sometimes called extreme programming -- is fast and effective and is becoming increasingly popular for small projects.
The idea for Facetop occurred to Stotts and Smith accidentally. Instead of a computer monitor, Stotts projects his
What i'd like to see is a voice controled program, instead of hurridly bending down to click the mouse at a conference, you simply say 'back', 'foward', 'pause', or even program in new words through a macro system built into the program. Oh and don't try and steal it, thats my damn intelectual property now, hah!
There are photos on the article
w w.wired.com/news/images/full/cimg0401_f.jpg
http://a1112.g.akamai.net/7/1112/492/2002091464/ww w.wired.com/news/images/full/cimg0407_f.jpg
http://a1112.g.akamai.net/7/1112/492/2002091464/ww w.wired.com/news/images/full/imgp0173_f.jpg
http://a1112.g.akamai.net/7/1112/492/2002091464/w
(\(\
(^.^)
(")")
*This is the cute bunny virus, please copy this into your sig so it can spread
here's an Endeavors article about the project at UNC
FaceTop
The easy fix for that is to mirror what the other person sees so that it's the right way around for them, and then mirror the image of the person as displayed on the screen so that their hand movements match what you're looking at on your screen.
This concept was extensively researched by Hiroshi Ishii and his team between 1991 and 1994 while he was at NTT.
I saw the concept videos in my HCI class at the time. They went through all the various issues of pointing alignment, video flipping and the like.
Fine, i'll do Slashdot and Wired's jobs for them:
Screenshots
--
Mod up a post Rob doesn't like and you'll never mod again
-Ian
Erm, this refers to computer desktop. And since I don't believe that computer desktops can have books or papers randomly around the screen, I think you have the wrong idea.
"Your effort to remain what you are is what limits you."
No, he's talking about what's behind you in the camera's field of view (a bunch of junk) being overlaid on your co-worker's computer desktop during the conference.
Nope, that's a misconception we're seeing pop up from time to time. This is not like standing on either side of a pane of glass... this is like sitting side by side with the other user.
I just put up a stupidly simple FAQ of sorts at http://www.cs.unc.edu/~smithja/facetop/index.html and will be updating it this morning.
Seeing some misconceptions, tossed up a quick FAQ at http://www.cs.unc.edu/~smithja/facetop/index.html for your perusal.
I'll be adding material to it through the morning as issues pop up, but these are the ones we've seen the most of this weekend.
Actually, the camera can be anywhere, as long as you're in the field of view.
As for ease of use, it literally takes people about two seconds to calibrate their hand motions to the cursor movement, and they're off and running. It's exactly like you're standing in front of a mirror (assuming the camera is in front of you), and gesturing... the visual feedback you get from your own image is the key. The transparency lets you see both your 'reflection' and the document content simultaneously.
Don't worry, we're seeing a lot of people confusing the single-user mode (one head on screen) with the video-conferencing mode (two heads on screen), simply because they're not used to video conferencing including themselves.
No, not really. What windows can do Is a far cry from what this technology offers. The windows remote assistance feature lets another user take over your machine and operate it as if it were their own. You can see what they are doing on your screen. However, there is certainly no projection of that person on your screen allowing you to see them physically touch and interact with your system. This adds a whole new level of interactivity.
Like you said, this isn't a Windows Mac pissing contest. Its a cool new technology, might as well face it.
Actually, our experiments have found that it really doesn't matter.
First off, the translucency is adjustable. Looks too cluttered? Make it more faint. Secondly, it's much like being in a room full of conversations at a party - you select particular conversations to pay attention to, and the rest just 'fade away'. In this case, when the user turns their attention to the document content, they don't notice the video, and when they concentrate on the video (either for hand motions or interaction with a remote user), the document content is ignored. The brain is much better at this sort of thing than most people realize.
Sadly, no. Forty-five minutes was about the time it took me to write the initial proof of concept, not the full application. (That included reading the documentation on various APIs.)
But yes, Cocoa made it much easier to do so.
I'll take that as a compliment. ;)
Yes, that's precisely why it works - it's like you're looking into a mirror. Raise your hand to move the cursor up. Move it left, cursor goes left. It's just that simple.
I've tried it and it's pretty cool. It's great so see the expression on your opponents face when you roll your army of tanks into his left flank when he's least expecting it.
I'll take that as a compliment on our making it look like you're standing in front of a mirror.
;)
That's rather the *point*.
Trust me, it would have been much easier to take a picture of a reflection on the screen surface than develop the bloody thing.
Uh, a lot of work is done via remote locations nowadays.
For example, the current project I'm working on involves 3 companies spanning 5 or 6 states in the US. We can't just meet face-to-face without spending a hell of a lot of money.
And next week I'll be working on something else for a completely different company.
This is how stuff gets done. It allows talent from all over the globe to be used for minimal cost.
The ratio of people to cake is too big
As with everything these days, the IP is... interesting. As UNC employees, we're not allowed to just start tossing the tech around willy-nilly, UNC owns the IP. They have applied for a patent on this, and are looking at a licensing scenario for those who would like to commercialize it. (Or include it in their OS *cough*.)
It would be nice to just disseminate the thing, but I don't legally own it to do so.
D'oh, sorry.... no sleep makes me slack.
;)
;)
Without video conferencing, you lose a lot of the nuances of human communication.
Stotts' work in distributed pair programming almost invariable resulted in "We really wish we could *see* them..." which led to video conferencing, which led to "We still have to verbally describe things we could just point to."
Also, VNC clones the same desktop to both machines - which makes traditional video conferencing... difficult.
With our approach we can clone the entire screen if you really want, but our current system lets you choose which *documents* you're sharing. The other user doesn't see everything on your screen, only work you are sharing.
I'm sure you can appreciate the utility of that.
Apple has this feature not because they had some great new insight, but they actually just got it essentially for free with the PostScript-based window system they acquired from NeXT, which was designed some time in the early 1980's and is based on stuff from Adobe.
What Apple got for free was Display PostScript, which Adobe didn't want to support (for good reasons). Quartz is based on PDF, not PostScript, and Apple licensed the semantics, not the implementation. Quartz was designed and built by Apple engineers from scratch. The initial version did not have hardware acceleration. With Quartz Extreme they integrated Quartz compositing with OpenGL, leveraging the very extensive work the Mac OpenGL team (previously Conix) put into hardware acceleration. All of this took a ton of work. It definitely was not "for free".