Slashdot Mirror
Video Chat Via Transparent Desktop Overlay
Jason0x21 writes "Wired News has an article about UNC Comp. Sci. researchers developing a transparent desktop overlay for video conferencing, allowing remote coworkers to literally point and interact with things on your screen. The researchers say that Apple's Quartz graphics engine let them go from idea to prototype in 'about 45 minutes'. Windows versions predicted in the future."
One of my favorite pieces of technology I've ever gotten a chance to play with is the SmartBoard Interactive Whiteboard It's a whiteboard that's touch-sensative. Basically, combine it with your favorite projection monitor and you've got a 60 inch touchscreen monitor. Just like any other touch screen, anywhere you tap the board is treated like a mouseclick in whatever application you're using. As an added bonus, "magic crayons" (really nothing more than plastic styluses) are at the bottom of the board. When the board detects one of the pens removed from its holder, it treats all touches as requests to draw on the screen.
:)
It's a great presentation tool to liven up a powerpoint and avoid the need to have to walk accross the room to get the next frame. Furthermore, playing solitare with foot-high cards is quite fun.
Windows has had the ability to draw transparent windows since 2000. However, there's a limit to how far they can go.
Particularlly, you can't do any blending against windows that are being drawn with DirectX/DirectDraw which is the way that any program that wants to approach full-motion video or 3D graphics has to do things. And that's what prevents Windows from handling this application.
Mac's OSX is a lot cleaner in this department because in their universe there are no exceptions to the rules... everything passes through Quartz, so there's a chance to capture and play with anything on the screen. DirectX and DirectDraw are painted onto the screen after all mortal windows are drawn in Windows, and that's why there's no chance to add an overlay to them.
Is cocoa more responsible for fast development prototyping? Quartz is just an API. And I'm pretty sure there isn't a NSVideoConferenceWindow () in it.... or is there?
ollaborating with co-workers in the same office is painful enough, but it's nigh impossible over a network.
For a couple of decades, researchers have tried to blend shared workspaces -- systems that allow two or more people to work on the same document -- with Internet video-conferencing systems, with little success.
Now researchers at the University of North Carolina at Chapel Hill have designed a new system that cleverly blends a video-conference feed with a transparent image of a computer desktop into one full-screen window.
Called Facetop, the system simultaneously transmits a video feed of users along with a shared, transparent image of the desktop. It allows two colleagues to work on the same document, Web page or graphic, while communicating face to face.
The system also tracks the position of the users' fingertips, which can control a cursor. As well as operating the shared desktop -- opening and closing files or selecting text, for instance -- the collaborators can use natural pointing gestures to communicate ideas about the document.
Developed by David Stotts, an associate professor of computer science, and graduate student Jason Smith, Facetop was conceived for collaborative tasks like programming or editing text. But the researchers say it has obvious uses in other areas such as medical imaging or remote teaching.
"So far, from the feedback we've received, it works fantastically," said Smith. "It's a very natural interaction. You can see the facial expressions and all the nuances of face-to-face communication."
"It's spectacular technology," said Robert Gotwals, associate director of Chapel Hill's Morehead Planetarium and Science Center, who saw a demonstration of an early version. "I've done lots of video-conferencing work. This is pretty cutting edge. It's a fast-moving field and the stuff David (Stotts) is doing is pretty cool."
The system can also be used for delivering lectures or PowerPoint presentations: The speaker is projected in the background of the document allowing her to point out bullet points or important passages. According to Smith, users easily switch attention between the subject and the desktop.
"The brain is really good at picking out what part of the screen the person is interested in," said Smith. "It's like being in a room full of conversations but having no trouble paying attention to only one.... People adapt to the system really naturally."
Facetop may also be used to as an alternative to the mouse, for controlling a machine simply by pointing with a finger.
The system is implemented in Mac OS X and is made possible largely by the system's Quartz rendering engine, which can make any part of the interface transparent. Thanks to Quartz, a quick prototype was whipped up in about 45 minutes, Smith said.
A PC version will likely be delayed until the release of Longhorn, the next major version of Windows, due in 2006, which will include a similar graphics subsystem.
The system is fairly inexpensive; it has been implemented on a pair of Apple PowerBooks and two $100 FireWire cameras. So far it has been tested only on Ethernet networks and not the Internet, though the researchers say there's no reason it shouldn't work just fine. They are also trying to hook it to Apple's iChat instant-message/video-conferencing software and other similar systems.
Facetop was initially developed for "pair programming," an increasingly popular form of collaborative coding that pairs programmers in teams of two: one to program, the other to suggest and correct. Stotts said programmers normally sit next to each other, and he has been interested for some time to see whether they could collaborate over the Internet.
According to Stotts, pair programming -- sometimes called extreme programming -- is fast and effective and is becoming increasingly popular for small projects.
The idea for Facetop occurred to Stotts and Smith accidentally. Instead of a computer monitor, Stotts projects his
Where are the screenshots? - seems logical to post considering it only took 45mins for a prototype.
Why boast how easy it was to get it happening then not showing it happening?
odd
I had an internship at DoE lab outside of Chicago, Argonne Natl. Laboratory, at which we worked on a project similar to this. The system allowed multiple users (of various geographic, or digital distances) to connect to a Desktop Server, on which all users could interact with icons, windows and programs in tandem as if they interfacing with a local deskptop in windows. Althou, we used BeOS as our platform because it had a small footprint. Interesting that three years later private companies have out-done the DoE's work. Sad.
What i'd like to see is a voice controled program, instead of hurridly bending down to click the mouse at a conference, you simply say 'back', 'foward', 'pause', or even program in new words through a macro system built into the program. Oh and don't try and steal it, thats my damn intelectual property now, hah!
here's an Endeavors article about the project at UNC
FaceTop
What if your background isn't completely black or a solid color I know my office back ground isn't a flat color its a bunch of books and papers and folders not so neatly organized that overlaided on a coworkers desktop would really add to the confusion.
This must be Thursday, I never could get the hang of Thursdays.
!siht ekil skool gnihtyreve weiv fo tniop rieht morf ,yletanutrofnU
Show me on the doll where his noodly appendage touched you.
So, let's say those clever folks over at whatever-Gator-calls-themselves-now gets the brilliant idea that they could download one of them thar transparent-overlay-thingies whenever you browse to random-evil-webpage. Then, whoosh! They can sell remote access to your desktop so that advertisers can move all the annoying icons out of the way so that you can see the advertising more clearly. Or whatever. An since the overlay is transparent, the user can't figure out what is happening and simply thinks their system is posessed by the devil.
Assuming you have a touch screen, Windows has been able to do this since the release of Windows XP, using the remote assistance feature. Also, for the record, I hate getting into "I can pee this far, how far can you pee" debates. I just felt the need to reassert my "Windows shill" status by posting this ;-).
iRooster, the Mac OS X a
Heaven forbid that people should actually have to talk to each other face to face!
This concept was extensively researched by Hiroshi Ishii and his team between 1991 and 1994 while he was at NTT.
I saw the concept videos in my HCI class at the time. They went through all the various issues of pointing alignment, video flipping and the like.
Fine, i'll do Slashdot and Wired's jobs for them:
Screenshots
--
Mod up a post Rob doesn't like and you'll never mod again
-Ian
Business to business relationships have already become so depersonalised. This is just the next logical step - advancing technology that allows people to sit on their chairs to help other people. Heaven forbid that you would have to get up from your desk to help somebody!
Perhaps this kind of überchatting software is THE place where they can use those 3D desktop environments / window managers.
I don't really know if it would be useful, but perhaps it is cool to lean a window so you can see your partner while keeping an eye on the app content.
Anyway, beeing so far from the world as *I* am (yep, there are places on the south of the globe), where the bandwidth is kinda expensive, i can tell that i'll not be using this kind of technology for a while...
--krahd
mod me up scottie!
One possible feature for them to implement: one party can "flip the bird" to restart the whole session (as opposed to ALT-F4 or CTRL-C or whatever), thus giving new meaning to "Giving your co-worker the finger" for bad suggestions.
The Wknd Sessions - Malaysian and South East Asia independent music
Unless your camera sits exactly behind your monitor, (i.e. your monitor screen is transparent/one-way)
the image of your hand (despite the touch screen) on your collaborator's screen has to be computer generated. Or am I missing something here?
If the hand is CG, then all we have is a glorified cursor (but this too would be a pretty good hack if they got it done in 45 mins).
But wait, those pics don't seem to show the hand pointing in the right direction either!
Comment removed based on user account deletion
Seeing some misconceptions, tossed up a quick FAQ at http://www.cs.unc.edu/~smithja/facetop/index.html for your perusal.
I'll be adding material to it through the morning as issues pop up, but these are the ones we've seen the most of this weekend.
Actually, the camera can be anywhere, as long as you're in the field of view.
As for ease of use, it literally takes people about two seconds to calibrate their hand motions to the cursor movement, and they're off and running. It's exactly like you're standing in front of a mirror (assuming the camera is in front of you), and gesturing... the visual feedback you get from your own image is the key. The transparency lets you see both your 'reflection' and the document content simultaneously.
Don't worry, we're seeing a lot of people confusing the single-user mode (one head on screen) with the video-conferencing mode (two heads on screen), simply because they're not used to video conferencing including themselves.
I'll take that as a compliment. ;)
Yes, that's precisely why it works - it's like you're looking into a mirror. Raise your hand to move the cursor up. Move it left, cursor goes left. It's just that simple.
I've tried it and it's pretty cool. It's great so see the expression on your opponents face when you roll your army of tanks into his left flank when he's least expecting it.
Windows has had the ability to draw transparent windows since 2000. However, there's a limit to how far they can go.
Transparency in window systems is an idea that goes back almost as long as window systems have been around. People were even asking for it in the earliest versions of X11.
The only reason it hasn't been implemented more widely is because hardware hasn't really been up to it and applications didn't need it. Those applications that really did need it just used special graphics and visualization libraries.
Apple has this feature not because they had some great new insight, but they actually just got it essentially for free with the PostScript-based window system they acquired from NeXT, which was designed some time in the early 1980's and is based on stuff from Adobe. And now that hardware is up to handling it, it will just be a standard component of desktop window systems.
As with everything these days, the IP is... interesting. As UNC employees, we're not allowed to just start tossing the tech around willy-nilly, UNC owns the IP. They have applied for a patent on this, and are looking at a licensing scenario for those who would like to commercialize it. (Or include it in their OS *cough*.)
It would be nice to just disseminate the thing, but I don't legally own it to do so.
This looks similar to the eyetoy for PS2. It works the same way.
This is a better use of the idea in my opinion. I'd like to use this to replace a mouse, plus the collaberation use looks great. Kudos to the ones who put it together!
Can we do this with X11?
AB HOC POSSUM VIDERE DOMUM TUUM
If you don't have stickers, make a few small circles on the mirror using your girlfriends lipstick.
Now step about 3 feet away from the mirror.
Move your finger so that when you look in the mirror, it looks like you are touching the stickers but you don't physically do so, it just looks like you do to your eye.
Notice that you can do this regardless of your angle to the mirror, you just have to adjust your finger.
Now imagine that the stickers are icons on your desktop
and VIOLA!
D'oh, sorry.... no sleep makes me slack.
;)
;)
Without video conferencing, you lose a lot of the nuances of human communication.
Stotts' work in distributed pair programming almost invariable resulted in "We really wish we could *see* them..." which led to video conferencing, which led to "We still have to verbally describe things we could just point to."
Also, VNC clones the same desktop to both machines - which makes traditional video conferencing... difficult.
With our approach we can clone the entire screen if you really want, but our current system lets you choose which *documents* you're sharing. The other user doesn't see everything on your screen, only work you are sharing.
I'm sure you can appreciate the utility of that.
If I get this correctly--and it appears most everyone else does not--I see your alpha-blended image as if you were sitting in my place; similarly, you see mine. Not flipped, or virtualized, but 'reflected' as if we were sitting in each other's place. Anyone who has worked developing and using and testing collaborative solutions will recognize that there is, potentially, a real AHA! here. VNC, NetMeeting and WebEx and all that clever crap is limited or useless, except in the hands of sophisticated people doing support or in very narrowly defined lecture settings, because these applications all abstract the notion of the absent party. You know I am moving the cursor on your machine because on your screen it has a red box around it, while I see an unboxed cursor. I may be talking to you over a voice channel, and we learn to abstract a collaborative session in our heads from voice and visual cues. If you think ordinary mortals can learn and map that cue-based interaction to natural behavior so they can just work together on a document then you labor in ignorance. The AHA! here is that the absent party is not abstracted, they are substituted via alpha-blending. Hard to say without seeing and feeling this is in action, but my hunch is this is literally a big step forward from the user perspective. People will get whom is doing what without confusion. Contemporary marketing practice would argue UNC make the code open source but patent and license the cute little red finger-tracker-dealies for this use.