Pencigraphy: Image Composites from Video

jafuser writes: "Prof. Steve Mann (of cyborg fame) has a detailed technical description on his site that demonstrates a method of transforming video into a high resolution composite image. Pictures are seamlessly mosaiced together to form one larger picture of the scene. Portions of the video that were "zoomed in" will result in a much clearer region in the final picture. I wonder if this could be used in a linear sequence to 'restore' old video to higher resolutions? It's on sourceforge; download and play!" Mann has been experimenting with such composites using personal video cameras for years.

Comments

Not in real-time.

[Christopher Thomas]

The new method, which allows an image to be created by ``painting with video'' is used in conjunction with a wearable wireless webcam, so that image mosaics can be generated simply by looking around, in a sense, ``painting with looks''.

Just in case anyone was wondering - this wasn't being done in anything close to real-time the last time I checked. There's a cluster in Prof. Mann's lab which is dedicated to compositing these images (my cube is in the next room).

Still an interesting project. The affine transformation approach has been well-understood for some time (you do a brute force and ignorance test of promising-looking affine transformations [rotations and scalings] to find one that matches the new image to the old). As far as I can tell, he's doing the same thing with a different coordinate system that has a bit less distortion.

Clearing up some confusion

[Astin]

My undergraduate design project was with Steve Mann on this technology (objective was the "parallelization" of the software on a Beowulf cluster - shout out to Mike and Anna :) ).

The main use of this system so far has been to stitch multiple images into one panoramic shot. Like any auto-stitching program, this requires a certain amount of overlap between frames - the more overlap, the better the stitching. The code works remarkably well, automatically rotating, zooming, skewing and otherwise transforming the images to fit together and then mapping them into a "flat" image as opposed to a parallelogram-shaped one.

Yes, the higher resolution from multiple shots of the same scene works, and is a very cool effect of the system. Of course, this requires a more or less static scene.

Finally, it's not necessarily "video" that it uses, although pulling individual frames from a video would work. It's based of the head-mounted cameras of the wearcam systems, which essentially use a stripped-down webcam for image-gathering, so you already know the fps and resolution limitations involved with those.

Of course, in the 2 years since I've been there, the technology has probably improved, although I doubt the webpage has. :)

Mann has a bunch of cool projects involved with the wearcam/wearcomps. This is a great one, another is the Photoquantigraphic Lightspace Rendering (painting with light), which can also be found on the wearcam site.