Pencigraphy: Image Composites from Video

jafuser writes: "Prof. Steve Mann (of cyborg fame) has a detailed technical description on his site that demonstrates a method of transforming video into a high resolution composite image. Pictures are seamlessly mosaiced together to form one larger picture of the scene. Portions of the video that were "zoomed in" will result in a much clearer region in the final picture. I wonder if this could be used in a linear sequence to 'restore' old video to higher resolutions? It's on sourceforge; download and play!" Mann has been experimenting with such composites using personal video cameras for years.

Comments

Thanks Steve!

[Anonymous Coward]

First he whines about there being spy cameras everywhere (IEEE ISWC 2001 Zurich) and then he does work to make them more effective. What's the deal?

Videobrush software did this in 1995

[AsOldAsFortran]

There are two main applications of this idea

Using it to stich mosaics together.

Using it to use overlapping images to increase the effective resolution of the camera. This is called "super resolution".

Computer vision types have been doing this for years. Shmuel Peleg of Hebrew University has done some good work and had the work show up in commerical products, including Videobrush - you could take a webcam, wave it around, and in real time get a mosaic. In 1995 (I think.) Don't know if you can still get that product.

Do a google search for him and you'll find his home page and superresolution papers (Peleg and Irani is an accessible paper and one of the first - the concept they used, however, comes from Bruce Lucas' thesis at CMU.

Applications include: combining NASA satellite images of the Earth to get higher resolution, ditto for images of the human retina; and, a personal favorite, smoothing images of the system used at the Superbowl a couple of years ago where they had 75 cameras and could show any play from any angle in liive video. That was done by Takeo Kanade, Luca's' advisor.

Re:Restoring old video

[Anonymous Canard]

It talks about combining multiple frames from one camera to create higher resolution where they overlap. Perhaps you need to read the article.

If the camera is pointed only in one direction then you don't need an isomorphic mapping and knitting algorithm to clean up the signal, you can do it by directly overmapping sucessive frames. Again, this technique has nothing to do with cleaning up old video. I suppose if you had a movie that contained a slow pan of an area then you could map successive frames to get a wide angle still image of the same area, but that still wouldn't have anything to do with cleaning up video since the final output would be a still image.

To get improved video you would have to have continuously running frames from multiple angles, all shot from the same vantage point (or the same location if you prefer), just as I originally wrote.

Re:Not in real-time.

[Steve Franklin]

"The new method, which allows an image to be created by 'painting with video' is used in conjunction with a wearable wireless webcam, so that image mosaics can be generated simply by looking around, in a sense, 'painting with looks'"

Reminds me of a technique that was used to photograph large airplanes in a hanger with a limited number of light sources. The lights were turned out, the shutter of the camera was opened, and the technicians would "paint with light," illuminating various parts of the airplane in sequence. This from someone who worked in the photography department of a large aircraft manufacturer.

Er, but does it *work*?

[pla]

Can we say "documentation", people?

I have three pictures, with roughly 2/3rds overlap.

I ran them pairwise (1 and 2, then 2 and 3) through estpchirp2m. Good, I get two output sets of 8 reals. I stuff them into a single file, one on each line.

So I pintegrate that file, using picture #2 as the reference frame. Cool, I now have three sets of eight reals.

Next, I pchirp2nocrop all three separately, passing the appropriate line from pintegrate on the command line (why bother with text files here, if I need to cut-and-past at this step anyway?). I now have three new .pbm files, which seems like what I should have according to the extremely limited documentation.

Step four, I cement the three new .pbm's together, and get a single file as the output. "Great!", I think, it worked and didn't give me too many problems.

So I open up the picture. Or try to. It seems that whatever the output file has in it, valid .pbm data doesn't top that list.

I tried again, but since I had followed the (limited) directions carefully the first time, my results did not differ.


So, I have three suggestions for Mr. Cyborg...

First, it doesn't matter *how* cool of a program you write, if no one can figure out how to use it (WRITE SOME REAL DOCS!!!).

Second, it doesn't matter how cool your program *sounds*, if it doesn't work.

Third, 99% of people playing with this will either not want to tweak any of the in-between stages' results. Of those that *do*, 99% will just hack the source. Ditch the four (and then some) programs, and make a single executable that takes as its arguments just the name of the input files, in order, and perhaps a *few* tweaking options (like enable or disable filtering, which sounds useful, except YOU DON'T HAVE IT DESCRIBED ANYWHERE!).


Ahem.

Otherwise, great program. No doubt one of the many companies doing the same thing for the past 20 years will soon have their lawyers send their congrats.