Slashdot Mirror
Colorizing Images and Video by Scribbling
Guspaz writes "Up until now, colorizing a video or image has been a painstaking and mostly manual task. However, researchers in Israel have come up with a new way of colorizing images just by making a few scribbles. The technique works on the premise that 'neighboring pixels in space-time that have similar intensities should have similar colors,' and also allows colorization of videos by 'marking' about one in ten frames."
Back when voyager(s) were flying by planets I recall reading how the cameras worked. From what I remember, the cameras actually capture images in black and white. The cameras can detect much more "color" depth than color cameras could (or can?). The scientist would process the pictures to colorize them, you identify one area of color you know and the algorithm would process the rest of the 1 billion shades of gray into a color mapping for people to view. Now why cant identify this gray shade as the color red; anytime you see it then that is red. Go on for each color spectrum or have the algorithm adjust what a little red hue is for a given little hue of gray. It appears that is what the scribbles are doing which is quite clever and the algorithm doesn't have to work (guess) so much.
from looking at the before and after images, this technique looks pretty cool and will probably have applications for recoloring an image that is already color. For instance, the image where he recolors the fabric on the chair.
Isn't that just a fill tool? Paint does that.
No. Fill just goes until it meets a boundary. This colorization is a lot smarter than that. It appears to notice the boundarys by the sudden changes of the temperature in the color of pixels. That way it can then make an educated guess on how much to color and when to stop. You can then optimize this by putting in more than one input of the colors you want to change. This effect is really quite amazing. Scroll down and look at the gif video of the birthday party. JUST AMAZING.
I'm a virgo and on Slashdot. Coincidence? Yes.
I am so curious what this could do for so many old movies...
:) A lot of the appeal of older B&W movies is the fact that they aren't in color. You get a much broader range of contrast when it's filmed on B&W film than a color image which has been desaturated.
Ruin them?
If you meant older color movies which have degraded, then I agree. This seems like a very useful technique for restoring the original vibrancy of colors to films whose media hasn't stood the test of time.
Want to improve your Karma? Instead of "Post Anonymously", try the "Post Humously" option.
Read about it earlier this week, it looks cool, colorization looks nice, but with so few samples its hard to tell if it will work with a wide range of inputs, or only ones with contrast ratios like those shown.
Video Production Support
This work is very similar to some work that was presented at last years siggraph using graph cut optimization titled Interactive Digital Photomontage by some researchers at the University of Washington. This stuff is really cool and has applications outside of just re-coloring black and white. For example, compositors in the film industry adjust the color composition of scenes that were filmed during the day to look like they were filmed at night. Sometimes they just need to tweak the color because the art director isnt happy with it. Other times it's because they introduced CG elements into live action scenes and they dont quite match. If they can tweak those colors interactively, without authoring masks, it is faster than re-rendering the scene and that saves money.
Very cool stuff.
Pete
What's a sig? Pete Brubaker
I'm not sure if this was intended to sound like some new ground breaking technique, but it really isn't. I am a masters Electrical Engineering student and am currently taking an Image Processing class. Using neighboring images to reconstruct an image is a VERY VERY common task - in fact, it's almost the only way to do it. How else are you supposed to guess the colors (or what pixel is *supposed* to be there) without knowing what's around it. It's obvious that the highest correlation will be between the nearest neighbors (except on some edges).
Maybe next time we can make a program that just guesses the colors and look at how interesting those come out!
No kidding. Even if you are not doing colorization, the boundry detection algorithm he is using kicks ass over the "magic wand" tools in both photoshop and gimp. Perhaps it is the fact that it is doing several "magic wands" at once and boundries are determined by what matches the best, rather than just "does this match good enough".
The site is slashdotted so I cannot read it, but i wonder if something akin to this could be used for compressing motion video. For example the intensity is encoded with currrent techniques, but instead of the color being encoded at a lower resolution, instead only a very small amount of colored points are encoded. Then during the decoding, the decoder uses an error function, intensity, and the time domain of previous and future frames to 'fill' the colors out.
You mean like dubbing Mel Gibson in Mad Max?
It happens already.
Im not sure about that. I'm taking a wild guess, but I believe they simply start propogating out from all scribbles, so that, as the growth of two colors approach each other, the stronger one wins out. The wand tool in photoshop starts only from the point you click (rather than the 10+ scribbles in this algorithm). There's no competing areas of propagation.
Create a magic wand tool that requires multiple clicks on the various regions of the image and you'd have pretty good results.
-dave
http://millionnumbers.com/ - own the number of your dreams
This is a really cool thing to see in person. At NAB last year the Discreet people had this as a feature coming from there high end Smoke systems. Since all Discreet programs require a Tablet the pen scribble thing is used for color correction by scribbling around the area you want changed. Masks are done by making a rough trace around the area and it figures out what your focusing on. Its a time saver but you pay hundreds of thousands of dollars for their systems. After a couple of years other programs like Shake and After Effects start to pick up the plugins. Most major tech leaps in video happen at Discreet first then trickle down.
I tried out their matlab code and put a few example colourings on my web page, for the interested:
http://www.inference.phy.cam.ac.uk/cjb/
" The technique works on the premise that 'neighboring pixels in space-time that have similar intensities should have similar colors"
Interestingly, the retina exploits that same property of natural scenes to compress images. This correlation between luminance and color is an opportunity to throw out redundant information. The eye multiplexes color and luminance information over a single channel, transmitting luminance while discarding color at high spatial frequencies and transmitting color while discarding luminance at low spacial frequencies. First reported by C.R. Ingling, color/luminance multiplexing is an inherent property of the linear color-opponent center-surround receptive field. For a good explication of the subject, see:
Vision Res. 1985;25(1):33-8."The spatiotemporal properties of the r-g X-cell channel."
Ingling CR Jr, Martinez-Uriegas E.
Abstract: Analysis of the simple-opponent r-g receptive field of the X-channel shows that it is tuned to both high and low temporal frequencies, high and low spatial frequencies, and that its spectral sensitivity is both chromatic and achromatic.
Ceci n'est pas une signature.
Some folks seem to be excited (or angry) with the possibility of coloring B&W movies with this technique. Forget realistic coloring, this looks amazing for artistic recoloring.
Go take a look at the "recoloring examples" in the coral cache. Also look at what a slashdotter did with the code. Photographers, designers and painters could do neat things with a filter like this in Gimp...
Prescriptive grammar:linguistics
Apparently, one of the critical differences between Octave and Matlab is that Octave apparently doesn't support multidimensional matrices/arrays with dimensions greater than 2.
:(
Which means that the current code is completely incompatible with Octave, as it depends on Matlab's implementation of imread() which returns image data as a three-dimensional matrix.
Going to see if I can get it to work easily, but there's a good chance I won't be able to.
retrorocket.o not found, launch anyway?