Slashdot Mirror
Reverse Engineering Doctor Who Into Color
Lanxon writes "In 1967, the BBC set about junking its Doctor Who archive: a moment sci-fi fans wish they could travel back in time to prevent. There are 108 vintage episodes missing, but since 1978 a number have been rediscovered as 16mm black-and-white films. The BBC shot many of these series in color, but made monochrome copies for countries such as Australia, where many TV companies were still broadcasting in greyscale. The reels had sat in archives since. Now, the Doctor Who Restoration Team, an independent group contracted by the BBC, is using a new technique to regenerate The Doctor in color."
I traveled back in time and re-posted it. . In Colour!
. .
Since this is the BBC, they shot *none* of them in color but many of them in *colour*....
nt
So the article was devoid of anything of particular interest other than some jargon. The jargon, on the other hand, led to fascinating little technique about reconstructing the color of the grayscale image from "chroma dots". The actual method was discovered by a BBC engineer, and you can read more about it here: colour-recovery.wikispaces.com.
Can they reverse engineer the scripts instead? Color or black and white, those old episodes are damn unwatchable. We'd be better off giving Wikipedia descriptions of the episodes to the writing staff of Golden Girls. Those old droning 5-part episodes would be turned into 22.5 minutes of tightly scripted comedy starring Bea Arthur as the Doctor. And any of the other old hags as K-9.
Fascism trolls keeping me up every night. When I starts a preachin', he HITS ME WITH HIS REICH!
The good news is that they've figured out how to restore colour to the B&W negatives. The bad news is that it requires Kodachrome processing...
When you have nothing left to burn you must set yourself on fire
this isn't a new technique - TFA even says it's a refinement of a technique that's been used before.
damn cool though, to get the crap from the colour subcarrier that spilled into the luma image and re-generate the original from it.
good thing those old kinescopes were in focus!
"Computer Science: it works, bitches!"
For a taste of recolored Who, see Babelcolour's videos (hand-recolored, frame by frame)
It almost is reverse engineering. The chroma subcarrier in a video signal has a center frequency picked to allow the sidebands to fall between those of the main lumanance (black and white) video. The spectrum of those extends out from the main visual carrier frequency (or up from D.C. for the baseband signal) at multiple of the horizontal scan rate. The goal was to add color broadcast information to an existing greyscale system while introducing a minimal amount of interference. Here people are figuring out what is going on from the visual interference.
The added signal amplitude represents the amount of color added/subtracted from the greyscale white, and the phase represents the hue. The phase of the signal is compared with a short burst (a minimum of eight cycles) sent just after the horizontal sync pulse prior to the start of video on each scan line. PAL, as used by the BBC, is very similar to NTSC, except the scan rates differ, the phase of the reference signal is inverted on every other line to help cancel out the effect of small phase errors on tint.
Basically, those trying to recover color from the back and white films of on-air video have to use a comb filter to pick off the frequency (precisely related to the inverse of the spacing) of the resulting dots that are there from the color signal. The position of the dots from left to right carries the phase information. Considering that the dot pattern is probably quite weak, the resulting color would be noisy. Depending on the filtering used, the bandwidth (detail) may also suffer. But it is still a good starting point to know what the colors were.
The dots aren't on/off like pixels. It's actually a sinusoidal intensity variation. I recall some older Zenith B&W sets had particularly good detail (and maybe some video peaking - enhancement) making it easy to see which programs were broadcast in color, and what parts of the picture were deeply saturated. In addition to a notch in the video response at 4.5 MHz to filter out patterns from the sound, some sets rolled-off or notched centered at 3.58 MHz (3.579545 actually) video response to reduce the interference. Better later sets (and color generally) used "comb" filters to separate the interleaved spectral components without those loss of detail seen with more primitive methods. Failure to filter color signals could cause wild colors/patterns on things like striped neck-ties when a shot zoomed in/out.
It's pleasing to see that there are still a few around that understand the old analog technology well enough to realize there were visual color cues remaining. Even those that understand the electronics well often don't associate a particular visual characteristic with the responsible signal attributes.
Although partial signal recovery is easy to envision with analog electronics, something along the lines of a GIMP/Photoshop plugin could work as well. Some might think of it as being similar to watermark detection.
Like everything from Colin Baker. Seriously, aside from Peri's chest, there was nothing of interest in those episodes.
Does this sig remind you of Agatha Christie?
At least, if they restrict this to those serials that were originally shot in color. I would be a bit uncomfortable if the older, black and white originally, serials were colorized.
Since in this case, film is meant to be run through a projector of some sort, the "white" areas are actually transparent, and the "gray" areas are, well, transparent areas that are partially obscured by black pigments.
And AC simply because I don't care about karma whoring.
I think the clue is in the name, you are speaking English not American.
...oh, and yo momma's so fat, her Schwarzchild radius is visible to the naked eye.
On the restoration processes used in the past can be found on the RT's website, if you dig around a bit: http://restoration-team.co.uk/
This post expresses my opinion, not that of my employer. And yes, IAAL.
The politics behind the Chroma Dot story is intriguing and in some places unpleasant. The instigator of the team was James Insell, and a method was created to perform the chroma dot extraction by a man named Richard Russell. Insell became a bit proprietorial over it all, and he and Russell parted ways, and now Russell it doing it alone. The original Colour extraction blog is here but they don't seem to have made any huge advances since Russell left. There is some more info, plus a link to Russell's own work (including software download) on my own Dr.Who webpage here
My web domain.
My mom told me once she was watching a black and white TV with her family, and someone walked on the screen with green hair. Everyone watching the TV instantly started laughing because the guy had green hair. I don't entirely understand your post, but it does verify that my mom was not crazy, and average people watching in those days could distinguish even if they didn't know what was going on.
"First they came for the slanderers and i said nothing."
A method of digitally replacing, Jon Pertwee, Peter Davidson, Colin Baker, Sylvester McCoy and Paul McGann with... I don't know ... Ewan McGregor or anyone!
Posts, MyBio or Sig, may contain satire, sarcasm, bolded nouns be sardonic or even witty & be Church of SD
Good show, old chap, well done indeed. I, for one, miss the smell of hot dusty valves, and the high pitched whine of an ageing flyback transformer.
All they really need to do is to give the prints to Turner.
I am sure this is a dumb question but why doesn't it just work? If the colour subcarrier is there then why doesn't it just show in colour when displayed on a colour TV? I thought this was why some patterned ties, shirts, etc. with fine monochrome lines would show as a glittering rainbow of colours.
"the guy had green hair. I don't entirely understand your post, but it does verify that my mom was not crazy,..."
No, she was colorblind, it was a redhead.
At least she was polite enough not to comment on his transparent trousers. You were really naive to think she was laughing at his green hair.
My mom told me once she was watching a black and white TV with her family, and someone walked on the screen with green hair. Everyone watching the TV instantly started laughing because the guy had green hair. I don't entirely understand your post, but it does verify that my mom was not crazy, and average people watching in those days could distinguish even if they didn't know what was going on.
http://en.wikipedia.org/wiki/The_Boy_with_Green_Hair
If Slashdot were chemistry it would look like this:Cadaverine
" ... Why doesn't it just work ... "
Because colour TVs don't interpret chroma dots to display colour, they use three different signals for RGB. Engineers would have had to build chroma-dot interpretation into the colour TVs.
Things don't just work. They have to be made to work. Are you in middle management by any chance?. Do you use the phrase "Make it so Number One"?
I know enough to know this is wrong, PAL uses a colour difference signal. Fine patterns in the luminance signal do show up as colours, check patterns would often show as strobeing colour on older colour TVs.
This along with the MGM fire that destroyed the original Tom and Jerry prints, is more proof of how piracy can help us. If this stuff had been pirated all over the net like it would be today, it wouldn't have been lost in the first place. Hopefully they would have used a loss less format though... :-)
You mention the fact that pal alternates the phase on every line. IMO this is probablly among the hardest problems for those doing the recovery. This means each line has (roughly) the same color information but encoded in opposite ways.
When you are working directly off a video signal this is no problem, counting lines is easy. But when that video signal has been translated into a 2D analog image it means you are going to have to seperate the lines (which are probably bleeding together). I suspect this is a bit that still needs some manual intervention to tell the computer which interpretation is correct.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
Why not re-record them with the new cast? I'd love to see the old stories, but where there's nothing left but audio & a few stills...
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
Ah, memories of watching Tom Baker followed by an episode of 'The Tripods' then to be lulled to sleep by 'The Star Hustler'.... On Fridays and Saturdays I could stay up WAY past 9:00...
I seem to remember that episodes starring his immediate predecessor weren't bad either.
Before that it was black and white. I think I only ever saw Dr #1 discover the daleks, and Dr #2 probably wasn't bad, but he did look like one of the Three Stooges...
I also remember not liking to watch Doctor Who anymore starting with the Fifth Doctor there may have been more doctors? I stopped caring.
Then as an adult, starting with Christopher E. , I've been a fan again. I've really enjoyed every one of the new Doctors since then.
It's been long enough since watching the old Tom Baker episodes that I don't remember the plots anymore, but having gone back and watched a few on Netflix, I don't see myself watching any more of them.
The special effects are very dated of course. I mean, green slime covered green lightbulbs wrapped in bubble wrap skin are obviously just what I described. But that's not the real reason. I just don't think the old episodes have much to offer someone who can/has viewed the new ones. The best of the old is part of the new character and plotwise, with HOUR long episodes to flesh things out more deeply.
...
If "color" is incorrect, tell the Spaniards. The French have already been telling you "colour" is incorrect; it's "couleur".
The colour distortions NTSC and PAL suffer from on high frequency luminance changes are interesting and a fascinating insight into the world before digital. Every geek should IMO understand this because it's interesting and cool and easy to understand. I'll try and simplify the explanation...
The artefact in question results from a colour (analogue) TV trying to display a picture where the luminance (brightness to you and I) changes rapidly, i.e. with a high frequency. Viewers of old will be used to people with stripy shirts suddenly and stragely developing psychedelic colour patterns. Why do you get them? That's the interesting part:
When NTSC was devised they had to design a system that was backwards compatible i.e. used the same bandwidth to transmit a colour signal. Worse than that they had to have the signal such that the colour signal would be interpreted by a B&W receiver as a B&W image correctly.
So what they did was look at the signal and realise that although the bandwidth allocated allowed for rapid luminance changes, no set at the time (in domestic usage anyway) was capable of displaying them. Therefore they could transmit a high frequency signal on top of the luminance signal that existing sets would filter out, they would act like a low pass filter, and average it out back to the original signal. They could therefore superimpose a high frequency signal on top of the old luminance only signal. Now, how to carry information with this signal?
Well you kept the frequency of this superimposition constant but you could vary both its amplitude and its phase* relative to a reference signal. This combined with some maths behind luminance and chrominance processing (wikipedia is your friend here) meant that you had good old luminance information for old sets, and two colour components that could be extracted by a compliant set, but would be ignored by older sets.
Now where the artefact in question comes from should now be obvious, a high frequency luminance change looks indistinguishable from no intensity change with a certain colour information. The system was designed to minimise these, but they had to happen at least slightly. As TV sets and studio cameras have improved over the years there have become more of these artefacts as we have got better at displaying fine detail, so what wasn't a problem now shows the limits of the system.
If you want information about how the colour reference signal was sent or to explain more please do ask, PAL and NTSC are one of my favourite geeky subjects. It's engineering from an almost forgotten era and a sign of just how clever some engineers are/were.
*The difference between phase modulation and frequency modulation are somewhat academic for the purposes of this discussion, let's just say that phase modulation makes more sense because we have a reference waveform to compare it to.
"The weirdest thing about a mind, is that every answer that you find, is the basis of a brand new cliche" -
English is almost entirely made up of foreign meddling - it's a peasants combination of many languages that adapts and changes with time. Very few words (except modern technical terms) have no root in a foreign language.
Now there's one hoopy frood who really knows where his towel is!
Tradition says that technical terms be based in ancient greek or latin. Thus television, for example. Just about no-one spoke latin when the word was coined, but it's just tradition.
Amazingly enough, the original implementation of 'Colour Recovery' was done in BBC Basic for Windows - an extension of the language used on the old Acorn computers in the 1980s.
That link is a much more technical overview of the process - the first time they used Colour Recovery, they used it as a supplement to more traditional computer colorization. They had an outside firm (Legend) do a hand-colorization of a black and white episode using an improved version of the old Ted Turner colorization process. While it works, it tends to turn out a somewhat washed-out and unsubtle colorizing - it's really hard to get the gradations of skin tones right, so they desaturate the color a little bit so it just looks washed out rather than as if everyone is wearing pancake makeup.
The Colour Recovery process is error-prone (you're using artifacts of the original color signal recorded on a different medium - there are drop-outs and inconsitencies due to the original transfer process) but where it works, it really captures the original color range and gradations much better. And it also takes out some of the guesswork - the Legend team based a lot of their work on still photos, but where none existed, they used their best judgment. In one case, they made flower in a jungle scene purple, where Colour Recovery picked up the fact that the cheap (er, cost-constrained) BBC effects department had just made everything different shades of green.
In the end, they blended the two techniques - using CR to correct the Legend colorization. The link has Legend/CR/blend shots - the blend really looks good, while each of the originals have (different) problems.
This is at least the 2nd time the BBC restored a B&W Doctor Who episode to color [sic]. The first was by combining an early color videotape recording (by a fan in Texas for the color) and the B&W film for the image itself. They just superimposed the fuzzy chroma on the film image. The result was surprisingly sharp and colorful (but then I watch NTSC standard definition so I ain't picky). They had to adjust the picture shape just a tad because the VHS image wasn't an exact match for the film. *sic: Since the color tape was from a U.S. fan, I spelled it the U.S. way.
No, those are the original spellings of the words as first adopted into English. The spellings you advocate were created solely in the US, many by Webster.
Well, I've heard of people being able to determine the content of a wav file by just looking at the raw data, so...
What most people don't realize is that K-9 was actually a zebra!
In the era of the 1980 home computers, It was an advanced programming technique, on the same level as VGA programming in the 1990's, or GPU shader programming is now. It was more of technological curiosity, with only one or two games used it (Tetris), since the results varied from TV to TV. It did seem strange how using a 320x200 monochrome screen could generate red, green, yellow or blue pixels just because the pixel size was smaller than the sampling rate of the CRT tube.
Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
It was supposed to be appalling.
Best Slashdot Co
You have an analogue signal that is split up into three difference luminance levels (red,green,blue). These would each be aligned so that they matched the red/green/dot pattern of the CRT screen. As the image is recorded onto black-white film which would have a fine crystal grain size, the individual intensities of each red, green or blue dot would have been recorded. Although the film was in monochrome, it would still have recorded the color information, but as a stipple or dither pattern. If you can convert this image into a high-resolution digital image and write a suitable algorithm, you could reconstruct the original color information.
It would be no difficult from taking a close-up shot of a LCD screen, converting it to monochrome, and then writing a photoshop plugin to convert the image back into color.
Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
My mom told me once she was watching a black and white TV with her family, and someone walked on the screen with green hair.
What I wrote about is only a subtle dot interference pattern that has no directly visible color.
But I think I know what your mom saw. There was some experimentation in the mid 60's with trying to get the perception of color on a black and white television. I haven't found a description in a search yet, I forget what it was called. There was a popular tv show called Combat! that was in black and white (color came later in the final season). There was publicity beforehand about an ad that was going to run during the program with color visible on black and white sets. It was for a soda... Squirt or Bubble Up or something similar. As I recall the effect was used just on the logo on the bottle. It depended on the brain perceiving some illusion of color when things flashed at a certain rate. Some people saw the effect more than others. It was used in some other ads beyond the one that got the publicity, but never caught on. The color effect had visible flicker to it and certainly couldn't pass as a normal full-color picture. I seem to recall seeing violet, but there may have been green too. I think besides not working consistently, there was also fear of triggering seizures in some people. I think it would have been fun to use in a 60's sci-fi show. Put out a gag news items saying that there had been suspected reception of tv from a parallel universe, causing color to been seen on black and white sets... then use the effect later.
I've more recently seen other ads, and I think music videos, that were released in black in white, but with color in isolated spots, like maybe just one persons face, for shock value. That color still takes a color set to see though.
You may be pink, but black and white are my favourite colours.
For those that don't bother to RTFA I'll distill it down for you: From the summary we know that theyhave some long lost missing episodes that are in B&W and are converting them to color. The article says "...they didn't filter off the colour carrier [encoded as a 'chroma dot' pattern in each frame]" And continues with "...used the signal to reverse-engineer raw colour pictures that could be retouched frame by frame. 'It's very, very labour intensive -...' " Uhmmm... So basically TFA says "we reversed engineered it, and it is hard." I guess its for them to know and for us to find out. And here I thought TFA would maybe give me some INSIGHT on how they did it.
I have roughly the same memories of watching public TV in South Fla and starting with Who, and endng with The Star Hustler.... I am very tempted to get some old eppies of it to show my son after he's done watching Who/Sarah Jane, just to see if the stars attract him as much as they did me...
Hmmm, I guess I described it wrong. What happened was (I think it was a comedy show), someone walked in with green hair, and everyone laughed and said, "He has green hair!" It was part of the routine. Then they stopped and suddenly started wondering how they could possibly know that, since they were watching black and white.
In other words, it would seem that people are subconsciously picking up on the color after years of watching B&W TV. Or, were.
"First they came for the slanderers and i said nothing."