Digitizing 100 Years of Astronomical Data

Maximum Prophet writes to mention that a collection of glass plates containing astronomical information from the late 19th century through the mid-1980s is being considered for digitization. "The accumulated result weighs heavily on its keepers on Observatory Hill, just up Garden Street from Harvard Square: more than half a million images constituting humanity's only record of a century's worth of sky. 'Besides being 25 percent of the world's total of astronomical photographic plates, this is the only collection that covers both hemispheres,' said Alison Doane, curator of a glass database occupying three floors, two of them subterranean, connected by corkscrew stairs. It weighs 165 tons and contains more than a petabyte of data. The scary thing is that there is no backup." I'm sure that anyone with a spare $5 million or so would be welcomed with open arms.

Comments

Glass plates will outlive the digital"backup"

[gatkinso]

now there is some irony.

Re:Glass plates will outlive the digital"backup"

[KokorHekkus]

now there is some irony.

But currently they also makes them vulnerable to a single point of failure (as indirectly pointed out in the article). If you have some data that has any real value for you then having only one copy (or only one storage facility) isn't any real protection whatever method you use. In this case we have data that would be readily accepted for backup by organisations all around the globe and barring a worldwide upheaval the safety of the data would be much better than any single glassplate could offer.

Of course the ideal would be if we could develop a cheap digital permanent storage that had guaranteed physical longevity, say several millenia. That combination would allow easy dissemination of the data and safety by using a multiplicty of sources.

Re:Glass plates will outlive the digital"backup"

[Cecil]

Ever tried to maintain archival backups for a petabyte-worth of data?

Yes, as a matter of fact. Definitely a lot of work is involved, but do you believe that you wouldn't need a team of document managers, millions of dollars worth of floor space, and expensive climate controlled facilities for archival of microfiche? You most certainly do. It's a lot of data. Period. No matter what you try to do with it, it's a lot of data. It's going to require a lot of resources. That's just a fact of life.

Anyway, noone in their right mind would choose microfiche for that type of data. If you're only storing plain text pages it's adequate (though I still don't think it would be the "right way to do it" in this day and age), but for photographic plates? Not going to work.

Microfiche is vastly overrated, in my opinion. My current project involves taking 2 floors worth of 30-50 year old microfiche and scanning it, OCRing it, and PDFing it. Yes it certainly does age. Quite poorly, in fact. The quality is absolutely terrible compared to the paper versions, some of it is stuck together, and indexing and cataloging it is a nightmare all of its own.

Yes, there are challenges in the digital world too, but most are easily surmountable given a little bit of common sense in understanding that digital is not magic. It doesn't mean you can "fire and forget". The documents will still require maintenance, cataloging, protection and monitoring. Format obsolescence is very nearly a nonissue, it is blown way out of proportion. That's where the "maintenance" comes in. The key benefit of digital is that you can and should losslessly upgrade your format whenever obsolescence is becoming a concern. Formats do not disappear overnight and suddenly everyone forgets what to do with them, you have plenty of time to make your transition if you're paying attention (which you must be: again, digital is not magic).

Re:Glass plates will outlive the digital"backup"

[FST777]

By that time, other techniques will be available to copy the digital archive over. Heck, it might even be possible to make a copy of the digital data on glass plates, complete with descriptions of the used protocol.

It's true that digitized data is more prone to failure than most analog carriers. The whole point is that digitized data is much easier copied over and over again, without loss, independent from whatever carrier used.

This sounds like a job for Google

[MDMurphy]

Google provides views of the Earth, Moon and Mars, why not stars? If the information was made available for them to deliver to their users, they might be interested.

A Million People With $5

[Stranger4U]

This seems like a great opportunity for either corporate sponsorship, or a grass-roots donation drive. In all honesty, $5 million isn't a whole lot of money for the likes of any real corporation, and it probably wouldn't be that hard to raise it through small donations from individuals. Espectially if you could ascribe names to some or all of it. How would it feel to be able to personally identify which plates you paid to have scanned? (this image of the Crab Nebula brought to you by John Smith) I'm surprised Paul Allen or Richard Branson aren't all over this like stink on shit.

Google

[blhack]

I'm sure that a company like google would be MORE than willing to fund a project archiving these. The positive press, proliferation of their intended "do no evil/good guy/just another bunch of geeks" image, having their name on a major scientific project would easily be worth the investment.

Re:Why does it have to cost so much?

[JohnnyGTO]

Those plates as well as being old and delicate contain a LOT more information then a piece of paper. Considering that something less then 1/4 the size of the period on the end of the sentence is important your scanning at a much higher resolution.

Re:Would Google archive it, perhaps?

[ScrewMaster]

Google might do it just because it would be un-evil, and worth quite a few brownie points with scientists around the globe, not to mention that it would be cool archive to search.

Re:InfiniBytes

[modecx]

here is a practically infinite amount of data on each of those plates, limited by our precision in measuring them.

And limited by the lenses/mirrors, and limited by atmospheric effects, and inconsistencies in the glass, and the silver, and, and....

I can't testify to the quality of the glass negatives, but I can testify to the fact that as much as people like to believe, even the best modern analog capture sources aren't anywhere near practically infinite, even in the best laboratory conditions.

Re:InfiniBytes

[Doc Ruby]

Well, the lenses/mirrors that are now lost to history do introduce noise. But the atmospheric effects, and inconsistencies in the glass and silver, and probably much of the "writing" noise from the optics do all hold the possibility of being filtered out. Maybe not now, with today's early signal processing tech. But in another hundred or more years, that signal info could be available. If we don't damage them in the interim.

All this stuff should be digitized and made public

[syousef]

When I completed my Astronomy masters access to publicly available data from various sources (most notably NASA data made free to the public) was a real boon. It meant we could do analysis on actual real data instead of artificial or sanitized textbook material. A couple of the students built on this to do some original research. (Sadly that's not the way I went, as my time was more limited).

There are also lots of amateurs out there running a wide variety of very specialized packages to do everything from discovering asteroids to keeping tabs on the brightness of stars and watching for supernovae.

Luckily glass isn't a liquid....

[Joce640k]

Luckily glass isn't a liquid so they won't distort.

Re:Why does it have to cost so much?

[geekyMD]

Holy crap dude, you just won the asshat of the year prize. Do you have any idea of the magnitude, delicacy, or importance of the data you're talking about? To say nothing of the needed precision when scanning.

"I scanzord 90 filing cabinets of paper into teh computerz"

You know what, I used to launch model rockets. Its really easy to make stuff go up. Just buy the kit, attach a little engine and off it goes. $30 easy! Freakin NASA I bet they're spending all of our tax dollars on pr0n.

"cheapish 20megapixel camera" - Ever hear of the Hubble? I hear people like it for more than those weird nebulae pictures. I guess we should have just given one of those astronuts a Nikkon and let him go to town. Much cheaper.

And I guess we should use lossy compression, its just empty space out there right? I bet we could get the infinite sky down to a couple hundred GB. (JPEG, its for astronomy too!)

Re:InfiniBytes

[monopole]

Having worked with holographic media for decades (which is about as fine resolution as you can get optically) the maximum resolution is on par with the grain size 40 nm (Afga 8e75) and considerably worse both due to the wavelength of light and the expansion of grains during exposure. To get 'molecular' resolution you'd have to go over to dichromate plates far too slow.

Due to speed considerations the grain of these plates would be much worse. But well within the resolution of the 'scope used for recording.

All that said these plates are a goldmine once digitized due to the ability to do massive searches both spatially and temporally.

GoogleSky

[12357bd]

Seriously, let Google index not only that collection, but any stellar image information and launch GoogleSky.