The Astronomical Event Search Engine

eldavojohn writes "Google has signed on with the Large Synoptic Survey Telescope project that will construct a powerful telescope in Chile by 2013. Google's part will be to 'develop a search engine that can process, organize, and analyze the voluminous amounts of data coming from the instrument's data streams in real time. The engine will create "movie-like windows" for scientists to view significant space events.' Google's been successful on turning its search technology on several different media and realms. Will they be successful with helping scientists tag and catalog events in our universe?" The telescope will generate 30 TB of data a night, for 10 years, from a 3-gigapixel CCD array.

Comments

3/4 LoC a night

[Oddscurity]

Will they be successful with helping scientists tag and catalog events in our universe? Will they defeat the monster and get the girl? And will they be home in time for tea? Find out next on GoogleTrek.

Seriously though, processing something the equivalent of 3/4th's of the LoC every night is nothing to be sneezed at. Over the course of those 10 years that's about 110 Petabyte (40TB * 365.25 * 10) of unprocessed data.

Re:3/4 LoC a night

[Wavicle]

I actually did a small, insignificant portion of LSST's computation feasability study at LLNL during my internship there a couple summers ago. And yeah, the computational requirements were nothing to sneeze at. I'm not sure where they are at now, the specs changed seemingly every month, but when I left the CCD array was up to 3 gigapixels of 16 bit greyscale. I believe the observing cadence (at that time, again everything was changing on a regular basis) was two of those for the same piece of sky every 30 seconds. Wish I could have stayed... ahh well. I did get a really nice full-color research poster (that I had to design) out of it though!

Re:3/4 LoC a night

[Ingolfke]

No human can look at that much data, and google's experience indexing the web seems only tangentially related.

Google's PHd's and big thinkers could certainly play a part here. Google is about solving problems with large chunks of constantly changing data that has patterns and creating systems to identify and use those patterns. The web is simply a way for Google to apply the model.

Re:3/4 LoC a night

[Joebert]

Pffft, I sift through that much data every night on limewire looking for por... err, movies.

Re:3/4 LoC a night

[Nuroman]

According to Google (how appropriate), 30 terabytes * 365.25 * 10 = 107.006836 petabytes.

Great news

[rolyatknarf]

Now Google will be serving up advertisements on Uranus.

Re:Great news

[matr0x_x]

How long until Microsoft wants to jump into this marketspace as well?

Near Earth Objects

[Oddscurity]

I saw a documentary not long ago about doing just this photographing of the same piece of sky, only with longer intervals than 30 seconds. Anything moving would automagically be flagged by the software, it's vector computed. Correct me if I'm wrong, but from what I can tell of this project, it's going to do exactly that (and more), but on a larger scope, and with better accuracy?

Re:Near Earth Objects

[Wavicle]

Correct me if I'm wrong, but from what I can tell of this project, it's going to do exactly that (and more), but on a larger scope, and with better accuracy?

Well, I was a very small cog for a very large telescope. But my understanding is pretty much exactly what you just said.

Re:30 TB will be nothing in a few years.

[Spikeles]

You are a bit behind the times there.. 1TB consumer drives are here http://www.engadget.com/2007/01/05/hitachi-breaks- 1tb-hard-drive-barrier-with-7k1000/

Re:Why Google?

[edflyerssn007]

Skynet?

Re:Why Google?

[Ingolfke]

It says their smart enough to take on challenging and related problems that they can learn from and use to enhance their information business. This is a real-time application. Imagine if Google could, based on all of the data Google is collecting and indexing, provide a real time view of current trends and patterns of consumers on the web. An immediate zeitgeist presented in a way that a business can use to make sure it's selling its products at the right time to the right people. Cool stuff.

Re:30 TB a night...

[Capt'n Hector]

You can't compress this stuff unless you do it losslessly. Compression artifacts mess up photometry - if you're trying to compute apparent brightness, you need to factor in things like how bright the ambient sky is, and how much point sources get spread out (FWHM, seeing). That is, a point source that passes through the atmosphere looks like a normal probabliity distribution because of atmospheric distortions. So to get an apparent brightness, you have to correct for this effect. If compression artifacts are introduced, FWHM is thrown off, and you have no idea how "crisp" your image really is. That's why these data sets are so large. Quite literally, they're doing a pixel dump from their massive ccd all night. But hey, somehow I doubt they'll be using this telescope for anything but object detection. There's no reason to store it all except to compare a current picture to one in a base set, kinda like KAIT on stearoids.

Lots of data, but not as much as the LHC

[Phat_Tony]

That's a lot of data, but it's less than 1/10 as much data as the Large Hadron Collider will put out, and the LHC is supposed to be coming online within a year, not in six years. By the time the Large Synoptic Survey Telescope comes online, the LHC may have produced more data than the Large Synoptic Survey Telescope will over the life of the project.

I'd be interested to know more about the data handling methods they have in place for the LHC. I don't think they'll be using Excel.

*Note the correct, non-Frudian-Slip spelling of "hadron"

Re:Lots of data, but not as much as the LHC

[dido]

Funny, but CERN itself makes that same misspelling of 'hadron' here. "This is the underground tunnel of the Large Hardon (sic) Collider (LHC)..."

Re:Lots of data, but not as much as the LHC

[mcelrath]

The LHC will produce more data, but we also don't care about most of it. The vast majority of it is junk. The "interesting" physics (particles like W and Z bosons, top quarks, higgs, etc) are about 10^-9 of the events. It is a huge needle in a haystack problem and we throw out most data. We have many experts and professors who design "triggers" which, based on a subset of information that can be delivered to them in a reasonable time, decide whether a given proton-proton collison contains new physics. Many theorists these days are making dents in walls with their heads trying to think of ways these triggers might be missing important information, so that we can suggest changes before it's too late. This is a lot of dedicated silicon, FPGA's, VME crates, etc. Slashdotters should drool. Anyway, we throw out the vast majority of information.

By comparison, LSST is trying to store everything. Scroll up for an interesting comment about calibrating ambient brightness and seeing. I can't answer which will deliver more information, but both are incredibly interesting challenges.

Data challenges abound. We have designed the LHC Grid to distribute this information. There will be several data warehouses located around the world at national labs and universities. Even after the triggers decide what is "interesting", more sophisticated algorithms, with access to all the data in a single proton-proton collision are applied. Then, humans are applied to the data and we will try to dig out new signals from this.

In all this we expect to find (among other things) the origin of mass and Dark Matter, and we're working hard to prepare for the onslaught of data. :)

-- Bob

Search this!

[xebecv]

Hm, Google searching space... I'm waiting for the time google will search in people's bodies and catalog their illnesses.

LSST v. PanSTARRS Approach

[cmholm]

The shop I'm at has been working the image processing and data storage problem for PanSTARRS, another sky survey project that is a bit further along (they have a test scope up and running on Maui). It's interesting to me that both projects are at once using conventional solutions and thinking outside of the box.

Conventional: LSST will use a single large telescope and detector; PanSTARRS (as it stands) intends to use a dedicated compute cluster for data reduction.
Novel: LSST is leaning towards distributing its data reduction task over Google's huge server farm; PanSTARRS will use four off-the-shelf 1.8m telescopes, each with a 1.4GP detector, mounted together to image the same piece of sky, and merging the overlapping images in post processing.

When I was working on the project, one of PanSTARRS requirements was to finish analyzing one night's viewing before the following sunset. Early on, the principal investigators decided to solve the image storage issue by not storing them permanently. Instead, once the science for a night's imaging had been extracted (astrometry, LEO or supernova detection, etc), the original images would hit the bit bucket. Whether they've stuck with that I don't know.

Dull viewing

[caluml]

The telescope will generate 30 TB of data a night, for 10 years, from a 3-gigapixel CCD array.

I bet it makes dull viewing. Sort of like the recent Ashes Tests in Australia. If you're English.

Re:+5 Informative

[VitaminB52]

The telescope will generate 30 TB of data a night

That's a lot of info.

No, that's a lot of data. Info is the result of analysing the data.

Re:30 TB a night...

[hogghogg]

You can't compress this stuff unless you do it losslessly. Compression artifacts mess up photometry

This is not strictly true. What's true is that the current standard lossy compression techniques mess up photometry. However, if you know what you are going to photometer and how you are going to photometer it, it is certainly possible to compress in a lossy way without ruining the photometry. In a trivial sense, photometry is lossy compression of data (you have turned huge images into a few numbers with error bars)!