Sloan Digital Sky Survey

Swannie writes: "There's a story in today's Chicago Tribune about a joint project that Fermi Lab is taking on with Rensselaer Polytechnic Institute in New York. The goal is to produce a 3D map of the universe using a really big digital camera, and a really creative way to add "depth" to the image. The article has some decent technical details for a newspaper, including a pretty picture." Update: 03/12 15:44 GMT by M : The blurb is in error. A particular scientist from Rensselaer is mentioned in the article, but Rensselaer isn't part of the project as an institution.

Comments

What is a true map?

[phunhippy]

I think it would be really cool if when they take all this data they are collecting, they produced a 3-d image of the COSMOS and a 3-d image of the cosmos with every star's location shifted to show its theoretical place today... or in the case of billion light year stars.... nothing if they are burned out by now.... that woould truly be an intersting map to look at :)

Re:What is a true map?

[Treeluvinhippy]

Hmm that would be some heavy-duty calculations to make a real time 3-d map of the universe. First off, allot of stars would have changed classes, you know.... blown up into red giants or shrunk into white dwarfs and such. Also interactions with other star's gravity wells would cause some of them to change course or even collide. This would be difficult, since in a lot of cases we only know the state of the star a few billion years ago. Nebulas dissipate, novas great new ones. Everybody knows about supernovas. Hell it wasn't until a few years back astronomers (or astrophysists) discovered that the Milky Way is absorbing another galaxy in the constellation Sagittarius. (I read this in a Scientific American article a while back. Please correct me if I'm wrong.)

It would be tough, but really cool if it could be done. It might even be considered a Wonder of the World.

So the real question remains. How big should we make the Beowulf Cluster of Imacs?

Re:What is a true map?

[stereo_Barryo]

The phrase "3-d" is thrown around, but I don't see anything about HOW the distances are determined. I understand doppler shifts for distance, standard brightness of a type, etc. but we are talking about having computer analysis of the billions of points of light and calculating a distance for each to produce a 3-d map? HOW? Barry

Re:What is a true map?

[john_ee]

Yep, that's what we're talking about. Their data pipeline is described here.

Re:What is a true map?

[tconnors]

I think it would be really cool if when they take all this data they are collecting, they produced a 3-d image of the COSMOS and a 3-d image of the cosmos with every star's location shifted to show its theoretical place today... or in the case of billion light year stars.... nothing if they are burned out by now.... that woould truly be an intersting map to look at :)

Except that galaxy evolution (I have just started a PhD on the topic) relies on all sorts of complicated things that this survey wouldn't give us. Star formation and supernova feedback, stellar evolution in general, gas, the cosmological paramaters that we haven't got a physical handle on yet, galaxy mergers - and most importantly - dark matter haloes. The whole evolution of galaxies is dictated by the rather invisible mass surrounding them, that we are only starting to be able to model.

But having said that, there was a paper on astro-ph the other day claiming that one very high resolution simulation that focussed on one particular galaxy seems to show the thing changing between all the differnt kinds of galaxies - including a spiral growing a bar! Now, there needs to be all sorts of other work to see if this is feasable - such as whether the chemical signatures make any sense, but it would still make one hell of a movie seeing that thing evolve!

Re:What is a true map?

[FastEddieCoder]

Discover magazine November 2001 had the Sloan Digital Sky Survey as it cover article - online at //www.discover.com It did a much better job of explaining the 3d process and the special nature of the telescope. The impression that most of the pictures are not looked at is very misleading. Each image is scanned by software to identify 500 or so "most interesting" light sources. The coordinates are then transmitted to a machine shop. There they make a custom aluminum plate with 500 small holes corresponding to the selected sources. The plate is returned to the telescope and later reattached with fiber optic cables attached to each of the holes. The telescope is repositioned to where it was when the original image was taken. Then the light from each of the selected sources can be run through a spectrograph. This allows them to determine the distance to the star or galaxy based on the red shift. It is this very accurate 3rd dimension over a huge number of sources that makes this survey so unique and exciting.

Asking for funding...

[flipflapflopflup]

[Scientists] "Right, we'd like some funding to build a map of the universe"

[Investor] "Sounds good. How are you going to go about it?"

"Well, we are going to get a really big camera..."

"click......"

Re:Asking for funding...

[grid geek]

And of course we need a "Cave" environment to run it in ...

"hard to say why people should study astronomy"

[da5idnetlimit.com]

"It's hard to say why people should study astronomy," said Gunn. "But in the scheme of human intellect, it is important to know where we came from and what's likely to be in store for us."

Oh, you mean this is a $85 million Horoscope Machine....

Re:"hard to say why people should study astronomy"

[teamonkey]

Well we already know from HIPPARCOS data that there's one star heading right for us. Due to pass close to us in few thousand years. Who knows what will happen? Can't remember it's name though, although if you're curious you could always search through HIPPARCOS data for stars with near-zero radial motion. --jaa

Re:"hard to say why people should study astronomy"

[Doctor Fishboy]

I think you mean a near-zero proper motion on the sky, and a large radial motion towards us.

A zero radial motion means the star is not heading towards us.

Dr Fish

What about this one ?

[kraf]

I've been playing around with this prog, it has quite a big 3d map in it.

Re:What about this one ?

[Mortenson]

You should go take a look at the Celestia project on Sourceforge. It is very well made. And free

http://celestia.sourceforge.net

All that Data and no-one to look at it?

[fruey]

So with all the data that few have seen, and few practical business applications, it seems to raise the question as to why are they mapping the universe.

Because it's cool, OK... and because some day the data will be useful, viewable, etc? It will be a map for space travel?

Each tape and a backup copy are sent overnight to Fermilab, where they are transferred to a host of Linux servers. Stoughton said the amount of data is small compared with Fermilab's other projects but is the largest capacity project ever assembled in astronomy.

Cool. They are using the penguin...!

"It's hard to say why people should study astronomy," said Gunn. "But in the scheme of human intellect, it is important to know where we came from and what's likely to be in store for us."

Well, it is interesting to know all about that. But collecting data that can't be used... tough cookie.

In general, these kind of projects get funded by curious people who can't use the data. Loads of data written to disks is not ever looked at, and this article raises that question. This is the discussion which interests me, quite apart from the greatness of some liquid nitrogen cooled super telescope with so many megapixels that at any kind of CRT resolution, for example, we would be decimating 99% of the data in order to get something reasonable to look at.

Re:All that Data and no-one to look at it?

[davecl]

SDSS has been running for quite some time, and has already produced some spectacular results. These include the most distant quasars known, and the largest most homogeneous quasar survey yet (in fact I'm working with some of this data at this very moment).

Whilst much of the data is not eyeballed by a person it is still processed, so there isn't a vast amount of data sitting on disk being ignored. Yes, people can look at the images, but there are too manmy of them and the human eye and mind are too inconsistent to be able to select items of interest in a coherent way. That's why we have computers! Particle physics operates in much the same way - you get a vast number of reactions in an accelarator and only a few of these are of interest, so you use computer-based filters to just select these. In much the same way with the SDSS images, most of them are of blank sky or of stars and galaxies that aren't of interest to the project. They're still cataloged and characterised though, and put into lists to be scanned by later data miners, while the specific goals of SDSS are dealt with at higher priority.

Re:All that Data and no-one to look at it?

[t]

Well, this like many things, is addressing the chicken-egg problem. Yes there is nothing to use the data but the reason for that is obvious, there was no data. Maybe now that we have it someone will find the motivation to do something world-changing with it. Or not.

t.

Re:All that Data and no-one to look at it?

[john_ee]

The general point is that astronomy is shifting to data mining rather than eyeballing. The juicy stuff for researchers isn't the raw image data, but the derived data. The system identifies objects automatically and then records a few hundred elements of "derived" data for each object identified. As a result, astronomers can just enter SQL-ish queries to do their research:

"Find all binary systems containing a white dwarf."

"Find all star-like objects that ar X% rare."

"Find objects with characteristics similar to quasars with redshifts between X and Y."
etc...

Creative Depth?

[hcdejong]

...a really creative way to add "depth" to the image

Really creative, yes. Pity we don't get to hear about it. Or am I overlooking something?

Re:Creative Depth?

[Gary Yngve]

It looks like they use red-shift data: MSR link

On that note, I heard Jim Gray give a talk the other day (5th Turing Award winner I've heard talk) on the database/scalability aspects of SSDS. Cool stuff.

I guess it may be tricky to analyze multiple wavelengths via CCDs... my guess is they would need some sort of splitter to guarantee that pixel (37,52) on the near-infrared CCD corresponds to the same light as pixel (37,52) on the far-infrared CCD. A quick google search seems to indicate that some Australian dudes did some work with this stuff: news story

Re:Creative Depth?

[hcdejong]

It looks like they use red-shift data

But using red-shift isn't creative. It's one of the standard ways of estimating astronomical distances.

Excellent! Finally SDSS Gets Public!

[hondo_san]

Howdy all. I've been following the project online for over a year. The cool part is that this is sort of a googlebot for the heavens. See: http://www.astronomy.com/content/dynamic/articles/ 000/000/000/502vwthx.asp

Impossible

[Czarnian]

A true map (100% correct - however see my comment below) cannot be currently made, the science behind the calculations needed to make such a map is uncertain (dark matter, universal expansion speeds, unpredictable effects effects of undiscovered objects - black holes, etc, astrophysics is evolving all the time) and you'd a Beowulf cluster of processors the size of a galaxy to do the math.

In fact, even if we had all the science needed to make the calculations and the equipment to do so, a true map is theoretically impossible, based on the Uncertainty Principle it is impossible to determine with 100% accuracy the state of even an atom, let alone a universe.

Re:Impossible

[phunhippy]

Sheesh.... take all the fun of it!! hehe

Re:Impossible

[t]

You're right. It's impossible. We shouldn't even try.

And by the way, phunhippy said "... show its theoretical place today" not 100% accuracy.

t.

Re:Impossible

[Gary Yngve]

Oh, give me a break. The original poster wanted a 3-d map of the universe. By your argument, we don't even have true maps of Earth.

A map with "approximate" positions of the stars/galaxies would be plenty good. Oh no, the sun is off its true center by the diameter of an apple? I am wetting my pants! I can even see the difference of the sun's placement on my super-duper 1e10x1e10 resolution monitor (that provides the ultimate resolution for viewing porn: not only do I see all her parts, I see all the parts of the gonorrhea bacteria that she has).

The real interesting question is how does one navigate [in a user interface] around such a sparse map?

Re:Impossible

[oldmacdonald]

That should be a 1e10x1e10x1e10 monitor.

Re:Impossible

[FreeUser]

A true map (100% correct - however see my comment below) cannot be currently made, the science behind the calculations needed to make such a map is uncertain (dark matter, universal expansion speeds, unpredictable effects effects of undiscovered objects - black holes, etc, astrophysics is evolving all the time) and you'd a Beowulf cluster of processors the size of a galaxy to do the math.

While the last portion of your comment (needing a cluster the size of a galaxy to do the math) isn't really true, you are correct in your assertion that we lack the scientific knowledge, and likely the computational power, to make an accurate map of our galaxy as it is today. In other words, if we invented superluminal propulsion today we'd have to make our maps the old fashioned way ... by going there and surveying the space a la Star Trek.

However,

In fact, even if we had all the science needed to make the calculations and the equipment to do so, a true map is theoretically impossible, based on the Uncertainty Principle it is impossible to determine with 100% accuracy the state of even an atom, let alone a universe.

The uncertainty principle only applies to subatomic scales at the quantum level, not to macro objects like planets, stars, and galaxies, or even smaller objects like grains of sand. Calculating the location of every star and planet, if we had the scientific knowledge and computational capacity to do so, is perfectly possible in theory ... just not practical given the current state of the art (and our current ignorance, as you pointed out before, with respect to undiscovered objects, dark matter, etc.).

What fascinates me...

[petis]

.. is the creative process behind projects of this magnitude. Where do people get these ideas? Is the scenario below how it happened?

- Hey, I just got this great idea!
- Uhu, what's that?
- Let's make a 3D map of the entire universe!
- Sounds cool, let's get some funding!

Or is it more like:
- Hmmm, this problem would be easily solved if we had a 3D map of the universe..
- Sounds cool, let's get some funding for that project!

Or the third alternative:
- Hmmm, we need some funding!
- Sounds cool, let's make a 3D map of the entire universe!
- Great, that will keep us running for a couple of years!

:)

Re:What fascinates me...

[TMB]

That second one's pretty accurate. :-)= Though it's more than just one problem that they had in mind, the main motivator is understanding galaxy formation and evolution.

Of course, you forgot the third line... ;-)

- Hmmm, the Sloan Foundation will only give us the money if we name the survey after them. Fine by me!

[TMB]

Forgive me, but... `news'??

[Cally]

I know this is a lame sorta thing to say but really, the Sloan DSS has been running for... what, 3? 4? years now? Interesting, I grant you, but hardly news.

Now the 2df galaxy cluster mapping project which are giving us maps of our galaxy's position out to about 1B light-years -- /that's/ interesting AND news. hell,

Re:Forgive me, but... `news'??

[john_ee]

To me, the big news is that the NVO (National Virtual Observatory) got funded - $10 million from NSF's ITR program. Their proposal is here. In a nutshell, the point of the NVO is to federate as many sky surveys as they can under one interface, and in one pixel space.

Now in stock: 0.1Gpixel Web Cam

[ArcSecond]

Each detector is rated at 4 megapixels, giving the 30-CCD array a whopping 120-megapixels (120 million pixels) sensitivity. The higher the number of pixels, the greater the resolution.

Great!! Where can I get one? Does it have USB 2.0 or FireWire? And how many hi-res shots can I fit in memory? Does it take SmartMedia or CompactFlash?

I just hope that I can find a big enough LCD to view these pix at 100%...

Re:Now in stock: 0.1Gpixel Web Cam

[zer0vector]

The quality of astronomy grade CCD chips is so far beyond anything you've seen in commercial digital cameras, and so much more expensive. I've used a 2048x2048 chip, cost on the order of 50k, and keep in mind when they sell it to you you don't get anything but the chip, you have to build the rest yourself. Oh yeah, I wouldn't be waiting around to make a webcam, the single readout time for the full chip was about 2 minutes. Luckily this particular chip was recently upgraded to quad readout.

Re:How to view a 4D object in 3D

[Samedi1971]

We can use a similar technique to visualise a 4D object. By the way, dimensions come in different sorts. Our three spatial dimensions are bidirectional but time is monodirectional and we can only travel through time in one direction.

Time is an observational reference, not a measured dimension. We don't move forward in time, we experience infinitely small discrete moments, which do not necessarily sync with someone else's experienced moments.

My fourth dimension is quite clearly a bidrectional spatial dimension.
...A suitable object to conceptualise is a person changing from a baby to a child to an adult to a public servant.

Huh? Let's recap his logic:

1) Time is monodirectional
2) 4th dimension is bidirectional
3) 4th dimension can be compared to a human growing from baby to adult

Logical conclusions: Human growth has nothing to do with time, and is bidirectional. Oopth (or human growth and career path) can be measured using common spatial measurements.

I can only guess at the number of kilometers I've traveled in my progression from embryo to my current job title.

Good application to all that data

[hyrdra]

I can think of a good application for this data.

Why doesn't the SDSS code up a distributed program like SETI to help in the analyzation of all that data to find something unique or unknown. Convert the pictures to 2D FFT's and have a set of known astronomical element FFT's and then do constant comparisons against this set to see if there is anything "unknown". I'm sure it would be more complex than this, but this is how visual image recognition works so I assume it could be tailored for this application.

I would certainly download an run an application that looks for new things in astronomy. I'm sure others would as well. Somehow it's slightly less frivilous than what SETI is doing and we stand to gain more in a quicker amount of time.

That way, when we do actually find something that looks interesting SETI would know where to point that big antenna...

Re:Good application to all that data

[grid geek]

Why doesn't the SDSS code up a distributed program like SETI to help in the analyzation of all that data to find something unique or unknown

There are several reasons why this hasn't been done.

• The amount of data is fairly trivial compared to the Particle Physics data (AstroGrid, EU DataGrid, Grid Forum
• Trust. As in lack there of. SETI had problems with people returning bogus results in the past and had to spend time (and several articles) on improving security and getting the same unit processed by several seperate users.
• Lack of man power / time.
• Dependability - e.g. an astronomer wants to run an analysis now! Is there any certainty that there will be x amount of processing power available? That your computer will still be on the network in 5 minutes time? That the unit will get processed in a reasonable real time?

It's ok for seti to send out a unit and not get it back for a week, or a month but not for things which have to be done right now. More importantly seti units are discrete, they dont depend on other units for the results.
Physics data can & therefore any machine would have to have an always on connection to communicate with other machines. Bye Bye to your bandwidth.

Re:Good application to all that data

[p3d0]

Did you really just say "analyzation"?

Re:Good application to all that data

[john_ee]

Automated object identification is already a standard part of any sky survey data pipeline. Among other things, this is used by surveys like SDSS to not only find and identify objects, but to tag the ones that are galaxies for later redshift measurement. (That's where the "3D" part comes in - redshifts of galaxies. Measuring radial ditances to closer objects - say stars in our own galaxy - is a pretty involved bootstrapping process where distance is derived from a ladder of indirect indicators.)

As far as grid computing goes, the US NVO (National Virtual Observatory) will implement a grid computing model. The NVO will be a unified portal that will federate about 50 digital sky surveys. It will have a database of billions of objects with hundreds of characteristics each, and rich relational structure spanning those hundreds of dimensions. They'll use grid computing to perform astronomers' queries on those data. That's really the main reason why these sky surveys are being done - to allow astronomers to do their work by data mining. A "map" of the universe will be a byproduct.

Re:Good application to all that data

[hyrdra]

Yeah, heh, too early in the morning.

They need our help

[Scorchio]

From the article : "Surprisingly, with all the money and time spent in the quest for a road map of the celestial past, "most of the pictures have never been looked at," Stoughton said. Stoughton said that because of the immense amount of information seeing any part of it would take a lifetime"

What they need is SETI-like distributed software than farms the pictures out to us to look at, and we'd get through them in a week or two. Or stick them all on a website - www.AmIAMinorAstronomicalAnomaly.com - with user rankings. Job done.

Distance from earth

[zoon0]

'A really creative way to add "depth" to the image'

Well, this superlative piqued my interest. Unfortunately I then read the linked article (yes I do realize that isn't the done thing). The "really creative" way?

Red shift

I don't mean to undermine the goals of the project, which are clearly noble. But the top level comment is rather tabloidy.

I do have a serious question. What kind of accuracy do you get from this data? I understand latitude and longitude (or psi and phi) can be given to a tiny fraction of an arcsecond, but how about distance from earth? +-10%?

A flame may be fleeting but obscurity is forever.

Re:Distance from earth

[(void*)]

The redshift of an object can be measured really accurately to about 1 in a million. The important thing about the SDSS is that you can get good photometric, not spectrographic redshifts.

Re:Distance from earth

[FastEddieCoder]

They are also getting lots of redshift data. See Discover Mag Nov 2001. The Chicago Tribune article completely left out what is really unique about the telescope. They can do several thousand redshift determinations a night in sync with the photometric images.

Tricky part

[JJ]

The tricky part of this mapping is that any images recorded represent only a single snapshot of events. 15 billion year old quasars may be long gone by now, but we can't tell. We have very recent images of close stars, somewhat recent images of most of our galaxy and ancient images of most distant galaxies. Any map produced is really just a reflection of this snapshot of images from the past.

Insider's view

[srhuston]

Since I'm the sysadmin at Princeton's astrophysics department, perhaps I can shed a few more links for the picture-hungry (and the information hungry):

• SDSS image gallery
  
• SDSS Project Book
• Astro's webpage

You have travelled kilo-oopths (lol)

[Azahar]

You said
"Time is an observational reference, not a measured dimension. We don't move forward in time, we experience infinitely small discrete moments, which do not necessarily sync with someone else's experienced moments."

Can you extrapolate on that a little. I am seriously interested.

Just find the fractal

[Zero__Kelvin]

If one wants a true 3D map of the universe they should be seeking the fractal formula that describes it, IMNSHO. Anything less is a fundamentally innacurate map.

really big

[trb]

The goal is to produce a 3D map of the universe using a really big digital camera...

Of course they need a really big camera, they're making a map of the whole universe!

Re:really big

[Ioldanach]

Now just how far back do you have to stand to fit the whole universe in the frame?

Re:really big

[PlaysByEar]

Kinda far, you're going to have to walk back a bit more. You'd better take the picture yourself--if you give the camera to grandma she's bound to stick her finger in front of the lens and block a few million galaxies. Or she'll cut the cosmos' head off.

Ouch

[NiftyNews]

A sky survey? Man, that sounds like a rough job...

"Excuse me Mr. Cloud, but what do you think about the war in Afghanistan? Hello? Cloud? Okay, how about chocolate milk, do you like it? Hey, I'm talking to you! Where are you going? C'mon, answer my question! Dammit..."

Re:Ouch

[$uperjay]

Silly researchers, Cloud never talks. You ask the *supporting* cast in Square games when you want answers. Cloud will just say '...' and hit you with his sword.

Too much data

[overlord]

This people has so much data, that It will take a lot of time to make that map.

OverLord

Can this be used for torture?

[clmensch]

I don't remember the name of it, but it seems that this could be used for a torture chamber described in the Hitchhiker's Guide to the Galaxy series! The gist of it was that the victim entered a chamber where the entire Universe was projected all around him in stunning detail...and the sheer unthinkable magnitude of it all made the victim feel so minor and insignificant that it killed them.

I remember reading it and thinking that's not only frickin' hilarious, but it's COOL!

Re:What the heck?

[chris_sto10]

Alex and his team design and build databases used for distributing results to astronomers and the general public. See the SDSS skyserver to see what they have done.

Application of the project.

[Merik]

Seems the whiners are having trouble seeing merit in this project.

I didn't see it mentioned, but astronomers always complain about our inability to track dangerous objects(like we could only track ten percent of the sky). It seems that with a few iterations of this project, when a few complete pictures have been taken, on could begin to plot the course of the objects that move with simple algorithms.

Asking why a project such as this is worthwile is like asking why where decoding DNA.

Re:What the heck?

[john_ee]

Not to mention he's co-PI on the National Virtual Observatory project!

liquid nitrogen

[Alsee]

cooled by liquid nitrogen to keep moisture out

Excuse me? Somehow I think there are easier ways to keep moisture out LOL.

-

Re:liquid nitrogen

[linzeal]

Liquid nitrogen is pretty cheap. Especially if you are extracting it on site.

Re:How to view a 4D object in 3D

[DrMegaVolt]

Time is a name for our experience of a dimension with physical extent. The nature of our consciousness makes it so we can only "remember" it in one direction and "predict" it in the other.

Its physical extent is well known d = ct, where d is distance, c is the speed of light and t is the time elapsed

If you are 30 years old, your physical extent in the temporal direction is
300,000km/s x 30y/d x 365d/h x 24h/d x 60m/h x 60s/m = 283,824,000,000,000 km

no wonder I cannot remember my childhood so well, its really far away!

Distributed Human Processing

[MagikSlinger]

Wouldn't it be cool if...

We use a big cluster of *nices to serve the data up to everyone on the Internet. Every amatuer astronomer on the web can then visit the site once a day or so and cruise the catalog. All those eyeballs looking for stuff will do something.

Specific projects I can think of are:

• Near-Earth Asteroid/Comet spotting

3d part is better explained in Discover Nov 2001

[FastEddieCoder]

Discover magazine November 2001 had the Sloan Digital Sky Survey as it's cover article (online at //www.discover.com) It did a much better job of explaining the 3d process and the special nature of the telescope. The impression that most of the pictures are not looked at is VERY misleading. Each image is scanned by software to identify 500 or so "most interesting" light sources. The coordinates are then transmitted to a machine shop. There they make a custom aluminum plate with 500 small holes corresponding to the selected sources. The plate is returned to the telescope and later reattached with fiber optic cables attached to each of the holes. The telescope is repositioned to where it was when the original image was taken. Then the light from each of the selected sources can be run through a spectrograph. This allows them to determine the distance to the star or galaxy based on the red shift. It is this very accurate 3rd dimension over a huge number of sources that makes this survey so unique and exciting.

Kstars

[SgtChaireBourne]

kstars is fun, too. Sometime, I'd like to take one of these planetarium progams and find a way to keep an updated star map as a desktop background.