Slashdot Mirror

There are a number of projects out there to develop specialised telescopes that will be able to take quite high resolution photos of unprecedentedly large areas of the sky at once, and big enough to gather enough light to show reasonably faint objects without needing too long an exposure. Look at the Large Synoptic Survey Telescope, for instance https://www.lsst.org/. This aims to photograph all of the "available" sky (it's in Chile, so it never sees the stars around the North celestial pole) every few nights for 10 years. There's lots of infomation on their site and in their papers, but a few numbers that jumped out at me: 8.4m primary mirror, 3.2 GPixel camera, 15 TB of data each night!

Even this would have to get moderately lucky to see a supernova as young as this one, which was captured in it's first minutes or hours. It would also, ideally, need to identify what it was seeing almost instantly, so that it (or another telescope) could start a follow-up within seconds or minutes.

Maybe you should read again yourself? http://www.lsst.org/about/dm
According to their own website, the LSST will produce 15TB of data each night. So your back of the envelope calculation to fit on a 3TB disk is wrong. Yes, the amount of raw data is not as large as for the LHC, but that really is an apples to oranges comparison. The LHC data is heavily reduced by multiple levels of triggers. It would be impossible to read out everything. The data that is finally analysed has gone through many, many steps of background event reduction. The LSST on the other hand captures images. The analysis requires those images. So the challange is not so much storing the data, but being able to do analysis on that data set.

data from impact areas on moon, discovery rate, increase in counts with improvement in instruments are some factors:

http://www.lsst.org/lsst/publi...

• Immediately: take an introductory astronomy course at a local community college or continuing education program at your local university to demonstrate your interest,
• Then: assess your IT skills, and apply them directly to the support of an upcoming large ground- or space-based observatory. This is an especially sensible route if you do any database related work. The future of astronomy is big data and massive virtual observatories which collect together and make useful petabytes of information from a wide variety of facilities.
• Check the job listings at the American Astronomical Society, looking in particular for IT support positions where your domain knowledge would outrank that of PhD-trained astronomers (who learn to program "on the job" and rarely master grittier back-end systems). Realize that almost all existing and (especially) new astronomical facilities have substantial IT/engineering staff, and that your skills do not exist among traditional PhD scientists. Example: the LSST will produce 30 TB of data per night, which needs to be processed in semi-real time. Example #2: the incredibly successful Sloan Digital Sky Survey partnered with Microsoft database engineers to build its (at the time) state-of-the-art public-facing data archive. The late Jim Gray was instrumental in building the Sloan backend, and said his favorite thing about astronomical data is that it was "worthless" (by which he meant the usual access control layers were not necessary, freeing him to focus on much more rewarding and useful tools).
• Relocate to a mission control or operations center for the facility. These are often located at major research universities, or equivalent national facilities like the Space Telescope Science Institute in Baltimore, the National Radio Observatory in Charlottesville, VA, the Gemini Observatories (Hawaii/Tucson/Chile), etc. Advantage? You will very likely be immediately mixed in with groups of professional astronomers. You will be strongly encouraged to learn to speak their language, and to become more involved in the scientific aspects of the project. You will learn a great deal just through osmosis. You will likely be able to attend seminars, sit in on classes, bend the ear of willing faculty, etc. And the most significant advantage? You could be contributing directly to the forefront of astrophysics research within 3-5 years. Disadvantages: the pay might be somewhat less than similar background applied in the financial or health industries. Often the intellectual rewards bring talented engineers anyway. Also, may projects are time limited, so you positions are typically not permanent (but new projects are coming online all the time).

I work as a programmer & sysadmin supporting a solar physics archive. Although most scientists these days have to learn how to program to some degree (to be able to analyze their data), there's still a large number of IT people who work in these fields -- as programmers, sysadmins, DBAs, etc.

So, if you're in the Tucson, AZ; Menlo Park, CA; Princeton, NJ; or Seattle, WA area, keep an eye on the LSST hiring page.

There are likely to be other projects out there hiring, but I don't know what their various situations are. (I just know that LSST was soliciting at the last American Astronomical Society meeting). You can also look to universities, especially if you have kids (as future tuition benefits for dependants can be quite significant).

I know a hell of a lot more about astronomy & solar physics than I do before I started this job. I'm by no means an expert in the field, but my work does help the scientists do their research and improve our knowledge of the field.

Whoah. Are you even remotely aware of what is being done in cosmology these days?

Planck Sloan Digital Sky Survey
Square Kilometer Array
Ice Cube
Large Synoptic Survey Telescope
Euclid

Hardly "ideologically/branding driven pseudoscience". Who the hell modded you up?

Telescopes already use the convention:

Large
Very Large
Giant
OverWhelmingly Large

The 100 meter limiting size for significant near-Earth asteroids corresponds to a limiting magnitude of 26. In addition, short exposures are needed since the asteroids trail very quickly at more than 20 second exposures. So large aperture is important. This instrument would be unique. Its utility is diminished if it cannot cover the entire visible sky several times per month as there would be no competing telescopes to cover its holes or even follow its discoveries. At the present time, systematic surveys like Spacewatch have difficulty finding all 21st magnitude near-Earth Asteroids.

With its capability to detect objects as faint as 25th magnitude in 15 seconds, only the LSST will be able to find virtually all significant PHAs 100 meters in size and over 50% of all NEOs 100 meters in size. During its survey of the sky LSST can find 90% of the PHAs over 140 meters in diameter.

A PHA is a Potentially Hazardous Asteroid.

The Chelyabinsk meteor had an 85 meter size, so it would most likely not be found by LSST. There are some other studies to use satellites in the IR band to look for smaller size objects.

The camera covers a little less than 9.7 square degrees, not the whole sky. (It's not a square image but an array of sensor chips, the array is missing corners to more closely follow a circular image shape.)

The page http://www.lsst.org/lsst/science/concept_camera lists the sampling resolution as "better than 0.2 arcseconds" (with 6 color bands per pixel 300nm-1200nm). That would make the moon 9000 pixels wide (assuming 0.5 degree width - it varies a little).

More to be said - here's the scientific FAQ: http://www.lsst.org/lsst/faq-science
Choice bits:

...That combination is unique: wide field of view (10 square degrees), short exposures (pairs of 15-second exposures), and sensitive camera (24th magnitude single images, 27th magnitude stacked). ...
The etendue of LSST is 320 square meters square degrees. A primary mirror diameter of 8.4 m (effective aperture 6.7 m due to obscuration) is the minimum diameter that simultaneously satisfies the depth (24.5 mag depth per single visit and 27.5 mag for coadded depth) and cadence (revisit time of 3-4 days, with 30 seconds per visit) constraints....
The nominal high-SNR sample defined by i25 for point sources) will include four billion galaxies (55 per square arcminute) with the mean photometric redshift accuracy of 1-2% (relative error for 1+z), and with only 10% of the sample with errors larger than 4%. The median redshift for this sample will be z=1.2, with the third quartile at z=2. ...

Q: Will the full resolution, full depth image data be available to download?

A: Yes. There will be a range of data products and download portals. The LSST data system is being designed to enable as wide a range of science as possible. Standard data products, including calibrated images and catalogs of detected objects and their attributes, will be provided both for individual exposures and the deep incremental data coaddition. For the "static" sky, there will be yearly database releases listing many attributes for billions of objects. This database will grow in size to about 30 PB and about 20 billion objects.
As in the SDSS, we expect a power law of user interactions with the data. At one end of this distribution are simple lookup queries or color jpeg cutout downloads. At the other end are huge statistical calculations over the entire database, and image operation scripts on billions of objects. The data management system is budgeted to handle most but not all of that distribution. Institutions joining LSST early, and members of the LSST Science Collaborations, will have the customary advantage of deep familiarity with the LSST system and survey.

*WILL* generate. LSST isn't operating yet.

And yes, 30TB is a lot of data now, but we have some time before they finally have first light.

Operations isn't supposed to start 'til 2019 : http://www.lsst.org/lsst/science/timeline

We just need network and disk drive sizes to keep doubling at the rate they have, and we'll be laughing about how we thought 30TB/night was going to be a problem.

SDO finally launched last year with a date rate of over 1TB/day ... and all through planning, people were complaining about the data rates ... it's a lot, but it's not insurmountable as it might've been 8 years ago, when we were looking at 80 to 120GB disks.

Although, it'd be nice if monitor resolutions had kept growing ... if anything, they've gotten worse the last couple of years.

(Disclaimer : I work in science informatics; I've run into Kirk Bourne at a lot of meetings, and we used to work in the same building, but we we deal with different science disciplines)

The Large Synoptic Survey Telescope will use a 3.2 Gigapixel camera to take 15 second images of a 9.6 square degree view (~50x the area of the moon) in 6 color bands. This will image the entire night sky in 3 days with amazing detail (30 terabytes a night) for 10 years. This upgraded planetarium seems like a great way to view some of these images and movies of the real night sky.

http://www.lsst.org/

I admit, portability suffers a bit at this point, but aren't your pictures worth it?

I admit, portability suffers a bit at this point, but aren't your pictures worth it?

How about some balance?

- The cost of doing almost anything, anywhere in Antarctica is not far short of a space mission.

Nonsense. Sure, it's more expensive than putting a telescope on Kitt Peak, Mauna Kea, or Chile. But you're still orders of magnitude away from a space mission. A half-meter telescope on a "small explorer" (SMEX) NASA mission is over 105 million dollars, and that doesn't include the launch costs. Getting that 250 kg into space costs on the order of $20,000 USD per kg, still a fairly conservative estimate.

Based on the overland traverses that the Italians and French undertake to Dome C per year, getting to a site like Ridge A would be more like $10/kg (naturally assuming that you're making good use of the traverse and taking lots of stuff up there in one go).

So the costs aren't even in the same ball park.

- It's "daytime" for at least half the year.

And infrared and submillimeter astronomers can observe during the day. Incidentally, most of the big outstanding questions about the assembly of galaxies and star formation will be solved at these wavelengths -- which is where the Antarctic atmosphere is most advantageous.

- You can see barely half the sky - probably less.

You get the Southern sky only, true. But most of the Milky Way is in the South, and you can observe it without interruption -- 24/7. Time domain astronomy is something we've only scratched the surface of -- and there are major new projects devoted to it such as LSST. Antarctica could play a significant role here.

All things considered, Hawaii and Chile are far superior in most respects which matter.

As long as you ignore the poorer image quality, unstable atmosphere with large diurnal variations, comparatively soggy atmospheric water content, 100x higher infrared background -- yeah, Chile and Hawaii are far better. :)

Over 30 thousand gigabytes (30TB) of images will be generated every night during the decade -long LSST sky survey.

Really small rocks are hard to see.

Especially really small dark colored ones.

It should be taken as a sign of our improving detection ability that we were able to see this one before it hit at all.

Detecting and mapping the orbits of all of these near earth asteroids is one of the purposes behind the LSST project that we have been hearing so much about lately.

http://www.lsst.org/lsst_home.shtml

The 30 terabytes of data obtained each night

Obviously, this is a great achievement deserving of the /. homepage...
However, I'm more interested in hearing about how they are going to process/archive/use that much data!

I'll be honest and say that I'd never heard (or at least remembered) anything about the LSST, so I just did a brief lookover of their site and it seems like a ridiculously cool project.

LSST will rapidly scan the sky, charting objects that change or move

That means it will have to store multiple versions (history) to be able to do trend analysis. So at multiple TB's of data, how exactly are they planning on processing/analyzing it?!?

As is traditional in the US for many large ground-based telescopes, the LSST is a public-private project. Private support leverages even larger federal support. This traditionally has been true even for facilities where the data was not public immediately. LSST breaks with that tradition in that the data and data products from LSST are immediately public, without a proprietary time period. Thus, private funding for LSST supports open access.

It also mentioned that they are hoping that individuals (and groups) do interesting things with the data...but seriously, I was talking with my brother the other day about how cheap 1TB disks were now. But even if they were only $100 a piece, you'd still be dropping some major $$$ just to be able to begin to do anything (comprehensive) with it.

The 30 terabytes of data obtained each night

Obviously, this is a great achievement deserving of the /. homepage...
However, I'm more interested in hearing about how they are going to process/archive/use that much data!

I'll be honest and say that I'd never heard (or at least remembered) anything about the LSST, so I just did a brief lookover of their site and it seems like a ridiculously cool project.

LSST will rapidly scan the sky, charting objects that change or move

That means it will have to store multiple versions (history) to be able to do trend analysis. So at multiple TB's of data, how exactly are they planning on processing/analyzing it?!?

As is traditional in the US for many large ground-based telescopes, the LSST is a public-private project. Private support leverages even larger federal support. This traditionally has been true even for facilities where the data was not public immediately. LSST breaks with that tradition in that the data and data products from LSST are immediately public, without a proprietary time period. Thus, private funding for LSST supports open access.

It also mentioned that they are hoping that individuals (and groups) do interesting things with the data...but seriously, I was talking with my brother the other day about how cheap 1TB disks were now. But even if they were only $100 a piece, you'd still be dropping some major $$$ just to be able to begin to do anything (comprehensive) with it.

Why is the tertiary mirror larger than the secondary? That's not like any telescope that I'm familiar with.

The LHC will generate several PB of data per year, as will the Large Synoptic Survey Telescope. These projects aren't all that uncommon.

The article included a lot of details on the immense Large Synoptic Survey Telescope which will dwarf Pan-STARRS when it's done in ~2016. (LSST is in the south, Pan-STARRS is in the north, so they don't really compete) The specs for the LSST data boggle the mind- the thing will cover the entire southern sky every 3 days down to 24th magnitude, generating 30TB of data a night or ~13petabytes per year and having over 100 TFLOPS of computers devoted to sorting it. Read the specs here: http://www.lsst.org/About/lsst_baseline.shtml

...Looks over at his little 6" and 8" scopes and sighs...

Actually, Pan-STARRS is the el cheapo quick and dirty version of the near-Earth survey. Look up the LSST (the Large Synoptic Survey Telescope), currently under construction. http://www.lsst.org/

• Ten times as sensitive (an 8-meter mirror) so it can detect down to 100m objects -- thirty times as small.
• A 3.2 Gpixel camera.
• An image every 15 seconds, doing a complete raster scan of the sky every three days.
• 30 TBytes of data PER NIGHT, and they plan to keep it all for ten years.

Google volunteered to be involved in the data handling, and Bill Gates and Charles Simonyi have contributed 30 million dollars to the construction. During the initial design, the astronomers actually said, "By the time the LSST goes online (2014) we expect that Moore's Law will allow us to process the data stream."

Well, it doesn't mean that the lens sees it, but that the lens can see the effect it has on the things you _can_ see. For instance you look at a galaxy field and you notice that some are distorted in certain ways, you can infer that there's a hidden mass between you and those galaxies. The LSST project on which I work has a similar goal.

This MS product does indeed sound very similar to Google Sky.

I think the difference between both of these and e.g. Stellarium/Celestia is the database that sits behind them. Usually "planetarium" software consists of a bunch of points for stars, with perhaps a few objects represented by pixels. You can upload images but you have to do it yourself.

In contrast, Google Sky (and presumably the MS telescope) show you pixels from large databases such as the Sloan Digital Sky Survey. The latter covers roughly 1/4 of the whole sky.

Google is heavily involved with the LSST project.
MS has been involved in the Sloan Digital Sky Survey for quite some time via the late Jim Gray.
Its great for astronomy that both of these companies are competing in an area with little prospects for "monetization".

Before I look at some of the eight specific physical mechanisms in the ECM, a note about the PDF document itself.

It is sub-titled "Poster Presentation Institute of Electrical and Electronics Engineers 33rd International Conference on Plasma Sciences Traverse City Michigan June 4-8, 2006". For an example - in terms of references, sources, etc (not content) - of what astronomers are used to, consider this one (big!) page LSST poster, from the 209th Meeting of the American Astronomical Society in Seattle, early last year (it's about the same size as the ElectricComet document) http://www.lsst.org/Meetings/AAS/2007/JanPosters/newman_086.06.pdf. Note too that ADS will turn up lots of papers by the authors of this LSST paper, but none by either W. Thornhill or D. Talbott.

Three of the eight specific physical mechanisms are easily addressed (EDM and electrostatic cleaning of the surfaces of comets, electrostatic deposition of dust and debris on their surfaces) - the 21 page PDF contains no external references, and only the following internal ones (simple re-statements omitted):

- the image titled "Carving of Surface Relief" has a caption which includes "the surface on the right, produced by electric discharge machining (EDM)" (note there's nothing to say whether "electrical discharge machining" is the same as "electric discharge machining" or not)

- "The jets flare up and move over the nucleus irregularly, leaving scars typical of electric discharge machining" - this repeats the link of EDM to the action of jets ("The observed jets of comets are electric arc discharges to the nucleus")

- "The asteroid appears to have attracted considerable surface debris electrostatically".

Note that the first time the terms "electrical discharge machining" and "electrostatic cleaning" are used, they are in quote marks, signifying that they have special, non-standard meanings (if the authors follow a common orthographic convention) ... yet no definitions are given.

So two of the three mechanisms are empty - you could rewrite the parts of the document which mention them using nonsense words and they would have just the same meaning.

EDM is linked to "jets"; next comment I shall examine four (of the other five) mechanisms (the formation of coma and tail, of jets and filaments, maintenance of coma and filaments) and also look at whether there's any more meat to EDM than there is to "electrostatic cleaning" or "attracted back to the nucleus electrostatically".

(to be continued)

That is nice to hear. Given that also Google is involved

http://www.lsst.org/News/google.shtml

it is not all that surprising I suppose. If there were any rivalries then certainly between MS and Google.

Their homepage mentions though that it is meant to take a whole sky survey over a short period of time, i.e. a couple of days (I'm confused about this actually). With that capability all kinds of transient phenomena could be observed.

I'm not sure that NEOs do have the highest priority there. Although this is certainly part of the mission and certainly would have the broadest impact with people.

The LSST and Google have also announced some degree of collaboration: http://www.lsst.org/News/google.shtml.

Indeed, an ex-Google "VP of Engineering", Wayne Rosling, joined the LSST project in June 05. That Google announced a joint effort with the LSST some time later is not therefore totally surprising--sometimes it's who you know.

LSST and Google share many of the same goals: organizing massive quantities of data and making it useful. Over 30 thousand gigabytes (30TB) of images will be generated every night during the decade -long LSST sky survey. The massive amount of data from LSST must be managed efficiently and analyzed in real time. Key areas in the Google-LSST collaboration will be: organizing the massive ingestion of information, processing and analyzing the continuous data streams in a 24/7 fault tolerant manner, enabling the new discoveries coming out of the LSST to be made available to the public and researchers in real time, and working with and managing large parallel data systems. In addition to aiding professional scientists and amateur astronomers, properly organized the LSST data will generate a new and dynamic view of the night sky for the public. LSST data will be valuable to curious minds of all ages, and will provide a powerful teaching tool.

In applying for membership, William Coughran, Google VP of Engineering, said "Google's mission is to take the world's information and make it universally accessible and useful. The data from LSST will be an important part of the world's information, and by being involved in the project we hope to make it easier for that data to become accessible and useful."

OTOH, it will generate 30TB per day.

According to http://www.lsst.org/About/Tour/software.shtml
"Current projects show that approximately 5000 mathematical operations are required per pixel of the image to process and classify survey data. Scaling this to the size of the LSST data stream shows that approximately a thousand of today's high-end processors will be required a feasible proposition. Advances in processor power over the next five years will reduce this number to a few hundred, by which time the required LSST computer system will seem quite pedestrian. Storing this data is also well within even today's technology. At current prices, a one-petabyte disk storage system costs less than $1 million; in five years this price should drop to well below $100,000. Keeping all of the LSST data online will certainly be affordable."

Windows may not play a central data reduction role, unless Microsoft can support 100 CPUs within the next six years. Of course six calendar years is a long time in techo-years. By then, perhaps the data analysis would be done on game consoles.

http://www.wired.com/techbiz/it/news/2007/10/ps3_supercomputer
http://www.physorg.com/news92674403.html

I'd guess that much will depend upon how much can run on a cluster, vice how how much must run on a SMP machine.

Here are links to the Large Synoptic Survey Telescope (LSST) home page and its layout and construction.

Here are links to the Large Synoptic Survey Telescope (LSST) home page and its layout and construction.

Yes, it probably will.

All modern control systems for research telescopes and instruments involve a supervisory layer and that is often run on a Unix or Unix-like system. LSST also has to do an unprecedented amount of soft-real-time processing on the data stream (see their tour page, and this kind of astronomical software typically runs on Linux and/or Unix.

The biggest one I know of is the Large Synoptic Survey Telescope, which is still being designed. They're pushing for a 3.2 Gpixel camera. Basically, it's an array of 201 16Mp CCDs.

I was talking to one of the folks dealing with their data infrastructure back in April -- they're expecting 6 petabytes of data per year, and are likely going to have to reformat and reprocess on the fly, rather than store processed and formatted data.

You ought to have Google store that data for you. Seriously.

Google has collaborated on other scientific projects before, and one in particular has many of the same needs as the LHC, the LSST. Of course, it doesn't hurt that one of the primary backers of the LSST is an ex-Google exec.

I'm confident that Google is capable of dealing with large data stores, even those on a multi-PB scale, with reliability and redundancy.

Don't worry, it's coming. I've seen previews of Google Sky at a couple of astronomical conferences so far. Also, check out partner number four for the Large Synoptic Survey Telescope.

If there is a life on earth ending event occurring from some asteroid they COULD find, does it matter at all? There is nothing we can do about it anyway.

Whoally shit, the tertiary really is 5.0 meters! Check this out: http://www.lsst.org/Images/images/optlayout.bmp

here's a background paper on the "data challenge" - http://www.lsst.org/Project/docs/data-challenge.pd f

seriously...where's the "Stuff that matters"....
if you dig enough you can find it based on the linked article. But if you go to slashdot for the sole purpose of not having to, you can find it here: http://www.lsst.org/About/datamgmt_fac.shtml

There is government funding. See Pan-STARRS and LSST. These surveys will find millions of objects, and thousands of PHOs.

The advantage to small telescopes is the ability to stare at a large section of the sky at once (in the case of the 10-cm telescopes used for this project, each exposure covers 6 degrees of sky). Compare this to the 10-meter Keck telescopes, whose imaging systems have fields of view of about an arcminute (1/60th of a degree). For transit searches, you want to keep staring at a star until you get lucky with a planet passing in front of the star, then confirm that as the transit happens again and again. So your best bet for optimizing your transit search is to look at a lot of stars at once, meaning a big field of view, which you get most easily from small telescopes.

It's not impossible to get large fields of view with large telescopes, but it takes a lot more effort. Check out the plans for building the 8.4 meter LSST for details.