Slashdot Mirror

← Back to Stories (view on slashdot.org)

6.7 Meter Telescope To Capture 30 Terabytes Per Night

Posted by Soulskill on from the very-large-yet-very-small dept.

Lumenary7204 writes "The Register has a story about the Large Synoptic Survey Telescope, a project to build a 6.7 meter effective-diameter ground-based telescope that will be used to map some of the faintest objects in the night sky. Jeff Kantor, the LSST Project Data Manager, indicates that the telescope should be in operation by 2016, will generate around 30 terabytes of data per night, and will 'open a movie-like window on objects that change or move on rapid timescales: exploding supernovae, potentially hazardous near-Earth asteroids, and distant Kuiper Belt Objects.' The end result will be a 150 petabyte database containing one of the most detailed surveys of the universe ever undertaken by a ground-based telescope. The telescope's 8.4 meter mirror blank was recently unveiled at the University of Arizona's Mirror Lab in Tucson."

67 comments

  1. Harddrive space doubles every year... by Anonymous Coward · · Score: 0, Offtopic

    Currently we have 1.5 TB hard-drives. That means in 2016 we'll have 200 TB hard-drives.

    30 TB a night won't be all that much data by then.

    1. Re:Harddrive space doubles every year... by Anonymous Coward · · Score: 0

      There's a physical limitation to data density, and before that we'll have reliability issues. It's not magically doubling, we'll always have too much data to store. /me has 3% of HD free

      We'll reach a plateau for data density eventually. And I'm sure the telescopes by then will be pumping petabytes of video every hour, in part thanks to the porn industry.

    2. Re:Harddrive space doubles every year... by lysergic.acid · · Score: 2, Interesting

      we might be reaching the physical (or practical) limit of data density for hard disk platters, but we'll probably just move to new technologies. it's very unlikely that magnetic disk drives are the pinnacle of data storage technology. there are probably more efficient storage mediums in the works already.

      i don't know what can currently match magnetic disks drives in terms of IO speed, but holographic storage shows a lot of promise. in theory, holographic storage can read/write millions of bits of data in parallel rather than one bit at a time as with conventional optical or magnetic media. the theoretical limit of holographic storage density is tens of Tb (terabits) per cm^3. and already commercial industries have achieved 500 Gb per square-inch (about 5x the density achievable on magnetic hard disks).

      holograms also have some very interesting properties that may or may not transfer to digital data storage. for instance, if you record a hologram of a 3D object onto a photographic plate, you can in essence reproduce a 3D image of the whole object with any piece of that photographic plate. Wikipedia explains this phenomenon thusly:

      Since each point in the hologram contains light from the whole of the original scene, the whole scene can, in principle, be re-constructed from an arbitrarily small part of the hologram. To demonstrate this concept, the hologram can be broken into small pieces and the entire object can still be seen from each small piece. If one envisions the hologram as a "window" on the object, then each small piece of hologram is just a part of the window from which it can still be viewed, even if the rest of the window is blocked off.

      since holographic data storage also uses optical interference patterns to store information, i guess it's possible that this phenomenon would also transfer over, though it might not since we're talking about digital data in this case rather than analog data. with analog data, losing a part of the interference pattern simply reduces the resolution of the holographic image, though it remains whole. with digital data, that loss of resolution could simply corrupt the data. but i don't know, i'm not a holography expert.

    3. Re:Harddrive space doubles every year... by hairyfeet · · Score: 1

      Not to mention there is an even easier way,which is simply adding more slots and/or shrinking the sizes of drives. Hell my going on 5 year old machine has slots for 4 drives(2 EIDE and 2 SATA) and we are now seeing plenty of boards with 6 SATA slots or more. And as we have seen drives like chips can still be shrunk a ways. So I kinda doubt we'll be hitting the wall for awhile yet.

      That said,where we ARE hitting a wall,and pretty hard at that,is easy to use consumer backup storage. The days of easily backing up your data to DVD is long gone. I was hoping that BD would turn out to be a decent choice,but like nearly everything Sony has made in the past decade they were more concerned with DRM than selling their product. So I really do hope that someone comes out with affordable holodiscs or some other easy to use format that will hopefully then spread into the consumer sector quickly like DVD. Because right now DVD is really the only choice a consumer has when it comes to backups.

      --
      ACs don't waste your time replying, your posts are never seen by me.
  2. Why not... by isBandGeek() · · Score: 1

    ... launch it into space? We need to replace the Hubble.

    1. Re:Why not... by Nyeerrmm · · Score: 4, Interesting

      The basic problem is that a 6.4 meter aperture can't fit in a launch vehicle, Ares V is only to be 5.5 meters.

      Hubble was built at the diameter it was (2.4 meter) because thats about the maximum you could build of a stiff mirror that held its shape well enough through launch to remain optically sound on orbit. When you require 10-nm level precision, it takes a hefty structure to keep things that stiff.

      In order to go bigger, the methods they're using for James Webb manage to double the aperture while halving the weight. The way they do this is using active controls and sensors to correct errors rather than rely on avoiding all errors. But looking at James Webb, you'll notice it focuses on IR which is very hard to observe from Earth, while no optical band concept is out there. This is because the new, big Earth-bound scopes use adaptive optics to eliminate seeing errors (the variations in the atmosphere that Hubble avoids), and get potentially better images than Hubble since larger mirrors can be used.

      Of course, if the money shows up, there are other advantages to having a space-based observatory, particularly access time and not having to worry about the effectiveness of the adaptive elements, so I'm sure we'll see a proper Hubble replacement eventually, but it's certainly not as critical for scientific progress as some might think.

    2. Re:Why not... by Iamthecheese · · Score: 1

      aperture science: We do what we must becauce we can. For the good of all of us.

      --
      If video games influenced behavior the Pac Man generation would be eating pills and running away from their problems.
  3. A waste by Anonymous Coward · · Score: 0, Insightful

    Money would be better spent on upgrading hubble, or better yet replacing that floating piece of space-junk with a new telescope. Making the most advanced telescope, ON THE GROUND, seems like an oxymoron to me.

    1. Re:A waste by Vectronic · · Score: 1

      So consider it a beta... I imagine that the reason its still on the ground is a lack of funding to get it up there... its far cheaper to set it up on the ground, and extremely cheap to fix and maintain in comparison, plus when its on the ground you (they) have essential complete control over it, when its in space you have to let other people play with your toy, or be sacrificed to debris...

      I highly doubt this is the be-all, end-all of this telescope (or at least the work the people involved are/will do) or any other telescope... if it proves effective, it'l be up-scaled (or down-scaled and perfected more) and thrown up in space unless something better comes along in the meantime.

    2. Re:A waste by resignator · · Score: 2, Informative

      It would be far less cost effective to upgrade Hubble than to build the LSST. Shuttle launches cost a TON of money not to mention you are risking astronauts lives to try and upgrade something we CAN build better on Earth. Hubble has had optical problems in the past and the 2 flights to repair it cost about one billion dollars. Imagine what a complete rebuild would require. "Making the most advanced telescope, ON THE GROUND, seems like an oxymoron to me." And the reason it is considered such an advanced telescope: http://en.wikipedia.org/wiki/Adaptive_optics

      --
      "At first, we thought it was just another snake cult."
  4. This is why... by ShadowFalls · · Score: 5, Funny

    It not being in space might have something to do with the amount of data it would have to transmit and the speed limitations... Besides, you can't replace Hubble, its impossible to exactly replicate that many technical difficulties...

    1. Re:This is why... by itlurksbeneath · · Score: 1

      Yes, yes.. the HST has had issues, but all in all, it's been in almost continuous operation for over 18 years. I don't think I've had anything for that long that didn't have problems (including the first wife).

      --
      Have you ever considered piracy? You'd make a wonderful Dread Pirate Roberts.
  5. 30TB raw? by Jah-Wren+Ryel · · Score: 2, Interesting

    When I worked at the CFHT a few decades ago, they had a bunch of "data reduction" algorithms they ran on each night's run that reduced the amount of data they needed to store by at least a factor of 10.

    --
    When information is power, privacy is freedom.
    1. Re:30TB raw? by $RANDOMLUSER · · Score: 4, Funny

      What, like storing the year as two digits, that sort of thing?

      --
      No folly is more costly than the folly of intolerant idealism. - Winston Churchill
    2. Re:30TB raw? by ShadowFalls · · Score: 5, Funny

      No, removing any data that could prove the possibility of Extraterrestrial life :P

    3. Re:30TB raw? by Jah-Wren+Ryel · · Score: 1

      Honestly, I don't know what the data reduction algorithms did. If I had to guess, I would guess that they included things like a high-pass filter to remove readings that were "in the noise" and maybe some sort of compression for repeated values over large areas like "empty space." Just guessing though.

      --
      When information is power, privacy is freedom.
    4. Re:30TB raw? by Kjella · · Score: 1

      What, like storing the year as two digits, that sort of thing?

      I know you're trying to be to be a smartass, but BMP<->PNG for example? Both are lossless but one takes much, much more space than the other. I'm sure there's other and probably better ways of compressing that data that's really specific to the application, like say FLAC is to music. Though I would guess that's already applied...

      --
      Live today, because you never know what tomorrow brings
    5. Re:30TB raw? by mrsquid0 · · Score: 2, Informative

      Back when I was using CFHT there was no high-pass filtering done on the data. That would change the noise properties of the data, which could render the data useless for certain types of analysis. The big space savings were done using lossless data compression. Depending on the type of data one can reduce the disk space required by up to about 90%. A second space-saving technique was to combine calibration data, such as bias frames and flats. In many cases combined calibration data is just as good as the individual frames, and in some cases better. Roughly half of the data collected each night is calibration data, so this can result in a big saving in space. I have not used CFHT since the 1990s, so I have no idea how they deal with their data now.

      --
      Just because you are paranoid does not mean that no-one is out to get you.
    6. Re:30TB raw? by michaelwv · · Score: 1

      They still stored all of the raw data, they just provided the reduced data products as a convenience to the astronomers.

    7. Re:30TB raw? by michaelwv · · Score: 1

      It's actually a fascinating example that information theory and compression really are true. The calibration images such as bias images (a readout of the CCD with no effective exposure time) or dark images (a readout of the CCD with the shutter closed but with an exposure time like those of the actual sky observations) indeed contain little information and so compresses by factors of 4-5 with straightforward things like gzip. Regular images of the sky compress by approximately a factor of 2.

  6. 30TB can hold a lot of duplicate stories! by Anonymous Coward · · Score: 1, Informative

    What does this story add that the following LSST stories didn't?

    http://science.slashdot.org/article.pl?sid=07/01/10/0111227

    http://science.slashdot.org/article.pl?sid=08/04/22/0116259

    http://science.slashdot.org/article.pl?sid=08/09/02/2346240

    1. Re:30TB can hold a lot of duplicate stories! by Vectronic · · Score: 0, Redundant

      This one was posted most recently, its fresh and clean... for us to poop on.

  7. Any colour you like so long... by Harold+Halloway · · Score: 4, Funny

    30 Terabytes, consisting mainly of #000000.

    1. Re:Any colour you like so long... by NixieBunny · · Score: 3, Interesting

      I used to think that until I started to work for astronomers. They actually take photos of noise, and then add noisy images together ("stacking") until pictures of interesting faraway things emerge from the noise. That's why a nightly sky survey is so useful - you can add together a few months of images and see stuff that you would never have seen in a single image.

      --
      The determined Real Programmer can write Fortran programs in any language.
  8. it will run on MySQL by datacharmer · · Score: 5, Informative

    This project was presented at the MySQL Users Conference 2008 in a dedicated talk and a keynote.

    The storage will be organized in clusters based on MySQL databases.

    Astronomy, Petabytes, and MySQL

    The Science and Fiction of Petascale Analytics

  9. Sounds like a big number, but.. by bertok · · Score: 4, Informative

    30 TB per night sounds like a lot, but 1.5 TB drives are about AUD 350 each, retail. By 2016, I'd expect vendors to have released at least a 10 TB hard drive at that price point, and I wouldn't be surprised if we're using 30 to 50 TB drives.

    So it all boils down to about $1000 per night of operation, or about $350K per year. Not exactly expensive for a science project. A single mars mission costs about $300M, but this telescope would generate more discoveries. That's not even considering that storage costs would continue to drop over the lifetime of the telescope, so the eventual total cost may be less than $100K per year. That's the salary of just one person!

    1. Re:Sounds like a big number, but.. by dwater · · Score: 3, Informative

      The cost of the storage might be reasonable, but what about the performance aspect? 30TB per night sounds like a lot to store in one night...being generous and calling a night 12 hours - I'm probably wrong, but I make that 43200 seconds which is 694 MB/s. Without looking up any performance stats for hard drives, that sounds fairly easily attainable (too).

      --
      Max.
    2. Re:Sounds like a big number, but.. by Anonymous Coward · · Score: 0

      Check this article: http://www.tomshardware.com/reviews/hard-drive-future,2009.html

      Your estimate seems abit conservative.

  10. mainly... by snikulin · · Score: 1

    But sometimes it contains nice #ffffff pixels! But I agree, it's very compressible anyway. RLE compression? ;)

    1. Re:mainly... by XSpud · · Score: 1
    2. Re:mainly... by Anonymous Coward · · Score: 0

      Into the Phail-Pail with your post!

  11. Holographic Storage = No More Bad Sectors? by Lumenary7204 · · Score: 1

    Presuming that the phenomenen reported by Lysergic Acid holds true and does, in fact, apply to holographic data storage:

    > if you record a hologram of a 3D object onto a photographic
    > plate, you can in essence reproduce a 3D image of the whole
    > object with any piece of that photographic plate.

    Would this mean the end of "bad sectors" as we know it?

    It would seem to me that if a part of the holographic storage device degrades in some way, one could simply read the data from any number of different "windows" (as described in the Wikipedia article) and get the proper result.

    In fact, this could be used as a form of default error detection and correction: If, for example, the hologram is stored in a sphere, one could read the hologram from coordinates (x, y, z, lat1, lon1) and (x, y, z, lat2, lon2) -- where latitude and longitude combine with x, y, and z to create a vector from the surface of the sphere to coordinates x, y, and z within the sphere -- and see if one gets the same result. If one does not, the sphere is simply rotated so the target coordinate address is read from a different relative point on the surface. Since the surface of the sphere is large compared to the block of data stored at x, y, and z, there is bound to be a set of vectors which will result in identical blocks of data (unless the complete and total degredation of the sphere has taken place).

    1. Re:Holographic Storage = No More Bad Sectors? by mangu · · Score: 1

      Would this mean the end of "bad sectors" as we know it?
      It would seem to me that if a part of the holographic storage device degrades in some way, one could simply read the data from any number of different "windows" (as described in the Wikipedia article) and get the proper result.

      This can be done without holography. The field of science that studies this is information theory. By a proper encoding, one can put redundancy in any data set such that the original information can be recovered, no matter how much degradation there is.

    2. Re:Holographic Storage = No More Bad Sectors? by Lumenary7204 · · Score: 1

      Mangu wrote:

      : "By a proper encoding, one can put redundancy
      : in any data set such that the original information
      : can be recovered, no matter how much degradation
      : there is."

      Granted, but according to the related article on channel capacity, the Shannon-Hartley theorem indicates that the maximum rate of data transmission across a noisy channel degrades logarithmically as noise increases linearly (if I am reading the article correctly).

      Therefore, it would seem to me that the amount of data redundancy required to cope with a given level of noise (channel degredation) increases exponentially with respect to a decreasing signal-to-noise ratio.

      So, while redundant encoding schemes to ensure read integrity may exist to cope with seriously degraded storage media, they are likely to be quite expensive in terms of bits per unit of information.

  12. It's galaxies all the way down ... by Anonymous Coward · · Score: 0

    not blackness.

    If you look deep enough, between any two points there's likely to be another galaxy, and everything's embedded in the CMB background radiation anyway. So, a perfect, all-seeing telescope would never deliver a zero pixel. ;-)

    Of course, real telescopes are a different story. :-)

    1. Re:It's galaxies all the way down ... by Harold+Halloway · · Score: 1

      I'm no astronomer or physicist but isn't the fact that there is so much dark sky, even with powerful telescopes, suggestive of the presence of dark matter?

      How bright would the night sky be if there were no dark space visible to the naked eye?

    2. Re:It's galaxies all the way down ... by ChatHuant · · Score: 1

      I'm no astronomer or physicist but isn't the fact that there is so much dark sky, even with powerful telescopes, suggestive of the presence of dark matter?

      No, but the existence of dark sky *is* interesting.
      Look up Olbers' paradox. Any matter that respects the laws of thermodinamics (be it dust, interstellar gas or the newly defined "dark matter") should heat up to the temperature of its ambient medium and start radiating, so absorbtion can't be an explanation for the dark sky. There are however other theories that could explain it (see the Wikipedia article for some of them).

    3. Re:It's galaxies all the way down ... by michaelwv · · Score: 1

      No. Dark matter is not really related to the darkness of the night sky. Dark matter is so named because it doesn't interact with light one way or the other. It doesn't absorb light, it doesn't emit light, so it ends up being separate from the question. But the question of why is the night sky dark is sometimes known as Olbers' Paradox and has been used as one suggestion that the Universe is finite.

  13. What about the storage? by Anonymous Coward · · Score: 0

    Could someone please explain to me what these people are using for storage? I have an AMD machine at home and am trying to put together a RAID for my movies, cause my 320GB hard drives just won't do. The motherboard will take 6 hard drives, so I figure I'll plug in 6 1TB drives to get 5TB. That should keep me going for another few months with bittorrent.

    But these guys with the big scope, eh, they want 150,000TB !! What the hell?! What hard drives do they use? Gosh, how do their computers look? I mean if 1TB or 1.5TB hard drives really are the largest drives out there, then they would have to get like 150,000 hard drives! That's like, a lot of hard drives....... I mean, that's a lot of money and computers and RAID boards and wires and... what about the person who has to walk around plugging them all in?! Poor girl!

    So like, seriously, what are these people doing for storage?

    1. Re:What about the storage? by isorox · · Score: 1

      What hard drives do they use? Gosh, how do their computers look? I mean if 1TB or 1.5TB hard drives really are the largest drives out there, then they would have to get like 150,000 hard drives!

      Quantum do 112 drives in a 4U rack, which with controllers and raid, and assuming they've moved to 1.5TB drives since I last saw it (when it was 1TB) give you about a petabyte in a rack -- maid to reduce power consumption.

      Pretty cheap too, under a million per rack.

  14. But. . . by zach_the_lizard · · Score: 1

    ... How many furlongs per meter? How many fortnights per night? I can't understand these eeeevil foreign units.

    --
    SSC
  15. Nits and Grins by martyb · · Score: 3, Informative

    From TFA title: (emphasis added)

    6.7 Meter Telescope To Capture 30 Terabytes Per Night

    <nit>
    That's 6.7 Meter effective diameter Telescope. The primary mirror has a diameter of 8.4m but the tertiary mirror (5.2m diameter) sits right in the middle of the primary, so its area needs to be subtracted from the primary. The area of the primary is pi*(8.4/2)^2 which is 55.4m^2 and the area of the tertiary is pi*(5.2/2)^2 which is 21.2m^2; a single mirror of that area would have a diameter of about 6.7m.
    </nit>

    6.7 Meter Telescope To Capture 30 Terabytes Per Night

    <grin>
    Hey!! I thought information wanted to be free! And here they plan to go off and capture 30 TERAbytes? Each night? OMG!!!!11Eleventy!! Say it ain't so!!
    </grin>

    1. Re:Nits and Grins by Anonymous Coward · · Score: 0

      Telescope diameters are always given as effective diameter. Imagine the nit if the title claimed a 8.4 Meter.

    2. Re:Nits and Grins by ForestDemon · · Score: 1
      </nit>

      6.7 Meter Telescope To Capture 30 Terabytes Per Night

      <grin> Hey!! I thought information wanted to be free! And here they plan to go off and capture 30 TERAbytes? Each night? OMG!!!!11Eleventy!! Say it ain't so!! </grin>

      what's the big deal? that's only .00000000003 yottabyte a night. :-)

      --
      Happiness is a belt fed weapon...
  16. That's all? by davidwr · · Score: 1

    Jennifer Gates was right, 640TB ought to be enough for anybody.

    --
    Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
    1. Re:That's all? by tabrisnet · · Score: 1

      Loading this link in elinks produces the following:

      The requested fragment "#PersonalLife" doesn't exist.

  17. It *IS* a big number! by mangu · · Score: 1

    30 TB per night sounds like a lot, but 1.5 TB drives are about AUD 350 each, retail

    Funny, but the idea of buying and installing twenty top-of-the-line new disks each day sounds like really big numbers to me...

    Not to mention that they need backups. How many tapes do they have to buy? And data transfer, too. All those bytes are worthless if no one gets to see them, so they need at least 30 TB / day data link capacity.

  18. Why the lenses? There's a better way! by Thagg · · Score: 1

    This telescope is amazing. The three-mirror configuration gives sharp focus, over a very wide field...the only problem is that the focus is on a spherical surface.

    The LSST fixes this by having three relatively small (small compared to the mirrors) lenses to flatten the field, and they use a very large image sensor.

    I am curious if they considered using a non-flat image sensor. It would be hard, but with e-beam or UV-laser lithography, I would think that you would be able to build a big sensor on a curved surface, and eliminate the inevitable light loss, distortion, chromatic abberations, and other problems with lenses.

    This is something that could be added in the future, too, much as Hubble was modified after-the-fact. It just seems to this layman that it's too good an idea to pass up.

    --
    I love Mondays. On a Monday, anything is possible.
  19. Comcastic! by Anonymous Coward · · Score: 0

    They had better not use Comcast as their ISP, or else all the astronomers will have to settle with getting a teeny tiny part of their data every month.

  20. They can thank ... by PPH · · Score: 1

    ... the p0rn industry for doing the groundbreaking research needed to manage this quantity of data.

    --
    Have gnu, will travel.
  21. 30 TB and not a single seed by Anonymous Coward · · Score: 0

    seed plz

  22. Space got slashgarbaged by davidwr · · Score: 1

    Slashdot took out the space in "Personal life."

    --
    Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
  23. common with "large science" by peter303 · · Score: 1

    Large Hadron Collider, Syntoptic Telescope Survey, Seismic Data Acquisition, Genome Decoding all use as much data capacity that exits. That now measures in the terabytes-per-day rate. Video tapes now have that capacity.

  24. Re:Why the lenses? There's a better way! by michaelwv · · Score: 1

    The detector surface is indeed effectively curved. It's made up of a large number of CCDs which will each be tangent to the focal surface at their location.

  25. Re:this sight by Anonymous Coward · · Score: 0

    I can see that Hooked on Phonics, indeed, worked for you.

  26. Re:Frosty Piss by MobileTatsu-NJG · · Score: 0, Troll

    CmdrTaco paid me 5 cents to suck his 2 inch penis!

    Wow.. the economy's going to put yo mama outta bidness.

    --

    "I like to lick butts!" by MobileTatsu-NJG (#32700246) (Score:5, Informative)

  27. Re:Why the lenses? There's a better way! by Anonymous Coward · · Score: 0

    Waffers that are used to make the CCD are cut along a particular crystal plane. you can't get the plane to, well not be a plane. In other words, no it would be far far more difficult to make a curved sensor surface.

  28. Wow.... by Terrorwrist · · Score: 1

    How much porn can one person save onto 150 petabyte drive? thats so much pornabytes. jeezes