Slashdot Mirror

← Back to Stories (view on slashdot.org)

SKA Telescope Set To Generate More Data Than Current Net

Posted by timothy on from the recycle-the-bits-responsibly dept.

angry tapir writes "The forthcoming $2.1 billion Square Kilometre Array (SKA) radio telescope could generate more data per day than the entire internet when it comes online in 2020, according to the director of the International Centre for Radio Astronomy Research (ICRAR), Professor Peter Quinn. SKA — which Australia with New Zealand and South Africa are competing to host, and which will help the search for Earth-like planets, alien life forms, dark matter and black holes — will be 10,000 times more powerful than any telescope currently used. Slashdot has previously discussed the proposal to use 'Skynet' — a grid-computing-based solution for processing and storage."

44 of 73 comments (clear)

  1. Uh oh. by ErikZ · · Score: 1

    Cue in the HAARP freaks in 3...2...1...

    --
    Democrats or Republicans. They are both taking us to the same place and they are not afraid of us anymore.
    1. Re:Uh oh. by CharlyFoxtrot · · Score: 1

      Cue confused rude boys in 3...2...1...
      Let's hope this satellite will pick it up, pick it up, pick it up ...

      --
      If all else fails, immortality can always be assured by spectacular error.
    2. Re:Uh oh. by davester666 · · Score: 1

      How can they afford this with Congress cutting the budget for everything except military spending and they aren't yet that delusional into planning for an interstellar invasion?

      This thing will blow through their ISP's data cap in the first couple minutes, after which they'll be on the hook for more than $1/Gb... Hell, it might even increment at a rate within 1 or 2 orders of magnitude of the national debt's rate of increase.

      --
      Sleep your way to a whiter smile...date a dentist!
    3. Re:Uh oh. by Xacid · · Score: 1

      Aww you beat me to it... ...but YES.

      Thank you, sir/madam.

  2. finally we can retire that... by MoFoQ · · Score: 1

    finally we can retire that saying/meme about Internet is for porn....or that the mass storage market is driven by porn

    unless...I guess we happen to be able to spot the alien equivalents via this SKA.....

    1. Re:finally we can retire that... by dwarfsoft · · Score: 4, Funny

      This is just Astronomical Porn. Rule 34 still holds true.

      --
      Cheers, Chris
    2. Re:finally we can retire that... by SnarfQuest · · Score: 2

      How about: Imagine a beowulf cluster of these...

      --
      Who would win this election: Andrew Weiner vs Andrew Weiner's weiner.
  3. Surprised it hasn't happened already. by blair1q · · Score: 1

    Really, how much "data" is "generated" by the internet every day?

    Sure, there's lots of traffic, but that's millions of copies of the same data.

    The new data going on to the internet probably isn't too heinous in quantity.

    And the summary blew the meme. It's not "generate more data per day than the internet", it's "generate more data per day than the earth does in a year, and conduct more internal networking traffic than the internet."

    1. Re:Surprised it hasn't happened already. by Gumbercules!! · · Score: 1

      Depends if you're logging that traffic. I would imagine if you logged all internet traffic, which by strict definition would be considered "new data", it would dwarf whatever data this SKA can produce. Now admittedly, that logged data would be "mostly useless and never read" - but I imagine most of the data the SKA generates will be "mostly useless and never read", too.

    2. Re:Surprised it hasn't happened already. by sFurbo · · Score: 1

      Sure, not that much new information is made by the net, but it sure is represented by a lot of data. Or, that is how I would use the terms, YMMV.

    3. Re:Surprised it hasn't happened already. by blair1q · · Score: 1

      if you start logging all traffic, you'll quickly run out of space to store it.

      if on the other hand you start logging only novel content, you'll take somewhat longer to run out of space to store it.

      just ask google how their cache works.

  4. SKA and other astronomy projects by JoshuaZ · · Score: 2

    Given that we just had a Slashdot article about how the space based James Webb telescope is already on the ropes with Congress, http://science.slashdot.org/story/11/07/07/0038247/Congress-Dumps-James-Webb-Space-Telescope, perhaps we should be worried that the same will happen for SKA. Unlike Webb, SKA is an international project, so it won't necessarily go down the tubes if the US backs out. Moreover, the US has backed out of European lead science projects before often with very little warning. SKA is going to allow some very interesting work. Among other things, SKA might be able to detect extraterrestrial life, either through direct radio signals (from intelligent life) which would be a really big deal, or more indirectly detect non-intelligent through the analysis of extrasolar planets' atmospheres (such as the detection of large amounts of oxygen). SKA will also be used for many other astronomy and astrophysics projects, such as examination of supernovas. SKA is very good science, let's hope that the penny pinchers who repeatedly cut tiny science programs while leaving defense, social security and medicare alone will not touch it. In the long run, science helps everyone.

    1. Re:SKA and other astronomy projects by fusiongyro · · Score: 1

      The US is not even involved in the SKA.

    2. Re:SKA and other astronomy projects by st0nes · · Score: 1

      The SKA isn't all-or-nothing. It can start small with a few receivers (South Africa already have the MeerKAT array) and scale when funding is available. It doesn't really matter if there are delays--good science can still be done with half an SKA.

      --
      Tempora mutantur, nos et mutamur in illis
  5. 10 years from now... by voodoosteve · · Score: 1

    Sure but think about how much more data the Internet generates today than it did 10 years ago. I'm sure we'll be able to cope in 10 years. Also, I know the LHC produces a ton of data. Does anyone know how it compares?

  6. Until... by Ryanator2209 · · Score: 1

    This will true until someone post the data on the pirate bay.

  7. NBN by dakameleon · · Score: 3, Interesting

    (Warning: Australian content ahead!)

    I hope this lays down a water-tight case for the NBN going ahead - or the combination of the two being a catalyst for each other. If there's one thing this is good for demonstrating, it's that future data requirements will outgrow the current infrastructure very quickly, and a project which is as far-sighted as installing FTTH throughout the country has a justification for the unforeseen benefits it can help happen.

    (and bah humbug to anyone who thinks the SKA isn't justified to begin with!)

    --
    Man who leaps off cliff jumps to conclusion.
    1. Re:NBN by BeanThere · · Score: 1

      That's absurd on the face of it, you don't need FTTH for the SKA, because the data isn't going to be distributed on a large scale to homes! For the SKA, you only need one fat pipe leading from the SKA, out to the under-ocean cables.

    2. Re:NBN by Canberra+Bob · · Score: 1

      How many major infrastructure projects have been attempted at all over the past 40 years? That is the entire problem, as far as I see it, is we are falling behind because we are not building anything. Successive governments keep playing it safe and if there is a surplus the money does not get invested into major infrastructure projects, it gets "invested" into winning the next election through middle class welfare. The mining boom will eventually end and we will be left in the tough times with nothing to show from the good times.

  8. Ska by supernatendo · · Score: 1

    I did not know that the music created by the likes of "Reel Big Fish", "The Clash" or other Ska bands could be used to find intelligent life somewhere...

    1. Re:Ska by Hsien-Ko · · Score: 3, Funny

      Implying that there's depth in ska to use a telescope for that is. A reggae telescope would be better. Let's all get together and see deep space, mon.

    2. Re:Ska by Kyont · · Score: 1

      No, there is just too much data. They chose ska because they needed something exactly like reggae except much, much faster.

      --
      You shall see a cow on the roof of a cotton house.
  9. is this a good thing? by Charliemopps · · Score: 2

    If the CIA invented a device that listened into every phone call in the entire world, real time and dumped it all as a WAV file on a storage device in the basement, would that really do them any good at all?

    1. Re:is this a good thing? by Anonymous Coward · · Score: 1

      If the CIA invented a device that listened into every phone call in the entire world, real time and dumped it all as a WAV file on a storage device in the basement, would that really do them any good at all?

      Why would they invent that when they already have a device that listens into every phone call in the entire world, real time and dumps it all as a bunch of AMR files on a storage array in the basement?

  10. Seems like... by z3alot · · Score: 1

    ...calling your global computing initiative 'theskynet' might be detrimental to your acceptance by the technologically affluent who operate the computers you need help from.

  11. Nonstandard unit fun! by z3alot · · Score: 1

    Finally a new unit for large amounts of data, appropriate for this new golden age of progress and hyperbole What I want to know is how many library of congresses are in an 'internet-year'...

  12. Sensationalistically inaccurate article... by Dahamma · · Score: 3, Informative

    The project is expected to deliver up to an exabyte a day of raw data, compressed to some 10 petabytes of data in images for storage.

    So, 10 petabytes of data - who cares about the raw source. I work for a video streaming company and we have several petabytes of H.264 video. If that were to be uncompressed into 30 FPS 1080p raw data, it would be 50-100x that, so already approaching a couple hundred petabytes. And think of all of the JPEGs out there - why don't we just uncompress all of those for the comparison as well?

    A (likely conservative) back of the hand calculation by Google estimated at least 5 exabytes accessible on the Internet (so even the wrong estimate is wrong). I'd imagine a huge percentage of that is compressed video, audio, and images. So, basically 5 exabytes vs 10 petabytes - it's off by 3 orders of magnitude.

    1. Re:Sensationalistically inaccurate article... by Anonymous Coward · · Score: 2, Informative

      Indeed ... while it's an impressive number, we already have experiments that generate more "raw data" per day than that: "CERN experiments generating one petabyte of data every second" http://www.v3.co.uk/v3-uk/news/2081263/cern-experiments-generating-petabyte That's 84EB per day. But "all" of it is crap, and they eventually store only about 25PB per year.

    2. Re:Sensationalistically inaccurate article... by fusiongyro · · Score: 1

      It's a specious analogy. Video and audio can be compressed with loss, and the algorithms make heavy use of human perceptual limitations. Scientific data produced by large instruments need to have breadth and depth; the instrument is a scarce resource and there are unlimited ways of reducing radio astronomy datasets to produce different data and different insights. Especially with radio, you're going to be collecting a ton of white noise-looking data, but you can't use a lossy compression algorithm to trim it, so you're stuck doing lossless compression on essentially random data, which doesn't work so well.

      At the same time, it's an empty boast. I'm a programmer on the EVLA project. WIDAR would happily produce hundreds of gigabytes per second if we let it, but the rest of our pipeline is completely unprepared for it, and the astronomers don't have a cluster large enough to reduce datasets that big anyway. Some of their analyses aren't even parallelizable. So in practice, you want an instrument that can gather ridiculous amounts of data for PR and future-proofing purposes, but your scientists aren't likely to use it that way. Certainly not in the year it goes live.

    3. Re:Sensationalistically inaccurate article... by discord5 · · Score: 1

      So, 10 petabytes of data - who cares about the raw source.

      Depends on what you're doing. If you're taking pictures of the Eiffel Tower on your vacation you're not really interested in if the CCD on your camera's LSB was a 1 or a 0. If you're taking images from a hyperspectral sensor for scientific purposes the better the accuracy the better (hopefully) your results. It depends on the type of application and how important accuracy is to you (and how accurate your sensor is).

      I work for a video streaming company and we have several petabytes of H.264 video.

      We've got several hundreds of terabytes in lossless compressed high resolution hyper spectral images. If you start using lossy compression you often lose too much information and gain artifacts that can (sometimes drastically) change the outcome of whatever algorithms you wish to run on those images. However, in some cases you can find a good compromise in compression factor, it all depends on what you're planning to measure or calculate. If you're doing scientific research you'll always want to determine what the effects are of lossy compression on your dataset before you start throwing away the raw data.

      In your case where you are streaming video though, lossy compression is very applicable. The human eye (and/or brain) is quite limited in what it can perceive visually.

    4. Re:Sensationalistically inaccurate article... by bertok · · Score: 2

      Your maths is quite a bit off.

      We have only 3 primary color channels (4 if you count rods separately, 5 if also counting tetrachromats), not "65000". We can't see 1 million different intensities simultaneously either -- while the human eye does have an enormous dynamic range, this adaptation takes a while (minutes). At any one time, we can see maybe 300-1000 distinct intensity levels per color channel. This only requires 10 bits to represent per channel. Even your 125 Mpixels is an exaggeration, because we have roughly 125 million sensor cells total, not 125 per color channel!

      This then works out to 125*10^6 cells * 10 bits * 30 Hz = 38 Gbps, which is a lot, but is almost 7 orders of magnitude less than your estimate!

      In practice, even 38 Gbps is overestimating things substantially -- the cells in the retina have a trade-off between temporal resolution ('framerate') and dynamic range, so we can't simultaneously get 10 bits of intensity detail and 30 Hz of temporal resolution. Additionally, the image on the retina isn't focused very well, reducing the actual image detail quite a bit, especially near the edge of our field of view.

      Whatever the real raw bandwidth is, the optic nerve transmits only about 8.75 Mbps per eye!

    5. Re:Sensationalistically inaccurate article... by dargaud · · Score: 1

      This kind of back of the hand calculations and the fact that hard drive capacity hasn't increased much in the last few years (from 2Tb 3 years ago to only 3Tb now) makes me wonder if there aren't much larger prototype HDs in use in large datacenters à la Google.

      --
      Non-Linux Penguins ?
    6. Re:Sensationalistically inaccurate article... by Dahamma · · Score: 1

      It's a specious analogy.

      If you RTFA, no it's not.

      The project is expected to deliver up to an exabyte a day of raw data, compressed to some 10 petabytes of data in images for storage.

      They clearly say it's compressed before it's stored. The raw data number itself was pure boast. Besides, no one said it was lossy compression, anyway. Lossless image compression can be very efficient depending on the image.

    7. Re:Sensationalistically inaccurate article... by Dahamma · · Score: 1

      So, 10 petabytes of data - who cares about the raw source.

      Depends on what you're doing. If you're taking pictures of the Eiffel Tower on your vacation you're not really interested in if the CCD on your camera's LSB was a 1 or a 0. If you're taking images from a hyperspectral sensor for scientific purposes the better the accuracy the better (hopefully) your results. It depends on the type of application and how important accuracy is to you (and how accurate your sensor is).

      I already pointed this out above, but it you RTFA they clearly say it would be compressed before it's stored. If they don't store the raw data, then, yes, who cares about it because it doesn't exist for analysis. And as I also pointed out, nowhere did it say it was lossy, just that the ~1EB of data compressed to 10PB.

    8. Re:Sensationalistically inaccurate article... by fusiongyro · · Score: 1

      It's a distortion on the part of the article. Radio astronomy raw data are not images. And the computational effort to reduce the raw data into data from which you could make images is large enough that you tend to store the result of the processing alongside the original data--it takes up more space, not less.

      It still doesn't matter, because there's no way they'll be running the telescope at full throttle until several years after commissioning.

  13. Almost! by HalfNormalForm · · Score: 1

    Calling the Clash a ska band is like calling the Beatles a ska band.

  14. My Home DVR by jageryager · · Score: 1

    In nine years my home DVR system will generate more data than the entire internet!

    --
    "They that give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety"-B.Franklin
  15. Early processing of raw data by BCMcI · · Score: 1

    This begs for research on how the neural system in the eye compresses raw data into information that can be transmitted through the limited bandwidth optic nerve. Collecting data is great but we can drown in data. We need information processing near to the source.

    1. Re:Early processing of raw data by ceoyoyo · · Score: 2

      We know a decent amount about it. And we don't want to replicate it in our scientific instruments. The brain does all sorts of extrapolating, interpolating and other forms of making shit up. Which is great if you're a mammal who needs to see the sabre toothed tiger stalking you, but not so good if you're trying to get accurate, quantitative data out of a scientific instrument.

    2. Re:Early processing of raw data by Anonymous Coward · · Score: 1

      Also, computationally intensive to do algorithmically, free when it happens due to quantum interference or neurotransmitter binding or whatever in the optic nerve.

  16. A grid-computing-based solution for processing and by oblivionboy · · Score: 1

    storage called Skynet... ...I'm sure this won't turn out bady.

  17. Perl one-liner by tonique · · Score: 1

    I wonder if the system will run on a Perl one-liner.

  18. Mostly noise by pentadecagon · · Score: 1

    This merely says they are unable to filter out all the noise. Producing huge numbers of bits isn't a particularly remarkable achievement. I guess there is far less than 1% actual information in that bit stream.

  19. The first image recorded from the SKA by jockeys · · Score: 1

    was a checkerboard pattern in black and white. When reached for comment, the leading scientist on the project had this to say,
    "Pick it up, pick it up, GO!"

    --

    In Soviet Russia jokes are formulaic and decidedly non-humorous.