Slashdot Mirror
SKA Telescope To Provide a Billion PCs Worth of Processing
Sharky2009 writes "IBM is researching an exaflop machine with the processing power of about one billion PCs. The machine will be used to help process the Exabyte of data per day expected to flow off the Square Kilometre Array (SKA) telescope project. The company is also researching solid state storage technology called 'racetrack memory' which is much faster and denser than flash and may hold the secret to storing the data from the SKA. The story also says that the SKA is unlikely to use grid computing or a cloud-based approach to processing the telescope data due to challenge in transferring so much data (about one thousand million 1Gb memory sticks each day)."
Could we get that in LoC's? Also, could we stick to the standard "one million thousands" unit, please?
A slashdotter who didn't build his own computer is like a Jedi who didn't build his own lightsaber.
So is that the processing power of one billion IBM PC 5150s?
Slashdot: Playing Favorites Since 1997
Let's hear it for Reel Big Fish and the Pietasters! Is there a Reggae telescope?
Free Martian Whores!
SKA telescope? Madness!
Seriously, how is a PC a unit of processing ability? And one thousand million GB sticks is an Exabyte (hence the name). Perhaps you can just say 10^18 bytes. This is slashdot, not msnbc.
"We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
Hey, IBM, you dropped your telescope.
Someone should pick it up, pick it up, pick it up, pick it up.
In Capitalist America, bank robs you!
I wonder how many football fields could you cover with that.
My other signature is a car
From TFA:
âoeIn the last year or two IBM has built machines in the order of a petaflop and in the last couple of weeks IBM announced an ongoing partnership with the US Department of Energy to build a 20 petaflop machine by 2011-2012,â he said.
âoeWe will need machines which probably have hundreds of thousands of processor cores in them and we roughly know how we can go about engineering it,â he said. âoeIt wouldnâ(TM)t be cost- or technically-feasible to bolt together 50 20-petaflop machines⦠and the power consumed would be crazy. By the time we deliver the 20 petaflop machine we will be well on the way to an exaflop machine.â
But you still end up with hundreds of thousands of cores, and you still need an OS that can effectively use that many cores, and nothing we have comes close other than Grid Computing.
Data manipulation on this scale simply must be divided and parceled out to be handled effectively.
So while they might not call it Grid computing, it still will be under the skin.
Sig Battery depleted. Reverting to safe mode.
Are theses telescopes of the 2 tone variety?
Come on... the moment IBM makes a computer with a billion cores, both Microsoft and Linux will be salivating at the change to get -something- to run on them. I mean, what's a GB sized array just to keep track of the CPUs. Pure insanity. Any real geek would love to tackle that.
This is my sig.
Define "somewhat". Case in point, I've had a F/W SCSI drive in 24/7/365 operation on my home system for 10 years. Somewhat indeed.
It must have been something you assimilated. . . .
These are clearly 4th wave ska telescopes, so RBF and the Pietasters are obsolete technology.
...run Crysis at full resolution!?
First of all, no one would be using manual storage to transfer the data.
Just throw up some numbers that makes sense to us. Like 99,420.5393 gigabit/second.
Most large ISPs use OC-192 as the backbones of their infrastructure. You'd need more than 10,200 of those to handle that data load, and that's ignoring the overhead.
Or to put it into numbers that the RIAA can understand: 1.5707309 * 10^9 music CDs every single day.
At 15 pieces of music per CD and $80,000/song that's $1.88 * 10^15 dollars/day flowing through that network. That's 632 times larger than the US federal budget for 2008.
No wonder the music industry is in trouble!
Imagine a beowulf of those
omglol!
Astronomy comes up with pretty pictures. Other areas not so much so. So what would you rather have you tax money go to... Pretty pictures. Or columns of crunched numbers.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
Where are the obligatory "beowulfcluster" tags and jokes already?
Sheesh, the standards around here sure are slipping.
So if I did my math correctly they're saying if they did distributed computing they'd need to transfer data at a rate of 92.5 Tbps.
I'm assuming a 1 Gb memory stick is actually 1 GB though...
Okay, if I do some rough math, just on the hard drives to dump that to .25eb/day of wiki
assuming 2tb drives, and ignoring the binary/decimal nonsense to be quick
assuming that the 1eb per day is correct and not the
assuming that 2tb costs $100 (volume discount, you know)
assuming no costs for things to hold these drives, and electricity, etc.
180 million drives. 18 billion dollars. Per year.
Let's assume by 2013 we've gone eightfold, to 16tb drives. Good, now we're at 2million ish drives and 2billionish dollars. Good
I realize they're planning for it all, but I just can't see how they're really going to store, let alone process, all that data. Whew.
I mean, they'd max out a btrfs/zfs system in 16 daysish at 1eb per. Perhaps this is just simply too much data...
stored on computers from birth to the grave
I have mod points, right now, but there isn't a mod for "-1, moron".
Sit, Ubuntu, sit. Good dog.
The telescope will send data via a dual-frequency acoustic transmission system.
A "two-tone" system, if you will.
Or...
...a PC with a couple ATI X2 graphics cards in a CrossFire setup.
But does it run Crysis?
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
My calculator (Qalculate!) tell me, that
((1 exabyte) day) ((1 exaflop) second) to byte day
= 11.574074... (0.1^flop) TB
That does not make any sense to me. Can someone elaborate? ^^
Any sufficiently advanced intelligence is indistinguishable from stupidity.
Doesn't matter anymore since you posted...
But seriously. I hate to guess what they are referring to when the OP refers to "more pressing problems" when dealing with computing power? I mean... computers will soon be our masters. We may as well make them the best damn masters we possibly can, right? If you were created by a bunch of self righteous peons who didn't give you the best of the best, you'd be pissed.
Every time I start to have faith in humanity, I ruin it by driving to work between 7 and 8 am.
They're gonna need a LOT of pigeons.
I lost a pile of dough because of that. Turns out that on a previous race something spooked the horse really badly. Now he never runs well on that track. Of course, the crooked bookie never told me about it.
When our name is on the back of your car, we're behind you all the way!
From the FAQ:
How far can this telescope see?
A: ONE STEP BEYOND!
nonsig. unsig. desig.
So if I wanted to crunch some of this data on my old Apple //e how many of the 143k 5.25" floppy disks would I need? When I try the math in Visicalc all I get is #err
So is that the processing power of one billion IBM PC 5150s?
IBM hasn't made PCs since 2005, when it sold its personal computing division to Lenovo. So it would be 1e9 of whatever computer Lenovo is selling.
Original article also compares a Peta of floating ops per *second* to an Exa of byte "processing and storing" per *day.* Journalism profs should save that article for class discussion.
If a system uses one petaflops to process one exabyte per day, that's the same as saying it takes roughly 86,400 floating ops to process each 1000 bytes of data. That sounds not unreasonable.
Maybe people don't get my post: I mean no harm to astronomy. I fully support scientific research and I realize its importance despite little immediate real-world benefit. My complaint is OTHER people aren't beating astronomy to the punch on pushing supercomputing and data storage. Space on supercomputers is in high demand and many of those projects deal with more immediate problems like crops and medicine- what I'm saying is I think the people supporting THOSE projects should be setting the records here. Astronomy doesn't get as much attention, so I'm assuming it doesn't get as much funding, which implies these other areas should have more money to spend on this.
Again, no complaints to the astronomers, they do brilliant work. I realize astronomers are the most realistic option for actually doing this kind of project- this is more of a "if only" than a real expectation.
My webcomic
ska: punk for band geeks.
THL phish sticks
If you mean that in defense of astronomy, you don't get my post. If you mean that because there aren't other problems that would use a computer like this, then I apologize for my ignorance. I know getting space on supercomputers is competitive and in high demand, and I know that one supercomputer is often used in parallel by a number of projects, but I don't know much else about supercomputing. I replied to my original post explaining more what I was getting at. If your comment has to do with my ignorance of supercomputers I don't think I would be the only one that could learn from a brief explanation of what makes it wrong to assume a computer used by astronomy could be used by other projects.
My webcomic
Use the tinfoil for something else, today. The military knows it's a thousand million times cheaper and easier to use actual intelligence instead of artificial intelligence. Instead of spending half a trillion on a computer in order to make it think, they'll give some guy a $25,000 scholarship to sit in front of it and do the decision making for it.
I am the richest astronaut ever to win the superbowl.
Since we are talking about ONE STEP BEYOND, and all that "Madness",
I propose "This is Ska" by longsy D as the theme song, has that cool
futuristic acid vibe to it.....
Youtube link
music lover since 1969
Some people might be interested in knowing where all this data comes from. There's a rule of thumb in astronomy that the angular resolution of your images is the wavelength of the radiation you're receive divided by the diameter of your telescope. Radio wavelengths are pretty long (up to tens of meters), so you need really big telescopes, which you get by scattering lots of little telescopes all over the place and then looking at the how the phase of the incoming radiation shifts based on location. So what you do is you sample the voltage of each antenna with 1 or 2 bits of resolution at the Nyquist frequency. So for 100Mhz radio waves you sample at 200MHz. That's 50MB/s for a single antenna. The SKA will likely have tens of thousands of little antennas scattered all over the place. So say 50MB/s times 20,000 antennas = 1TB/s = 100 petabytes/day, which is about what the summary says.
Now, it's not quite as bad as it looks. You don't have to pipe all this data to a central point to analyze it. You can take a small group of antennas and just look at the correlations between those, combine the data from that group and send the combined data to a second level of correlators, which takes data from a set of small groups, and so on, in a hierarchical fashion. You lose some information this way, but you get most of it, and the only wa to get all the information out of the data would be to bring it all to a central processing location so that data from all antennas could be compared to that of all other antennas, which is O(N^2) in the number of antennas and obviously infeasible for a telescope like the SKA. Even as it is, the system of hierarchical data collection is really pushing Moore's law, as the article shows.
the SKA is unlikely to use ... a cloud-based approach
Well, duh. You can't see anything when it's cloudy.
Have you read my blog lately?
OK, a new unit of measure. Most people that have some idea of what the SKA is could handle a unit like a TB instead of a GB. I suppose they could have used the old 'sheets of paper', or 'Libraries of Books' analogy, so we are lucky in that respect. as for the Billion PCs, are they Xeons? or P-IVs, dual or quad core?
There was an unknown error in the submission.
It will run better with Windows Vista! and Office too! SQL Server will definitely scale to handle the workload. Some custom .NET programming may be required, but Visual Basic will make it easy. So easy that anyone can do it. No problems.
ska is for skanking
tailgate down, or tailgate up, and...roof cargo rack, or not?
Fixed that for you.
I'm curious, DARPA, IBM and Lucent were working on proprietary holographic storage medium solutions. HDSS exerpt (from http://en.wikipedia.org/wiki/HDSS):
"During CES 2006, a workable holographic drive was tested and stored 300 GB of memory compared to blu-ray's 100 GB. It has been announced that hologram disks will be a post-blu-ray storage device."
I'm just curious since IBM was one of the groups working on this technology if there have been any advances that apply to the main article.
From the IBM website http://domino.research.ibm.com/comm/pr.nsf/pages/rsc.holo.html:
"Unique to holographic data storage is the ability to perform essentially immediate data searches through huge digital libraries by simply illuminating the media with all of the stored information (a holograph) with a pattern of the requested information."
Why aren't we hearing more about this?
Except for the part where ska predates punk by more than a decade.
punk: ska for non-band geeks.
Oh, come on. It's not a BS unit. It's not like we're trying to scientifically measure them, just convey some idea what these big numbers might mean in a more "real world" sense. And, in that sense, the cliche of "LOC's" actually carries some legitimacy, in that books contain data that can be easily compared to bytes. When somebody asks me what a "byte" is, I usually start by explaining that 1 byte is a keystroke. While not strictly true (Unicode can take up to 4 bytes per character) it's still a useful comparison, and gives an idea of the vast amounts of information we now deal with.
Seriously, who remembers the day when a GB was an unimaginably large amount of disk space? I remember dreaming over a series of "Ultimate PC" articles in PC/Computing where the author goes "all out" to build the biggest, baddest PC around - and it had a full GIGABYTE of disk space for its Pentium/66 motherboard....
Now moving on: what privacy implications would this much processing power have? How long would it take for a billion PCs (equivalent) collectively computing to crack a 1024 bit key? 2048 bits? 4096 bits? Can an RSA key easily computed on a P3 in a few seconds still hold the NSA at bay, when they could conceivably have a the equivalent of a BILLION cores under a single point of control?
I remember when a 128 bit key was strong, now 1024 bits are commonplace, and 4096 are used for more extreme cases. Since a $50 used PC can compute a new 4096 bit key in a few seconds, perhaps we should be increasing key length more?
I have no problem with your religion until you decide it's reason to deprive others of the truth.
IBM already has 2 operating systems in hand which they have massive experience, AIX and System/Z. They can both run as Hypervisors to Linux so they may use Linux too of course but the real unsurprising thing would be use of ZFS as filesystem for such data. It is already named after zetabyte.
They also use Plan 9 on Bluegene/L and certainly they aren't doing OS demos on multi million dollar supercomputers, so Plan 9 must be good on some purpose of Bluegene.
Microsoft? Can they scale really? I mean really, not some "demo show off" things which they abused name of Cray making Mr. Cray roll in his grave.
Here is what to do with international standards/systems when in doubt. If British didn't come up with their own weirdo standard, it is generally universal.
Back in day before DVB-* became standard, we had a map of TV broadcasting systems (analogue) on a big World map, you could see there is a single horrible variation (in compatibility sense) of PAL, PAL's British variant which has same video spec but some really needless variation of sound spec. So, if you don't know about it and assume British PAL is the PAL which World uses, you end up watching 5 trucks carrying thousands of VHS tapes parking as tapes came back as defect :) Happened to friend.
What makes you think every government machine/installation must be documented, submitted to top500/slashdot.org so you have to fool World's most elite Scientists if you could?
If they want a AI research, LISP monster machine, they buy it and they (any Govt. in World) has no obligation to make it public. If you look at Cray Research financial reports, they don't tell who they sell these monsters, not every single one. Even a company can ask for secrecy, for example if Apple tells them "we want a exaflop computer for our next product design, don't announce it", they will say "$120M installation to unnamed customer".
Now for the real tinfoil hat... The level of such machines which are so secret to be announced. Here is just one example which has likely upgraded to something else and in display in museum:
http://en.wikipedia.org/wiki/FROSTBURG
1993, 2TB of RAM. Enough said.
Seems to me that all this data is either (A) highly redundant, or (B) will contain large subsets of lots of existing data.
..
This data will be generated at a rate of 11.5 terabytes per second
Think of the data mining -> cherry picking possibilities!
Bible Code? HAH! Check out my SKA CODE, BITCH!
"His name was James Damore."
1 Exaflop matches up with some estimates of the net raw processing power of the human brain. Note I say raw and net, because that excludes likely optimization methods and weeding out parts of the mind that don't need to be simulated 1:1 (retina for example).
1 Exabyte far exceeds the memory capacity (by 1000x) to store a human brain... uncompressed.
They plan to have this thing running in 10 years.
$1000 computing equipment is hovering around 5-6 years behind Top500 http://www.top500.org/ computers (maybe less these days when you consider a single $100 graphics card bests 1997's 1 teraflop supercomputer).
6-7 years later (at a guess), when all the Top500 supercomputers reach at least the level of 1exaflop+1exabyte, their total storage capacity could store 1 billion human minds.
All I know now is my brain hurts.
After logging in slashdot still does not take you back to the page you were on. It's been that way for 20 years.
Ska and punk are two completely different genres. Punk is rock and roll, ska is kind of like fast raggae.
Free Martian Whores!
Actually, ska came decades before reggae, with rocksteady being an intermediate stage.
Also, the third wave of ska (late 80's through late 90's or possibly still continuing based on who you ask) did effectively meld ska with punk, and many bands that are considered ska bands are simply punk/rock bands with horns. Reel Big Fish, possibly one of the more popular constant ska bands of the last two decades, has released albums with no upbeats, a staple and possibly defining element of ska music.