Slashdot Mirror
SKA Telescope To Provide a Billion PCs Worth of Processing
Sharky2009 writes "IBM is researching an exaflop machine with the processing power of about one billion PCs. The machine will be used to help process the Exabyte of data per day expected to flow off the Square Kilometre Array (SKA) telescope project. The company is also researching solid state storage technology called 'racetrack memory' which is much faster and denser than flash and may hold the secret to storing the data from the SKA. The story also says that the SKA is unlikely to use grid computing or a cloud-based approach to processing the telescope data due to challenge in transferring so much data (about one thousand million 1Gb memory sticks each day)."
Could we get that in LoC's? Also, could we stick to the standard "one million thousands" unit, please?
A slashdotter who didn't build his own computer is like a Jedi who didn't build his own lightsaber.
Can it run Crysis?
So is that the processing power of one billion IBM PC 5150s?
Slashdot: Playing Favorites Since 1997
Fucking rude boys.
Let's hear it for Reel Big Fish and the Pietasters! Is there a Reggae telescope?
Free Martian Whores!
SKA telescope? Madness!
Seriously, how is a PC a unit of processing ability? And one thousand million GB sticks is an Exabyte (hence the name). Perhaps you can just say 10^18 bytes. This is slashdot, not msnbc.
"We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
Powered by Rude boys. They're dropping those packets so you can pickit up, pickit up, pickit up.
This is my sig. There are many like it but this one is mine.
I wonder how many football fields could you cover with that.
My other signature is a car
From TFA:
âoeIn the last year or two IBM has built machines in the order of a petaflop and in the last couple of weeks IBM announced an ongoing partnership with the US Department of Energy to build a 20 petaflop machine by 2011-2012,â he said.
âoeWe will need machines which probably have hundreds of thousands of processor cores in them and we roughly know how we can go about engineering it,â he said. âoeIt wouldnâ(TM)t be cost- or technically-feasible to bolt together 50 20-petaflop machines⦠and the power consumed would be crazy. By the time we deliver the 20 petaflop machine we will be well on the way to an exaflop machine.â
But you still end up with hundreds of thousands of cores, and you still need an OS that can effectively use that many cores, and nothing we have comes close other than Grid Computing.
Data manipulation on this scale simply must be divided and parceled out to be handled effectively.
So while they might not call it Grid computing, it still will be under the skin.
Sig Battery depleted. Reverting to safe mode.
Are theses telescopes of the 2 tone variety?
Come on... the moment IBM makes a computer with a billion cores, both Microsoft and Linux will be salivating at the change to get -something- to run on them. I mean, what's a GB sized array just to keep track of the CPUs. Pure insanity. Any real geek would love to tackle that.
This is my sig.
It's Madness I tell ye!
I'm glad that astronomy is helping to push the limits on computing, but you would hope there are more pressing problems to use record-breaking computer systems for. If astronomers and their sponsors are willing to pick up the tab, I wish them well, but I feel a lot of people that could be the ones paying for this computer research have their priorities in the wrong order.
My webcomic
Define "somewhat". Case in point, I've had a F/W SCSI drive in 24/7/365 operation on my home system for 10 years. Somewhat indeed.
It must have been something you assimilated. . . .
...run Crysis at full resolution!?
First of all, no one would be using manual storage to transfer the data.
Just throw up some numbers that makes sense to us. Like 99,420.5393 gigabit/second.
Most large ISPs use OC-192 as the backbones of their infrastructure. You'd need more than 10,200 of those to handle that data load, and that's ignoring the overhead.
Or to put it into numbers that the RIAA can understand: 1.5707309 * 10^9 music CDs every single day.
At 15 pieces of music per CD and $80,000/song that's $1.88 * 10^15 dollars/day flowing through that network. That's 632 times larger than the US federal budget for 2008.
No wonder the music industry is in trouble!
Imagine a beowulf of those
omglol!
Where are the obligatory "beowulfcluster" tags and jokes already?
Sheesh, the standards around here sure are slipping.
So if I did my math correctly they're saying if they did distributed computing they'd need to transfer data at a rate of 92.5 Tbps.
I'm assuming a 1 Gb memory stick is actually 1 GB though...
Okay, if I do some rough math, just on the hard drives to dump that to .25eb/day of wiki
assuming 2tb drives, and ignoring the binary/decimal nonsense to be quick
assuming that the 1eb per day is correct and not the
assuming that 2tb costs $100 (volume discount, you know)
assuming no costs for things to hold these drives, and electricity, etc.
180 million drives. 18 billion dollars. Per year.
Let's assume by 2013 we've gone eightfold, to 16tb drives. Good, now we're at 2million ish drives and 2billionish dollars. Good
I realize they're planning for it all, but I just can't see how they're really going to store, let alone process, all that data. Whew.
I mean, they'd max out a btrfs/zfs system in 16 daysish at 1eb per. Perhaps this is just simply too much data...
stored on computers from birth to the grave
Can we have the space in terms of numbers of mp3 music files ?
Or...
...a PC with a couple ATI X2 graphics cards in a CrossFire setup.
But does it run Crysis?
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
For those of us that can't afford a 1 Exabyte or 1,000 Pedabyte drives here is the alternative breakdown:
- Standard 4.7GB DVD Disks: 212,765,957.45
- Standard 750MB CD-ROM Disks: 1,333,333,333.33
- Standard 1.44 Floppy Disks: 694,444,444,444.44
My calculator (Qalculate!) tell me, that
((1 exabyte) day) ((1 exaflop) second) to byte day
= 11.574074... (0.1^flop) TB
That does not make any sense to me. Can someone elaborate? ^^
Any sufficiently advanced intelligence is indistinguishable from stupidity.
They're gonna need a LOT of pigeons.
I lost a pile of dough because of that. Turns out that on a previous race something spooked the horse really badly. Now he never runs well on that track. Of course, the crooked bookie never told me about it.
When our name is on the back of your car, we're behind you all the way!
From the FAQ:
How far can this telescope see?
A: ONE STEP BEYOND!
nonsig. unsig. desig.
So, are we talking about a thousand million PCs or a million million PCs?
So if I wanted to crunch some of this data on my old Apple //e how many of the 143k 5.25" floppy disks would I need? When I try the math in Visicalc all I get is #err
So is that the processing power of one billion IBM PC 5150s?
IBM hasn't made PCs since 2005, when it sold its personal computing division to Lenovo. So it would be 1e9 of whatever computer Lenovo is selling.
Original article also compares a Peta of floating ops per *second* to an Exa of byte "processing and storing" per *day.* Journalism profs should save that article for class discussion.
If a system uses one petaflops to process one exabyte per day, that's the same as saying it takes roughly 86,400 floating ops to process each 1000 bytes of data. That sounds not unreasonable.
I call BS. Ok, maybe not BS, but I call paranoia. An EXAflop machine... for telescope data? Yeah, that's awesome, and I support it completely - but really someone is paying the bills for this type of major research for a telescope? Really?
Yeah, it's definitely possible, but my paranoid side thinks the gov't is working toward something more - like an AI capable machine... Crazy? Yep. But still within the realm of possibility, IMHO.
Since we are talking about ONE STEP BEYOND, and all that "Madness",
I propose "This is Ska" by longsy D as the theme song, has that cool
futuristic acid vibe to it.....
Youtube link
music lover since 1969
Gb = Gigabit?
Or maybe Gibibit?
Gibibyte?
Some people might be interested in knowing where all this data comes from. There's a rule of thumb in astronomy that the angular resolution of your images is the wavelength of the radiation you're receive divided by the diameter of your telescope. Radio wavelengths are pretty long (up to tens of meters), so you need really big telescopes, which you get by scattering lots of little telescopes all over the place and then looking at the how the phase of the incoming radiation shifts based on location. So what you do is you sample the voltage of each antenna with 1 or 2 bits of resolution at the Nyquist frequency. So for 100Mhz radio waves you sample at 200MHz. That's 50MB/s for a single antenna. The SKA will likely have tens of thousands of little antennas scattered all over the place. So say 50MB/s times 20,000 antennas = 1TB/s = 100 petabytes/day, which is about what the summary says.
Now, it's not quite as bad as it looks. You don't have to pipe all this data to a central point to analyze it. You can take a small group of antennas and just look at the correlations between those, combine the data from that group and send the combined data to a second level of correlators, which takes data from a set of small groups, and so on, in a hierarchical fashion. You lose some information this way, but you get most of it, and the only wa to get all the information out of the data would be to bring it all to a central processing location so that data from all antennas could be compared to that of all other antennas, which is O(N^2) in the number of antennas and obviously infeasible for a telescope like the SKA. Even as it is, the system of hierarchical data collection is really pushing Moore's law, as the article shows.
the SKA is unlikely to use ... a cloud-based approach
Well, duh. You can't see anything when it's cloudy.
Have you read my blog lately?
OK, a new unit of measure. Most people that have some idea of what the SKA is could handle a unit like a TB instead of a GB. I suppose they could have used the old 'sheets of paper', or 'Libraries of Books' analogy, so we are lucky in that respect. as for the Billion PCs, are they Xeons? or P-IVs, dual or quad core?
There was an unknown error in the submission.
It will run better with Windows Vista! and Office too! SQL Server will definitely scale to handle the workload. Some custom .NET programming may be required, but Visual Basic will make it easy. So easy that anyone can do it. No problems.
ska is for skanking
tailgate down, or tailgate up, and...roof cargo rack, or not?
I'm curious, DARPA, IBM and Lucent were working on proprietary holographic storage medium solutions. HDSS exerpt (from http://en.wikipedia.org/wiki/HDSS):
"During CES 2006, a workable holographic drive was tested and stored 300 GB of memory compared to blu-ray's 100 GB. It has been announced that hologram disks will be a post-blu-ray storage device."
I'm just curious since IBM was one of the groups working on this technology if there have been any advances that apply to the main article.
From the IBM website http://domino.research.ibm.com/comm/pr.nsf/pages/rsc.holo.html:
"Unique to holographic data storage is the ability to perform essentially immediate data searches through huge digital libraries by simply illuminating the media with all of the stored information (a holograph) with a pattern of the requested information."
Why aren't we hearing more about this?
so it can watch you really skank? does it do calculations on the upbeats of the clock cycles too?
Oh, come on. It's not a BS unit. It's not like we're trying to scientifically measure them, just convey some idea what these big numbers might mean in a more "real world" sense. And, in that sense, the cliche of "LOC's" actually carries some legitimacy, in that books contain data that can be easily compared to bytes. When somebody asks me what a "byte" is, I usually start by explaining that 1 byte is a keystroke. While not strictly true (Unicode can take up to 4 bytes per character) it's still a useful comparison, and gives an idea of the vast amounts of information we now deal with.
Seriously, who remembers the day when a GB was an unimaginably large amount of disk space? I remember dreaming over a series of "Ultimate PC" articles in PC/Computing where the author goes "all out" to build the biggest, baddest PC around - and it had a full GIGABYTE of disk space for its Pentium/66 motherboard....
Now moving on: what privacy implications would this much processing power have? How long would it take for a billion PCs (equivalent) collectively computing to crack a 1024 bit key? 2048 bits? 4096 bits? Can an RSA key easily computed on a P3 in a few seconds still hold the NSA at bay, when they could conceivably have a the equivalent of a BILLION cores under a single point of control?
I remember when a 128 bit key was strong, now 1024 bits are commonplace, and 4096 are used for more extreme cases. Since a $50 used PC can compute a new 4096 bit key in a few seconds, perhaps we should be increasing key length more?
I have no problem with your religion until you decide it's reason to deprive others of the truth.
.. yea. That sounds like a great cover story for the next gen NSA surveillance system. We believed Hughes Glomar Explorer was being built to collect manganese nodules too.
IBM already has 2 operating systems in hand which they have massive experience, AIX and System/Z. They can both run as Hypervisors to Linux so they may use Linux too of course but the real unsurprising thing would be use of ZFS as filesystem for such data. It is already named after zetabyte.
They also use Plan 9 on Bluegene/L and certainly they aren't doing OS demos on multi million dollar supercomputers, so Plan 9 must be good on some purpose of Bluegene.
Microsoft? Can they scale really? I mean really, not some "demo show off" things which they abused name of Cray making Mr. Cray roll in his grave.
Here is what to do with international standards/systems when in doubt. If British didn't come up with their own weirdo standard, it is generally universal.
Back in day before DVB-* became standard, we had a map of TV broadcasting systems (analogue) on a big World map, you could see there is a single horrible variation (in compatibility sense) of PAL, PAL's British variant which has same video spec but some really needless variation of sound spec. So, if you don't know about it and assume British PAL is the PAL which World uses, you end up watching 5 trucks carrying thousands of VHS tapes parking as tapes came back as defect :) Happened to friend.
What makes you think every government machine/installation must be documented, submitted to top500/slashdot.org so you have to fool World's most elite Scientists if you could?
If they want a AI research, LISP monster machine, they buy it and they (any Govt. in World) has no obligation to make it public. If you look at Cray Research financial reports, they don't tell who they sell these monsters, not every single one. Even a company can ask for secrecy, for example if Apple tells them "we want a exaflop computer for our next product design, don't announce it", they will say "$120M installation to unnamed customer".
Now for the real tinfoil hat... The level of such machines which are so secret to be announced. Here is just one example which has likely upgraded to something else and in display in museum:
http://en.wikipedia.org/wiki/FROSTBURG
1993, 2TB of RAM. Enough said.
Seems to me that all this data is either (A) highly redundant, or (B) will contain large subsets of lots of existing data.
..
This data will be generated at a rate of 11.5 terabytes per second
Think of the data mining -> cherry picking possibilities!
Bible Code? HAH! Check out my SKA CODE, BITCH!
"His name was James Damore."
1 Exaflop matches up with some estimates of the net raw processing power of the human brain. Note I say raw and net, because that excludes likely optimization methods and weeding out parts of the mind that don't need to be simulated 1:1 (retina for example).
1 Exabyte far exceeds the memory capacity (by 1000x) to store a human brain... uncompressed.
They plan to have this thing running in 10 years.
$1000 computing equipment is hovering around 5-6 years behind Top500 http://www.top500.org/ computers (maybe less these days when you consider a single $100 graphics card bests 1997's 1 teraflop supercomputer).
6-7 years later (at a guess), when all the Top500 supercomputers reach at least the level of 1exaflop+1exabyte, their total storage capacity could store 1 billion human minds.
All I know now is my brain hurts.
After logging in slashdot still does not take you back to the page you were on. It's been that way for 20 years.