Slashdot Mirror
IBM Building 120PB Cluster Out of 200,000 Hard Disks
MrSeb writes "Smashing all known records by some margin, IBM Research Almaden, California, has developed hardware and software technologies that will allow it to strap together 200,000 hard drives to create a single storage cluster of 120 petabytes — 120 million gigabytes. The data repository, which currently has no name, is being developed for an unnamed customer, but with a capacity of 120PB, it's most likely use will be a storage device for a governmental (or Facebook) supercomputer. With IBM's GPFS (General Parallel File System), over 30,000 files can be created per second — and with massive parallelism, and no doubt thanks to the 200,000 individual drives in the array, single files can be read or written at several terabytes per second."
A billionaire's porn collection?
Do they back up to tape or external USB drive?
...about the sound and torque generated when all these disks start to spin-up.
Somewhere I can store _all_ my porn in one spot.
it's most likely use will be a storage device for a governmental (or Facebook) supercomputer.
Actually, given the explosion of data storage needs in the bio-informatics area, it's most likely use would be in storing DNA sequences for research purposes.
Someone had to do it.
All I know is that if you put it on my computer, I'll have it filled in two years and have no idea what's actually on it.
Woot! Torrent all the things!
When that thing crashes somebody is going to be mad. I wonder how long restoring from backup is going to take.
...for hoarding whorecookies.
I can see the likes of the LHC or the AEA using something like this - they generate enough data. But if it were a "good guy" why would they keep it secret?
I am trolling
My understanding is that the LHC generates so much data, that most of it is discarded immediately without going to disk. Seems like this would be a good solution to there data problems.
It's not the government guys, at least not the cloak and dagger kind. They're too paranoid to let you know how much data they can store. They also don't want you to know that even with all that data, they're still only able to utilize a fraction of it. People are still going through WWII wire intercepts *today*. No, the problem in the intelligence community is making the data useful and organized as efficiently as possible, not collecting it.
That leaves only one real option: Scientific research. Look at how much data the Hadron Supercollider produces in a day. ..
#fuckbeta #iamslashdot #dicemustdie
Facebook and presumably a spy agency?
You're repeating yourself.
Have gnu, will travel.
FTFS:
It's the tech equivalent of Prince - it's "the data repository with no name." We can denote it with some sort of unicode glyph that slashdot will mangle.
And of course it has amazingly fast read speeds - if each drive has a 32 meg cache, that's 6.4 terabytes just for the cache.
BTW, it's for the ^@#%^&^+++NO CARRIER
Perhaps this cluster can load Deus Ex : Human Revolution levels in a reasonable amount of time!
Fear is the mind killer.
Run around with a shopping cart and swap out drives as they fail. Kind of like they did back in first computer days with vacuum tubes.
WWJD -- What Would Jimi Do?
(Smash amp, burn guitar, take home the groupies)
I'd go with the flux capacitor personally. Then you can go back in time, invest in IBM, Microsoft, Google, and Apple when shares are still cheap and buy the 120PB cluster. Assuming you drive a DeLorean anyway.
They say a little knowledge is a dangerous thing, but it's not one half so bad as a lot of ignorance. - Terry Pratchett
With 200,000 hard drives, won't there always be at least one hard drive that is failing? You'll need an IT guy 24/7 swapping out the failed drives. As soon as he swaps out one drive, another one will fail. It just seems kinda ridiculous.
And the men who hold high places must be the ones who start
To mold a new reality... closer to the heart
This just kinda strikes me as who would need this. Backing up the entire internet has to take up some space.
What do I know, I'm just an idiot, right?
It's for storing images from Nikon's new "Petapixel Pro" D7000000 camera
1) Download the internets
2) re-host the internets
3) ????
4) I really don't know. I'm scared.
120 million divided by 200,000 = 600. Even on an enterprise scale they could could get a lot better densities.
~
Anyone else find it depressing that the two top suspects for the use of this system are Facebook and presumably a spy agency?
Can humanity come up with no better use for the biggest iron than a bunch of frivolous, narcissistic ad profiling and covert spying on people living in an allegedly free country?
No wonder F@H doesn't post more progress. Our hardware is going towards people sharing their naked bong photos and government spooks cataloging your naked bong photos.
You are trying too hard looking for something to be upset about (in a very attention-whorish manner to boot.)
If they could make a 120PB cluster using floppy disks, I would be much more entertained by this.
What do you mean: can big disk arrays be build so that replacements can be automated? Of course they can be build, it would not even be that hard. Well, as long as you don't put drive/server production and delivery of the components or auto assembly in the automated system. I could not find one on google, I guess on such a large drive array, you can afford a human to replace some disks now and then. Humans are more flexible and more prone to see other problems occuring as well.
even in "small" disk arrays the replacements are automated with hot spares. of course you periodically replenish the hot spare pool, but one doesn't need to go running every time a disk fails
the government is too busy with its War on Terrabytes to worry about the petafiles
3x 1 GW PSUs?
Someone should manufacture industrial sized hard drives for this type of application. Like full height x2, so you could cram 30 platters in there.
Pulling and replacing drives from hot-swap slots in a drive shelf would only be a slight change from the long-available robotic tape silo systems; but I've never heard of a situation where rigging such a thing up made economic sense...
Your hot-spares provide immediate 'replacement', which allows you to make physical replacement less time-critical just by adding more drives to the system, and most big-huge-storage systems have front mounted indicator lights for drive health.
Having a human on duty who gets paged with an aisle and rack number, grabs a spare drive, and goes over and swaps it out for the one with the red light just isn't all that expensive on the scale of such a system... With the homogeneity of the drives, and the automated monitoring, swapping dead drives is easier than stocking grocery shelves(the latter isn't rocket-surgery; but there are substantial variations in size, shape, density, crushability, changes in location according to daily/weekly promotional campaigns, etc. Yes, I've done both.) The biggest personnel expense, in the case of an "unnamed customer" buying a bespoke system from IBM, will probably be finding somebody who has sufficiently high clearance to touch the system; but is still willing to be a drive monkey...
If this were for an American spy agency, maybe that would be enough. But when I think about how I have ten times this much data in my Gmail, and that Gmail isn't limited to only the US, I suspect that Google has a lot more storage space than this. Of course it's probably all very decentralized.
It would be 122PB. 2PB lost on bad marketing. Gimme my 1024 bytes back. But all-in-all this isn't that surprising. You can get 1PB in a 42U rack these days.
As a fun side note: You'll also need 122PB of tape storage (or 1.5 systems like this) just for backups. That's a lot of tape.
Custom electronics and digital signage for your business: www.evcircuits.com
Because it's a target regardless of who owns it. God could own it and call it the garden of Eden and people would still blow it up
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
If the mean time between failure of a hard drive is around two hundred thousand hours, and this disk garden has two hundred thousand drives won't the technicians be replacing a drive every hour or so? Don't believe the MTBF figures from the drive manufacturers. Those appear to be butt numbers. http://www.pcworld.com/article/129558/study_hard_drive_failure_rates_much_higher_than_makers_estimate.html ....
Two hundred thousand water cooled hard drives? How much does this fucking thing weigh? Allowing for half a pint of coolant in the pipes for each hard drive the figure comes out to over one hundred thousand pounds. That doesn't count the plumbing and coolant distribution system and heat exchanger. ....
Two hundred thousand drives with plumbing for water cooling will take up a healthy sized volume. The drives alone require on the order of four million cubic inches. I have to wonder if this is a proof of concept storage array for DARPA on behalf of an alphabet agency that needs a place to park all their spy photos and sigint.
Hard to image? Yes. But forty years ago, the largest computing center on earth had 57GB of disc storage.
I looked into making a hard drive silo as a business. Even dropped the business proposal by some vendors. You would put bare SATA or SAS drives in a load port and they would be dropped into place in groups for reading/writing. Critical data would have four HDDs writing at a time (three way mirror, plus one HDD that would go offsite.) Non critical would get 5-8 HDDs writing in a RAID 6 configuration. It would have been nice to have because disks can be erased faster than tapes for security (just do an ATA level secure erase when the data expires before writing new stuff).
However, I encountered a few problems:
1: 3.5" drives or 2.5" drives? A lot of enterprise arrays are running on the smaller drives. One could do both, but essentially it requires two silos due to the completely different shapes of the drives (requiring different grippers and such).
2: Engineering grippers for the drives. Enclosures would make the setup a lot more expensive, and there wouldn't be any standard for those. So, the drives need to be moved around bare. This is harder than you think, as a bare drive isn't engineered to have reliable gripping surfaces.
3: Delicate mechanisms. If a robot drops a tape, who cares. If there is no physical damage, it will work. A HDD that gets any significant shock is pretty much toast, or at best will be unreliable.
4: I could not find anyone interested in making a robotic mechanism for this. The only party that would do the job was Seimens.
4: Nobody was interested enough to fund this project.
I wished this would have worked out. A silo like this could be used as a disk array, swapping out bad disks automatically, a VTL, a replacement for a tape array, a place for cloning disks to send out to remote sites, all kinds of uses.
We know the capacity. We know the transfer rate. But how quickly do disks need to be moved in and out of the system in order to keep it running?
200,000 is a lot of disks. I assume they are all hot swap with a great deal of redundancy because I would expect multiple drive failures every day. A raid0 with that many disks might never boot.
Out of curiosity: I've seen a number of systems that use screwless drive rails that take advantage of the fact that the 3 screw holes on each side, and 4 on the bottom(for 3.5", 2.5" has something slightly different; but also fairly standard) by having pegs, just slightly smaller than the screws would be, that slide in to the holes. As long as modest pressure is applied to keep them in the holes(they aren't threaded or tight enough to be friction fit), the setup is pretty solid. Could a gripper mechanism employ either similar peg-insertion directly, or have an internal supply of rails with a good grip on one side, and pegs on the other(possibly even two internal supplies, one for 3.5"s and larger 'shim' ones for 2.5"s)?
Just curious if anyone has experience managing large, mechanical disk arrays, if you installed an array of such a size using identical hard drives and bringing everything online relatively at the same time, would there be an increased likelihood of ALL the drives dying at roughly the same time? Could failure statistics bite you with enough simultaneous failures to negate redundancy?
I'll give you credit for "this used to be true" back in the day when a computer was a 486 on a modem. It's absolutely not true any more.
Govt is Big Brother, and they Like it. And they absolutely have the resources to do it.
Why? Because all they need to do is a Red Flag system. Joe Average doesn't really produce that much data per day all by himself, and .gov isn't trying to perfectly reproduce the entire activity. They just need to know if something is getting juicy.
"Look! Here's a 12 Gig file of Joe's activity for the month! Control-F and search for the words "Music" and "Movie" and "Copy".
Lights out.
The part you are glossing over is how much help they are getting from nice Corps. ISPs, Telecoms, Facebook, and Google.
So to play the "nah, don't worry" line is completely misleading.
My first Journal Entry ever, in 8 years! http://slashdot.org/journal/365947/aphelion-scifi-fantasy-horror-poetry-webzine
Can you just imagine the brown up when they power up the drive farm?
In practice they would be doing sequential spin up. I do however, wonder how long that would take to sequentially spin up 200k drives.
ELOI, ELOI, LAMA SABACHTHANI!?
based on 120 million GB and 200k drives, the per-drive capacity works out to 600GB a piece. Sounded like they're stringing together a bunch of WD Velociraptors.
ELOI, ELOI, LAMA SABACHTHANI!?
We should all be glad it's not Apple that's building this then!
Cool post bro, highfive \o
What's it for? No surprise, domestic spying.
I suspect just spying generally, including gathering information from non-spying sources, and including non-domestic. Why on earth would it be limited to domestic spying?
-- IANAL, this isn't legal advice, and definitely isn't legal advice for you. Also, Squee!
I don't know what it is for but I know the name of the drive is "C:"
Some drink at the fountain of knowledge. Others just gargle.
AFAIK it needs much less to maintain once it's started. So you just have to power the disks up in a 'wave' and ensure your peak power demand is below what you can supply.
Keeping some (large) capacitor banks charged can help with this as well. Put the cap banks in series with the power supply (being safe about it of course) and the caps should be able to provide for any peak shortfalls.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
Congratulations, you've been nominated for this year's Most Useless Comment Award! We take great pleasure in awarding the MUCA, but in order to claim it you'll need to reply without your AC cloak.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
That was thought of, but grippers have to do tens of thousands of moves, and reliably hitting those holes each time, every time, was an issue. If there was a misalignment, then the drive might not make onto the gripper's tray correctly. The good thing is that with a robotic gripper that has a tray for the drive to slide on, the chances of it falling are less, but trying to get it to a place where it can be read might be an issue.
Of course, an enclosure would remedy this completely, but there are no real standards for drive enclosures, and it would dramatically increase the cost per drive, unless one could make and sell a large number, so economies of scale could kick in.
It's way too big for any governmental use, NASA, or weather-related supercomputing application. Must be for Facebook
Sorry, but gray text on gray background is making my eyes bleed.
ASCII Pr0n
A unique way to learn a language: http://languageloom.com
This isn't a genuine statistical analysis, but a back-of-the-napkin calculation suggests that if they use hard drives with an MTBF around 3 years, they'll be replacing one drive every 7.5 minutes. If your employee can run fast, that's a 24/7 fulltime job.
What do you mean they cut the power? How can they cut the power, man? They're animals!
Assuming that you could get 50 Libraries of Congress onto a single petabyte drive, you ought to be able to get 6,000 Libraries of Congress onto one of these 120 petabyte arrays,. . .
I know of at least three companies (Apple, Google, Microsoft) just off the top of my head rich enough and ballsy enough to try AI on a scale that's never been tried before. I'm crossing my fingers.
If video games influenced behavior the Pac Man generation would be eating pills and running away from their problems.
or start referring to hard drive and memory in "jiggabytes."
Actually, I would be okay with changing this. All in favor?
/* No Comment */
the "milligoogle"
Help stamp out iliturcy.
Assuming you drive a DeLorean anyway.
But here's the thing - you don't really need the DeLorean! All you need is a beater that can get up to 88mph! And I used to do that all the time in my old Pinto... well, until that time I didn't quite make it up to 88 in time. But don't worry, the burns are healing nicely and the point still stands!
That is all.
Jigga, whaAt?
How exactly do you backup 120PB?
Easy, you just buy one of these clusters... (for arbitrary values of "easy" and "just")
What made you think this was the primary storage?
alias sudo="echo make it yourself #" ; # https://pipedot.org/~stderr & http://soylentnews.org/~stderr
A company I used to work for were in the process of developing some new products when I started. They were very good with lasers, since an early product of theirs was a storage device that stored a terabyte of data on optical media. At the time (early 1990s) the market for storing such vast quantities of data was limited. Worse, the main customers were three-letter agencies who had security concerns about buying from a non-U.S. company, so they sold the product line to a U.S. company and went on to other things.
They proceeded to develop some good stuff, but also developed some real crap, and supported it all miserably. Along the way they fucked me over badly. They were subsequently bought out and asset-stripped. I'm still here. They're not. Serves the bastards right... :-)
...laura
Finally a place to store all my zeros!
Fascism should more properly be called corporatism because it is the merger of state and corporate power. -- Mussolini
No wonder F@H doesn't post more progress. Our hardware is going towards people sharing their naked bong photos and government spooks cataloging your naked bong photos.
I don't get it, does your bong look classier if it's all dressed up? ;p
"I love animals! Some are cute, others are tasty, what's not to like?" - Betsy Schroeder, Jeopardy contestant
MS Word will still take 20 seconds to launch.
sig has been sent away for a few small repairs...
Given the fail rate on hard drives, replacing these would be a full time job. No?
http://www.awfullybigmoustache.com
Actually, 120PiB is 135 PB.
I am not impressed, its tiny. Do you know how much capacity our bittorrent network has folks? Image a beowulf cluster of these... and you got it.