Slashdot Mirror

← Back to Stories (view on slashdot.org)

World's Five Biggest SANs

Posted by CowboyNeal on from the still-need-more-drivespace dept.

An anonymous reader writes "ByteandSwitch is searching the World's Biggest SANs, and has compiled a list of 5 candidate with networks supports 10+ Petabytes of active storage. Leading the list is JPMorgan Chase, which uses a mix of IBM and Sun equipment to deliver 14 Pbytes for 170k employees. Also on the list are the U.S. DoD, which uses 700 Fibre Channel switches, NASA, the San Diego Supercomputer Center (it's got 18 Pbytes of tape! storage), and Lawrence Livermore."

48 of 161 comments (clear)

  1. ... That we know about by Anonymous Coward · · Score: 5, Insightful

    What about Google, Amazon, Yahoo, Microsoft, etc.?

    1. Re:... That we know about by Anonymous Coward · · Score: 5, Funny

      And there wasn't even a single Japanese firm listed. You'd think they'd have the biggest SANs of all.

    2. Re:... That we know about by Chapter80 · · Score: 4, Funny

      I was thinking San Diego, San Francisco, San Antonio, San Jose, Santa Claus.

    3. Re:... That we know about by GoodOmens · · Score: 2, Interesting

      I know it's not a PB but here at the Census we have one 150TB array thats used for one project and not 170K employees.

      It is interesting when you get in storage of this size. I remember sitting in a meeting where we discussed a storage cabinet we were ordering. The RAW size of the cabinet was 150TB but formatted it would be 100TB ... 50TB is a lot of storage to "throw away" for redundancy / formatting! Considering at this price your paying about $10k+ a TB (With staff and infrastructure costs fractured in)!

    4. Re:... That we know about by WuphonsReach · · Score: 2, Informative

      $10k per Terabyte isn't all that bad; maybe about 30-50% higher. Server level storage goes for $4-$8 per GB (so $4k to $8k per Terabyte). It may also depend on when that SAN was put into use. Were they able to use less expensive SATA drives, or did they need the raw performance of SCSI, etc. Plus cost per gigabyte slowly decreases over time (not as fast as it used to, but it's still a gradual decline of maybe 25% per year).

      --
      Wolde you bothe eate your cake, and have your cake?
  2. Not so accurate by cymru_slam · · Score: 4, Informative

    I work for one of the organisations listed and I have to say that what they described sounds NOTHING like our infrastructure :-)

    1. Re:Not so accurate by Znork · · Score: 4, Insightful

      "you shouldn't have end users plugged into a SAN."

      Exactly why shouldn't you have end-users plugged into a SAN? I run a SAN, and I find that diskless workstations PXE booting off gigabit iSCSI storage are a huge improvement to having local disk. For more or less exactly those reasons; performance, redundancy, flexibility, growth and sharing. Not to mention data consolidation and savings in less wasted local storage.

      I suspect the idea that SAN's are for servers is mostly spread by overcharging SAN vendors who dont want their profit margins eroded by inexpensive consumer devices. In fact, I'd say consumer storage is rapidly progressing beyond the server side and is these days the main driver behind storage expansion; I certainly know my home storage needs expands faster than the vast majority of the servers I admin (yes, there are the we-want-to-simulate-the-atoms-in-the-ocean exceptions, but most business application servers use less storage than you can get in an mp3 player).

    2. Re:Not so accurate by Bandman · · Score: 3, Insightful

      I'm not sure it's FUD, since that means "Fear Uncertainty and Doubt"

      It's more, inaccurate, or maybe a result of shallow researching, or at the very least simplified.

      --
      Check out my sysadmin blog!
    3. Re:Not so accurate by Anonymous Coward · · Score: 5, Insightful

      Well, it sounds like your environment is PC based. The environment I work in is server based. An end user could leave his/her computer in a taxi and we can have them up and running and productive on a new PC within minutes with little chance of actually losing anything. I say little chance because although we make every attempt to force things to the network through our computer system policies and document management systems, sometimes they still manage to put things in "My Documents" but that is the exception, not the norm. It is more then just a single user though. With that system in place, our entire office in downtime Washington DC could be blown up and the bulk of the offices business operations can be up and running from another one of our offices in another city or our companies DR site in a short period. For our environment, it is much easier to manage a backend and provide adequate remote user tools (Citrix for example) then it is to attempt to manage storage on a thousand or so individual computers. Imagine trying to do disaster recovery or emergency planning for an office that had a bunch of individual personal storage devices and a local PC based file storage system.
      Not everyone needs a SAN for storage but using a SAN is a very sound decision for those that need the capabilites it provides. A SAN is not just a buzz word although I do not doubt some people bought them without understanding what they were getting and why.

    4. Re:Not so accurate by DarthTaco · · Score: 5, Funny

      How about SHallow and Inaccurate Tripe?

    5. Re:Not so accurate by LWATCDR · · Score: 2, Insightful

      "but most business application servers use less storage than you can get in an mp3 player)."
      Yes they do.
      I am migrating the our support call, issue tracking, and RMA data base to a new server. We take a good number of calls a year and have almost six years of data on the server. The dump file is only 16 megabytes. Most business data is still text and text just doesn't eat up that much space.

      For home use doesn't and workstations does NAS make more sense than SAN? I am on a small network so we only use NAS for shared drives.

      --
      See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
    6. Re:Not so accurate by Philosinfinity · · Score: 4, Insightful

      Go to a law firm and ask them about their document management systems or their litigation support applications. Go to a bank and ask them about their financial records. What about email archives for compliance? Size up the disk space utilization and I think you will see many application servers that are significantly larger in storage than an MP3 player. Point taken, SANs can be used at the desktop level. But I partially wonder why? Wouldn't it be better to synchronize users' data folders with shares on a server that is diskless to the SAN? Why waste all that 'spensive storage just to make workstations diskless? Unless you are using a Compellant SAN or some SAN that is running a deduplication engine on the fly, you're stuck storing an OS install for each workstation.

      Besides this, I've always felt that the big advantage of a SAN is the ability to replicate an entire environment to another site in case of disaster. SANs are really utilized to the max in enterprise environments where these features are necessary for successful business operations.

    7. Re:Not so accurate by gnuman99 · · Score: 4, Funny

      our entire office in downtime Washington DC could be blown up


      I sense the little counter at NSA/"homeland security" click up -- Internet chatter about possible attack just increased! Few more like that, and terror alert will go up!! Geez people, watch what you type!

  3. Very U.S. Centric... by Noryungi · · Score: 5, Insightful

    Yes, I know, US web site and everything but, seriously, have you checked the data storage of CERN (birth place of the web) lately?

    If I remember correctly, these guys will generate petabytes of data per day when that monster particle accelerator goes online in a few months...

    --
    The right to offend is far more important than the right not to be offended. (Rowan Atkinson)
    1. Re:Very U.S. Centric... by barry_the_bogan · · Score: 5, Funny

      They're talking about the "world", as defined by the World Series Baseball people. Lame story.

    2. Re:Very U.S. Centric... by palpatin · · Score: 3, Informative

      Well, don't know about the storage capacity, but the LHC will produce around 15 petabytes per year, when they turn it on.

    3. Re:Very U.S. Centric... by TapeCutter · · Score: 2, Interesting

      IIRC they need a massive cache where the "sampling algorithm" throws a heap of data away. A quick google gives the following precise measure - "The LHC will generate data at a rate equivalent to every person on Earth making twenty phone calls at the same time." - but as you say it only stores a fraction of that.

      Now asuming the phone calls are made over POTS, the bitrate from the sensors should be...20 * 6*10^9 * 1220bps...

      --
      And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
    4. Re:Very U.S. Centric... by torako · · Score: 2, Informative
      Most of the caching is done using custom hardware that lives right in the detectors with latency in the order of s. The data output for permanent storage is (for the ATLAS detector, that's the one I know some stuff about) 200 MBytes / s, which is not gigantic. There are some PDF slides of a seminar talk on the triggering mechanism for ATLAS on my homepage in the Seminar section (in English even).

      Still, the data aquisition and storage system is impressive. Most of the storage will be distributed over different sites, so I don't know if there will be a huge central storage system.

    5. Re:Very U.S. Centric... by perturbed1 · · Score: 4, Informative

      I'll talk about one of the experiments, ATLAS. Yes we "generate" petabytes of data per day. It's rather easy to calculate actually. One collision in the detector can be compressed down to about 2MB raw data-- after lots of zero-suppression and smart-storage of bits from a detector that has ~100 million channels worth of readout information.

      There are ~30 million collisions a second -- as the LHC machine runs are 40Mhz but has a "gap" in its beam structure.

      Multiplying: 2 * 10^6 * 30 * 10^6 = 6* 10^13 Bytes per second. So ATLAS "produces" 1 petabyte of information in about 13 seconds!! :)

      But ATLAS is limited to being able to store about ~300 MB per second. This is the limit coming from how fast you can store things. Remember, there are 4 LHC experiments after all, and ATLAS gets its fair share of store capability.

      Which means that about of 30 million collisions per second, ATLAS can only store 150 collisions per second.... which it turns out is just fine!! The *interesting* physics only happens **very** rarely -- due to the nature of *weak* interactions. At the LHC, we are no-longer interested in the atom falling apart, and spitting its guts (quarks and gluons out). We are interested in rare processes such as dark-matter candidates or Higgs, or top-top production (which will dominate the 150Hz btw) and interesting and rare things. In most of the 30 million collisions, the protons spit their guts out and much much *rare* things occur. The catch of the trigger of ATLAS (and any other LHC experiment for that matter) is to find those *interesting* 150 events out of 30 million every second -- and do this in real time, and without a glitch. ATLAS uses about ~2000 computers to do this real-time data reduction and processing... CMS uses more, I believe.

      In the end, we get 300 MB/second worth of raw data and that's stored on tape at Tier 0 at CERN permanently -- and until the end of time as far as anyone is concerned. That data will never *ever* be removed. Actually the 5 Tier 1 sites will also have a full-copy of the data among themselves.

      Which brings me to my point that CERN storage is technically not a SAN (Storage Area Network)... (My IT buddies are insisting on this one. ) I am told that CERN storage counts as a NAS (Network Attached Storage). But I am going to alert them to this thread and will let them elaborate on that one!

    6. Re:Very U.S. Centric... by bushki3 · · Score: 4, Insightful

      from TFA

      "We at Byte and Switch are on the trail of the world's biggest SAN, and this article reveals our initial findings."

      and this

      "Again, this list is meant to be a starting place for further endeavor. If you've got a big SAN story to tell, we'd love to hear it."

      oh, and this too

      "we present five of the world's biggest SANs:"

      notice how everything in TFA clearly says this is not THE 5 BIGGEST SAN's in the world but the 5 largest they have found SO FAR.

      I know -- I must be new here, but I'm getting there. I didn't read the whole article, just a few sentences from the first page.

      --
      011100110110100101100111
    7. Re:Very U.S. Centric... by perturbed1 · · Score: 2, Informative

      Can't imagine why you write this as AC, but ok...

      Answer: "That's only 300MB/s 24/7 for more than half a year for writing the raw data to storage. Then there are the other three experiments with the same amount of data, actually one of them does 1.2GB/s of raw data. The data ends up on disk first with an aggregate write speed of ~1.5GB/s (let's not exaggerate). The data is read immediately from disk again to be written to tape (our final storage media), so ~1.5GB/s reads ... Then, all this data is being exported to external computing centers pretty much immediately too (multiple copies etc. etc., so aggregate is much higher than 1.5GB/s), so we get ~3GB/s of reads just from this data export (it can, potentially, be a lot more. we have already a total of 120Gbit/s of network connectivity to those sites). So, we are already at ~6GB/s of I/O and nobody even had a look into the data itself!! If we talk data analysis, we talk about repeated reprocessing runs over the entire collection of raw data in order to "create" the data format that physicists can more easily use for their analysis, we talk about several thousand people accessing all the accumulated data in a perfectly random way ... mind you, we keep all the raw data active, so 10 years there will be at least 100PB, probably more like 150PB, maybe even 200PB of active storage. The current estimate for the I/O caused by the data analysis is in the order of 50GB/s (big B). "

  4. The big surprise is by Centurix · · Score: 5, Funny

    that all the disks are formatted FAT32...

    --
    Task Mangler
    1. Re:The big surprise is by ichigo+2.0 · · Score: 2, Funny

      2 TB should be enough for everybody!

  5. Pronunciation is the key by User+956 · · Score: 4, Funny

    Finally somewhere to store all my porn

    We're talking about Petabytes, not Pedobytes.

    --
    The theory of relativity doesn't work right in Arkansas.
  6. 14Pb for 170k employees... by Joce640k · · Score: 4, Insightful

    14Pb for 170k employees isn't so much - 83 gigabytes per person.

    If you add up the total disk space in an average office you'll get more than that. If I add up all my external disks, etc. I've got more than a terabyte on my desktop.

    (And yes it's true, data does grow to fit the available space)

    --
    No sig today...
    1. Re:14Pb for 170k employees... by Enderandrew · · Score: 2, Informative

      When I generate ghost images for the PCs here at work, the average desktop user goes through about 4 gigs here, if that. 83 gigs per person is quite a bit.

      I'm also curious about Google and the like. Do they not disclose their storage?

      --
      http://blindscribblings.com - Tasty pop-culture in conceptual fashion.
    2. Re:14Pb for 170k employees... by StarfishOne · · Score: 2, Informative
      "I'm also curious about Google and the like. Do they not disclose their storage?"


      To a certain extend they have disclosed some numbers in a paper about their distributed storage system called "BigTable". The title of the paper is "Bigtable: A Distributed Storage System for Structured Data" and it can be found right here.

      Some numbers can be found on page 11:
      Project and Table size in TB:

      Crawl: 800
      Crawl: 50
      Google Analytics: 20
      Google Analytics: 200 (Raw click table)
      Google Base: 2
      Google Earth: 0.5
      Google Earth: 70
      Orkut: 9
      Personalized Search: 4

      Total so far: 1,155.5 TB

      It's a very interesting paper to read. One of the many papers Google has put online:

    3. Re:14Pb for 170k employees... by commlinx · · Score: 3, Funny

      14Pb for 170k employees isn't so much - 83 gigabytes per person. If you add up the total disk space in an average office you'll get more than that. If I add up all my external disks, etc. I've got more than a terabyte on my desktop.

      You'd find a lot of the 83GB on a typical office PC is crap you're not going to put in a SAN, my boot drive without data has 50GB used and other than the pain in the arse of re-installing I couldn't give a toss if I lost all that "data". Yes I've got a TB of storage too but subtract p0rn, DVDs and other contents that would get me sacked if I worked in a corporate environment, subtract the large amount of reference material (that would be shared between users in a corporate environment) and all my original work for the past 10 years amounts to well under 10GB.

      The data use has little use to do with the number of employees - it amounts to how much data you are getting from external sources whether they be a large number of customers or data acquisition (such as digital photographs). If you're talking text for example a fast typist is probably hitting something like 10 characters per second, a whopping 280K per working day, 70MB per working year, 3.5GB over their life.

    4. Re:14Pb for 170k employees... by somersault · · Score: 2, Funny

      I hope they filter all those clicks before they dump them in the landfill. Can you imagine the mess 200 terabytes of raw clicks would make?

      --
      which is totally what she said
    5. Re:14Pb for 170k employees... by fellip_nectar · · Score: 4, Informative

      No he didn't.

      --
      Worst. Signature. Ever.
  7. At Last! by Zymergy · · Score: 5, Funny

    Someone can install a FULL install of Windows Vista!

  8. Shouldn't this be written somewhere? by rm999 · · Score: 4, Informative

    SAN = Storage area network

    1. Re:Shouldn't this be written somewhere? by OverlordQ · · Score: 4, Funny

      This is slashdot, if you dont know what SAN stands for, please turn in your geek card and report to Digg.

      --
      Your hair look like poop, Bob! - Wanker.
    2. Re:Shouldn't this be written somewhere? by Eideewt · · Score: 2, Informative

      Why thank you. I was trying to figure out what a Storage Attached Network might be.

  9. ...and why does the article say "Pbytes", "Tbytes" by Joce640k · · Score: 2, Informative

    ...and why does the article say "Pbytes", "Tbytes", etc.

    The abbreviated units are "PB" and "TB".

    See: http://en.wikipedia.org/wiki/Petabyte

    --
    No sig today...
  10. That's nothing... by Jah-Wren+Ryel · · Score: 3, Funny

    "ByteandSwitch is searching the World's Biggest SANs, and has compiled a list of 5 candidate with networks supports 10+ Petabytes of active storage. What? That's nothing. I've got 100 petabytes just for my pr0n collection!
    --
    When information is power, privacy is freedom.
  11. Sooo... by Tastecicles · · Score: 2, Funny

    My home entertainment server at 3.3TB RAID6 isn't even in the running then?

    Bugger.

    --
    Operation Guillotine is in effect.
    1. Re:Sooo... by Chineseyes · · Score: 3, Funny

      Unfortunately your "entertainment server" is not among the winners for biggest SANs but it IS in the running for "copyright offender of the year". Our lawyers will be contacting you with your prize soon. Sincerely, Steven Marks Executive Vice President and General Counsel, RIAA

      --
      I think the invisible hand of the market has its middle finger extended

      --A wise old fart named SC0RN
  12. Let's hope CERN's data can be zipped... by Joce640k · · Score: 4, Funny

    Let's hope CERN's data can be zipped...if not, they'll be in trouble pretty quickly.

    Remember when you got your first copy of Napster and ADSL? That's how serious...!

    --
    No sig today...
  13. Just SANs... so what? by Duncan3 · · Score: 5, Interesting

    Kinda like saying the worlds fastestest runner that likes swiss cheese best. This isn't a list of fastest, largest, most used, etc. Just just some PR spin for SANs. Nothing wrong with that, but still.

    --
    - Adam L. Beberg - The Cosm Project - http://www.mithral.com/
  14. Details? by clarkkent09 · · Score: 4, Funny

    Ah, go on tell us. We won't tell anybody

    --
    Negative moral value of force outweighs the positive value of good intentions.
  15. They forgot about ... by chris_sawtell · · Score: 2, Interesting

    Google, the WayBack Machine, to say nothing of the 1.5 million machine bot-net we've been hearing about recently.

  16. Re:security, resilience, risk, etc by statemachine · · Score: 2, Informative

    And to that I ask: How paranoid are you, and how much money do you have?

    You also talk about copies of data as if a disk went bad, you'd lose the data. These storage arrays have multiple redundancies (RAIDs of VDisks which are RAIDs themselves) as well as having live replication capability to remote sites -- at which point you likely have a copy (or copies) of an entire datacenter in a different geographic location that is running as a hot spare.

    Within a datacenter, you would not have more than dual fabrics. Your fabrics' switches will also be redundantly connected within themselves. And if you're killing an entire fabric with an upgrade, you're doing it wrong.

    You'll also have service contracts with lockers of disks, switches, linecards, etc., *on site* with field technicians from the vendors on-call 24/7.

    Fibre Channel installations are not like some small company's Ethernet LAN.

  17. Discount Web host with scalable SAN by rjamestaylor · · Score: 3, Funny
    Recently I found a discount online web hosting company
    with an unlikely name that offers a scalable,
    distributable SAN, called an HDSAN
    (High Density Storage Area Network),
    for its customers:

    SlumLordHosting.com
    --
    -- @rjamestaylor on Ello
  18. Re:...and why does the article say "Pbytes", "Tbyt by imsabbel · · Score: 4, Insightful

    Because EVERY SINGLE FUCKING story with "TB" and "GB" causes arguments in the way of "this has to be "...bits", the number is too large for bytes" or vice versa even here.
    To avoid missunderstandings, 4 additionals bytes (B) dont seem that much of a price.

    --
    HI O WISE PRINCE. WHT TOOK U SO DAM LONG?
  19. What I want to know by teslatug · · Score: 2, Interesting

    How do they do backups (especially online ones) and restores?

  20. Re:mod parent up by j-cloth · · Score: 2, Funny

    I would, but I'm a grammar nazi and then != than

  21. I think we need to coin a few new terms by Teilo · · Score: 2, Funny

    Pr0ntab: A score, equal to the amount of time in tenths of a minute, that elapses from the moment a news article is posted to the first comment relating said article to a person's porn collection or viewing habits.

    Pr0ntible: The statistical likelihood that any given article will have a low Pr0ntab score, where 1.0 is the highest score, and 0 the lowest.

    Pr0ntabulary: A time sensitive, categorical table of subject matter, where each category is assigned a Pr0ntible, and said table is organized in descending order by Pr0ntible.

    Example: On today's Pr0ntabulary, the Storage category ranks near the top.

    --
    Mir tut es leid, Menschen daß Einfältigfehlersuchenbaumfolgendenaffen sind.