World's Five Biggest SANs

An anonymous reader writes "ByteandSwitch is searching the World's Biggest SANs, and has compiled a list of 5 candidate with networks supports 10+ Petabytes of active storage. Leading the list is JPMorgan Chase, which uses a mix of IBM and Sun equipment to deliver 14 Pbytes for 170k employees. Also on the list are the U.S. DoD, which uses 700 Fibre Channel switches, NASA, the San Diego Supercomputer Center (it's got 18 Pbytes of tape! storage), and Lawrence Livermore."

Comments

... That we know about

[Anonymous Coward]

What about Google, Amazon, Yahoo, Microsoft, etc.?

Re:... That we know about

[Anonymous Coward]

Google doesn't use conventional SAN architectures, so they probably wouldn't qualify for this list.

Re:... That we know about

[Anonymous Coward]

And there wasn't even a single Japanese firm listed. You'd think they'd have the biggest SANs of all.

Re:... That we know about

[Chapter80]

I was thinking San Diego, San Francisco, San Antonio, San Jose, Santa Claus.

Re:... That we know about

[curlynoodle]

I expect that government and non-profit organizations are more likely to release such info. It make them appear high-tech and cool. Beats me as to why Chase would comment on their storage, maybe for shareholders.

As for Google, Microsoft, Amazon, Yahoo, etc, I expect its important to business not to share system architecture with the public. However, read http://labs.google.com/papers/disk_failures.html to get an idea as to Google's storage systems.

Re:... That we know about

[homey of my owney]

The list seems arbitrary. The mother of them all is the Naval Oceanographic Office, which as I recall is a mix of IBM, EMC and Hitachi - isn't listed.

Re:... That we know about

[GoodOmens]

I know it's not a PB but here at the Census we have one 150TB array thats used for one project and not 170K employees.

It is interesting when you get in storage of this size. I remember sitting in a meeting where we discussed a storage cabinet we were ordering. The RAW size of the cabinet was 150TB but formatted it would be 100TB ... 50TB is a lot of storage to "throw away" for redundancy / formatting! Considering at this price your paying about $10k+ a TB (With staff and infrastructure costs fractured in)!

Re:... That we know about

[valen]

  Google don't release stats on the size of their clusters, obviously. But this is what they were doing four years ago;

  http://labs.google.com/papers/gfs.html
  http://labs.google.com/papers/bigtable.html

  Since then, they've double their engineering staff every year, and rolled out a lot of new purpose-build datacenters. I've seen public guesstimates as high as 400,000 machines. With SATA, it's not unreasonable to imagine 5 or 10 disks in every machine. That's a lot of storage. Of course, having one 'SAN' makes no sense; you need it distributed, so latency to each user is as low as possible.

  Bigtable in particular is interesting; a database designed to handle many petabytes of data in a low-latency environment.

  Sigh. I really really wish we could be completely candid about all this stuff. Maybe over time...

John

Re:... That we know about

[WuphonsReach]

$10k per Terabyte isn't all that bad; maybe about 30-50% higher. Server level storage goes for $4-$8 per GB (so $4k to $8k per Terabyte). It may also depend on when that SAN was put into use. Were they able to use less expensive SATA drives, or did they need the raw performance of SCSI, etc. Plus cost per gigabyte slowly decreases over time (not as fast as it used to, but it's still a gradual decline of maybe 25% per year).

Not so accurate

[cymru_slam]

I work for one of the organisations listed and I have to say that what they described sounds NOTHING like our infrastructure :-)

Re:Not so accurate

[bigmouth_strikes]

Not to imply that you don't know what you're talking about, but isn't part of the point of a SAN; to an end-user it practically looks and behaves as if it were a local device ? Of course, you might be in the know of the overall storage infrastructure, but then again... you might not.

So to save your reputation you'll have to spill your beans, so to speak.

Re:Not so accurate

[Znork]

"you shouldn't have end users plugged into a SAN."

Exactly why shouldn't you have end-users plugged into a SAN? I run a SAN, and I find that diskless workstations PXE booting off gigabit iSCSI storage are a huge improvement to having local disk. For more or less exactly those reasons; performance, redundancy, flexibility, growth and sharing. Not to mention data consolidation and savings in less wasted local storage.

I suspect the idea that SAN's are for servers is mostly spread by overcharging SAN vendors who dont want their profit margins eroded by inexpensive consumer devices. In fact, I'd say consumer storage is rapidly progressing beyond the server side and is these days the main driver behind storage expansion; I certainly know my home storage needs expands faster than the vast majority of the servers I admin (yes, there are the we-want-to-simulate-the-atoms-in-the-ocean exceptions, but most business application servers use less storage than you can get in an mp3 player).

Re:Not so accurate

[Bandman]

I'm not sure it's FUD, since that means "Fear Uncertainty and Doubt"

It's more, inaccurate, or maybe a result of shallow researching, or at the very least simplified.

Re:Not so accurate

[Anonymous Coward]

Well, it sounds like your environment is PC based. The environment I work in is server based. An end user could leave his/her computer in a taxi and we can have them up and running and productive on a new PC within minutes with little chance of actually losing anything. I say little chance because although we make every attempt to force things to the network through our computer system policies and document management systems, sometimes they still manage to put things in "My Documents" but that is the exception, not the norm. It is more then just a single user though. With that system in place, our entire office in downtime Washington DC could be blown up and the bulk of the offices business operations can be up and running from another one of our offices in another city or our companies DR site in a short period. For our environment, it is much easier to manage a backend and provide adequate remote user tools (Citrix for example) then it is to attempt to manage storage on a thousand or so individual computers. Imagine trying to do disaster recovery or emergency planning for an office that had a bunch of individual personal storage devices and a local PC based file storage system.
Not everyone needs a SAN for storage but using a SAN is a very sound decision for those that need the capabilites it provides. A SAN is not just a buzz word although I do not doubt some people bought them without understanding what they were getting and why.

Re:Not so accurate

[DarthTaco]

How about SHallow and Inaccurate Tripe?

Re:Not so accurate

[LWATCDR]

"but most business application servers use less storage than you can get in an mp3 player)."
Yes they do.
I am migrating the our support call, issue tracking, and RMA data base to a new server. We take a good number of calls a year and have almost six years of data on the server. The dump file is only 16 megabytes. Most business data is still text and text just doesn't eat up that much space.

For home use doesn't and workstations does NAS make more sense than SAN? I am on a small network so we only use NAS for shared drives.

Re:Not so accurate

[Philosinfinity]

Go to a law firm and ask them about their document management systems or their litigation support applications. Go to a bank and ask them about their financial records. What about email archives for compliance? Size up the disk space utilization and I think you will see many application servers that are significantly larger in storage than an MP3 player. Point taken, SANs can be used at the desktop level. But I partially wonder why? Wouldn't it be better to synchronize users' data folders with shares on a server that is diskless to the SAN? Why waste all that 'spensive storage just to make workstations diskless? Unless you are using a Compellant SAN or some SAN that is running a deduplication engine on the fly, you're stuck storing an OS install for each workstation.

Besides this, I've always felt that the big advantage of a SAN is the ability to replicate an entire environment to another site in case of disaster. SANs are really utilized to the max in enterprise environments where these features are necessary for successful business operations.

Re:Not so accurate

[gnuman99]

our entire office in downtime Washington DC could be blown up

I sense the little counter at NSA/"homeland security" click up -- Internet chatter about possible attack just increased! Few more like that, and terror alert will go up!! Geez people, watch what you type!

Re:Not so accurate

[Znork]

"For home use doesn't and workstations does NAS make more sense than SAN? I am on a small network so we only use NAS for shared drives."

Yes and no. Shared storage such as home directories, shared files, etc, are better on a NAS.

The advantage of SAN connections for the desktop lies in the OS and paging space area, for speed critical applications, and more in the area of maintenance and support than in convenience. Getting rid of the local disk gives you the ability to do things like migrate clients to new operating systems with a simple reboot (and still have the old disk image to fall back on), take snapshots of client disks, save large amounts of space by buying more cost effective disks (smallest disks available today are, like, 80 gb, where the OS needs maybe 10. And larger disks are far more cost effective per gigabyte, so sharing them is cheaper). Etc.

Basically you get most of the advantages of thin-client hardware, without suffering from as many of the disadvantages.

Re:Not so accurate

[Znork]

"What about email archives for compliance?"

Good examples of business uses, but still comparatively small (depending on company size). Compare with PVR storage, mpeg files, etc.

There are, of course, larger storage needs for some applications, particularly in large companies, but I see more and more cases of formerly huge applications where I had a terabyte storage ten years ago and the disk arrays were awe inspiring (and cost a fortune). Now the same systems will have a terabyte and a half storage, and there are almost consumer grade disks that large.

Enterprise storage simply isnt growing that fast anymore, nor is most of it extreme. The litigation support apps, financial records and email archives grow at a snail pace compared to things like PVR applications that record many hours worth of video per day (and that's not even in HD).

"Why waste all that 'spensive storage just to make workstations diskless?"

You use the cheapo storage instead. Gigabit ethernet and cheapo (well, cheap-er at least, and actually cheap if you can live with a diy solution) iSCSI servers are perfectly adequate for workstation use.

"you're stuck storing an OS install for each workstation."

Depends on how you do it. If you're on linux and for something like classroom use, you can use unionfs to overlay a shared image and save that space too.

But even if you are stuck using an OS install for each workstation, the fact is you need far less OS diskspace per client than the smallest available disks these days. Add to that the ability to non-destructively change OS images with a simple reboot rather than a reinstall, coupled with easier backups, better data control, etc, and there are several compelling advantages to it.

Re:Not so accurate

[Philosinfinity]

Good call on the overall disk space savings. I wasn't really thinking about the unneeded hard disk space on the client boxes. It is definitely an interesting potential configuration... especially if you could pipe a deduplication engine in front of the disk array. Granted you would take a performance hit, but you could save a great deal of space.

As far as the overall decrease of disk necessity in the enterprise, I'm not so sure I agree. Our email environment is about 3TB local with another 5. - 1.5TB in each of 8 remote offices. Every time we order more shelves for our SAN, the storage is claimed before the hardware comes in. If anything, I think programmers are taking the increase of disk size as a open door to utilize as much of it as possible. Even looking at the consumer end, this holds true. New digital camera shots are easily 10MB each. My old Sony Mavica could store about 10 full res images in a single floppy. Blank Word documents start off around 23K. I think the argument that storage necessity has consistently gone up dramatically over the last 15 years, 10 years, and even 5 years is nothing short of accurate. This is especially true in the enterprise. For instance, our firm received a new case with a bunch of documents to scan into our litigation support database. Between that, PST files, and other information, we utilized over 2TB of disk space. That's 1 case. Hi-res images of case documents, emails, and our financial DB grow exponentially. I've worked on DR plans for several clients throughout the enterprise and they all experience the same thing. Perhaps some areas are slightly decreasing in disk utilization, but the big apps are growing and growing.

Very U.S. Centric...

[Noryungi]

Yes, I know, US web site and everything but, seriously, have you checked the data storage of CERN (birth place of the web) lately?

If I remember correctly, these guys will generate petabytes of data per day when that monster particle accelerator goes online in a few months...

Re:Very U.S. Centric...

[barry_the_bogan]

They're talking about the "world", as defined by the World Series Baseball people. Lame story.

Re:Very U.S. Centric...

[palpatin]

Well, don't know about the storage capacity, but the LHC will produce around 15 petabytes per year, when they turn it on.

Re:Very U.S. Centric...

[Daniel Phillips]

If I remember correctly, these guys will generate petabytes of data per day Rechecking your facts would be in order. Some 10 to 15 petabytes are expected to be saved per year according to sources I have seen, though only a small fraction of the raw sensor data will be permanently recorded.

Re:Very U.S. Centric...

[JorDan Clock]

You would have been real upset before San Francisco was taken off the list.

Re:Very U.S. Centric...

[TapeCutter]

IIRC they need a massive cache where the "sampling algorithm" throws a heap of data away. A quick google gives the following precise measure - "The LHC will generate data at a rate equivalent to every person on Earth making twenty phone calls at the same time." - but as you say it only stores a fraction of that.

Now asuming the phone calls are made over POTS, the bitrate from the sensors should be...20 * 6*10^9 * 1220bps...

Re:Very U.S. Centric...

[torako]

Most of the caching is done using custom hardware that lives right in the detectors with latency in the order of s. The data output for permanent storage is (for the ATLAS detector, that's the one I know some stuff about) 200 MBytes / s, which is not gigantic. There are some PDF slides of a seminar talk on the triggering mechanism for ATLAS on my homepage in the Seminar section (in English even).

Still, the data aquisition and storage system is impressive. Most of the storage will be distributed over different sites, so I don't know if there will be a huge central storage system.

Re:Very U.S. Centric...

[bockelboy]

The CMS detector will take data at 8GB/s at turn on (that's gigabytes, not gigabits). This will be filtered and a few percent will be saved.

CASTOR's (the CERN data store) current stats are here:

http://castor.web.cern.ch/castor/

About 8 PB of files. If i recall correctly, there's around 500TB of online disk space and 10-30PB of tape storage (some of it is getting phased out).

FNAL has a similar setup, except with a storage manager called dCache. There is no use of protocols like iSCSI or Fiber Channel over IP, but rather physics-specific ones (xrootd, rfio, dcap) and grid-specific ones (SRM and gridFTP).

Re:Very U.S. Centric...

[perturbed1]

I'll talk about one of the experiments, ATLAS. Yes we "generate" petabytes of data per day. It's rather easy to calculate actually. One collision in the detector can be compressed down to about 2MB raw data-- after lots of zero-suppression and smart-storage of bits from a detector that has ~100 million channels worth of readout information.

There are ~30 million collisions a second -- as the LHC machine runs are 40Mhz but has a "gap" in its beam structure.

Multiplying: 2 * 10^6 * 30 * 10^6 = 6* 10^13 Bytes per second. So ATLAS "produces" 1 petabyte of information in about 13 seconds!! :)

But ATLAS is limited to being able to store about ~300 MB per second. This is the limit coming from how fast you can store things. Remember, there are 4 LHC experiments after all, and ATLAS gets its fair share of store capability.

Which means that about of 30 million collisions per second, ATLAS can only store 150 collisions per second.... which it turns out is just fine!! The *interesting* physics only happens **very** rarely -- due to the nature of *weak* interactions. At the LHC, we are no-longer interested in the atom falling apart, and spitting its guts (quarks and gluons out). We are interested in rare processes such as dark-matter candidates or Higgs, or top-top production (which will dominate the 150Hz btw) and interesting and rare things. In most of the 30 million collisions, the protons spit their guts out and much much *rare* things occur. The catch of the trigger of ATLAS (and any other LHC experiment for that matter) is to find those *interesting* 150 events out of 30 million every second -- and do this in real time, and without a glitch. ATLAS uses about ~2000 computers to do this real-time data reduction and processing... CMS uses more, I believe.

In the end, we get 300 MB/second worth of raw data and that's stored on tape at Tier 0 at CERN permanently -- and until the end of time as far as anyone is concerned. That data will never *ever* be removed. Actually the 5 Tier 1 sites will also have a full-copy of the data among themselves.

Which brings me to my point that CERN storage is technically not a SAN (Storage Area Network)... (My IT buddies are insisting on this one. ) I am told that CERN storage counts as a NAS (Network Attached Storage). But I am going to alert them to this thread and will let them elaborate on that one!

Re:Very U.S. Centric...

[bushki3]

from TFA

"We at Byte and Switch are on the trail of the world's biggest SAN, and this article reveals our initial findings."

and this

"Again, this list is meant to be a starting place for further endeavor. If you've got a big SAN story to tell, we'd love to hear it."

oh, and this too

"we present five of the world's biggest SANs:"

notice how everything in TFA clearly says this is not THE 5 BIGGEST SAN's in the world but the 5 largest they have found SO FAR.

I know -- I must be new here, but I'm getting there. I didn't read the whole article, just a few sentences from the first page.

Re:Very U.S. Centric...

[perturbed1]

Can't imagine why you write this as AC, but ok...

Answer: "That's only 300MB/s 24/7 for more than half a year for writing the raw data to storage. Then there are the other three experiments with the same amount of data, actually one of them does 1.2GB/s of raw data. The data ends up on disk first with an aggregate write speed of ~1.5GB/s (let's not exaggerate). The data is read immediately from disk again to be written to tape (our final storage media), so ~1.5GB/s reads ... Then, all this data is being exported to external computing centers pretty much immediately too (multiple copies etc. etc., so aggregate is much higher than 1.5GB/s), so we get ~3GB/s of reads just from this data export (it can, potentially, be a lot more. we have already a total of 120Gbit/s of network connectivity to those sites). So, we are already at ~6GB/s of I/O and nobody even had a look into the data itself!! If we talk data analysis, we talk about repeated reprocessing runs over the entire collection of raw data in order to "create" the data format that physicists can more easily use for their analysis, we talk about several thousand people accessing all the accumulated data in a perfectly random way ... mind you, we keep all the raw data active, so 10 years there will be at least 100PB, probably more like 150PB, maybe even 200PB of active storage. The current estimate for the I/O caused by the data analysis is in the order of 50GB/s (big B). "

Re:Very U.S. Centric...

[BoothbyTCD]

You mean like 'The New York World', a now defunct newspaper that was the original sponsor of the first MLB championship? I know we like to bash US cluelessness here, but, this particular bit of misinformation always irks me.

The big surprise is

[Centurix]

that all the disks are formatted FAT32...

Re:The big surprise is

[ichigo 2.0]

2 TB should be enough for everybody!

Re:The big surprise is

[CastrTroy]

Unless you run Windows, then they only let you create partitions up to 32 GB.

Re:The big surprise is

[drseuk]

Pah, 640k is more than enough for real hackers. P.S., At 14+ PB, the command line to unsplit the 2GB FAT splits is itself going to exceed 2GB. Anyone got a workround? (I'm running Vista SP0 I think).

Re:The big surprise is

[Anonymous Coward]

Whoosh!

Pronunciation is the key

[User 956]

Finally somewhere to store all my porn

We're talking about Petabytes, not Pedobytes.

Re:Pronunciation is the key

[BosstonesOwn]

Does that mean he has lots of PedoFiles ;)

Re:Pronunciation is the key

[Kjella]

Not to mention PETAbytes - "No animals were hurt during the production of this storage medium.

Re:Pronunciation is the key

[jamstar7]

Next time, try harder to injure the l*wy*rs.

14Pb for 170k employees...

[Joce640k]

14Pb for 170k employees isn't so much - 83 gigabytes per person.

If you add up the total disk space in an average office you'll get more than that. If I add up all my external disks, etc. I've got more than a terabyte on my desktop.

(And yes it's true, data does grow to fit the available space)

Re:14Pb for 170k employees...

[Enderandrew]

When I generate ghost images for the PCs here at work, the average desktop user goes through about 4 gigs here, if that. 83 gigs per person is quite a bit.

I'm also curious about Google and the like. Do they not disclose their storage?

Re:14Pb for 170k employees...

[StarfishOne]

"I'm also curious about Google and the like. Do they not disclose their storage?"

To a certain extend they have disclosed some numbers in a paper about their distributed storage system called "BigTable". The title of the paper is "Bigtable: A Distributed Storage System for Structured Data" and it can be found right here.

Some numbers can be found on page 11:
Project and Table size in TB:

Crawl: 800
Crawl: 50
Google Analytics: 20
Google Analytics: 200 (Raw click table)
Google Base: 2
Google Earth: 0.5
Google Earth: 70
Orkut: 9
Personalized Search: 4

Total so far: 1,155.5 TB

It's a very interesting paper to read. One of the many papers Google has put online:

Re:14Pb for 170k employees...

[commlinx]

14Pb for 170k employees isn't so much - 83 gigabytes per person. If you add up the total disk space in an average office you'll get more than that. If I add up all my external disks, etc. I've got more than a terabyte on my desktop.

You'd find a lot of the 83GB on a typical office PC is crap you're not going to put in a SAN, my boot drive without data has 50GB used and other than the pain in the arse of re-installing I couldn't give a toss if I lost all that "data". Yes I've got a TB of storage too but subtract p0rn, DVDs and other contents that would get me sacked if I worked in a corporate environment, subtract the large amount of reference material (that would be shared between users in a corporate environment) and all my original work for the past 10 years amounts to well under 10GB.

The data use has little use to do with the number of employees - it amounts to how much data you are getting from external sources whether they be a large number of customers or data acquisition (such as digital photographs). If you're talking text for example a fast typist is probably hitting something like 10 characters per second, a whopping 280K per working day, 70MB per working year, 3.5GB over their life.

Re:14Pb for 170k employees...

[WindowsIsForArseWipe]

1Gb of natilie portman and hot grits should be enough for anyone

Re:14Pb for 170k employees...

[somersault]

I hope they filter all those clicks before they dump them in the landfill. Can you imagine the mess 200 terabytes of raw clicks would make?

Re:14Pb for 170k employees...

[asserted]

> I'm also curious about Google and the like. Do they not disclose their storage?

there's much more than that, but GFS is no SAN at all. google can do better than that, and does.
GFS is all about cheap storage, lots of it. and yes, 14 P is basically nothing, in google terms.
what article really is about is "who wasted more money on over-priced enterprisey SAN crap".

Re:14Pb for 170k employees...

[fellip_nectar]

No he didn't.

Re:14Pb for 170k employees...

[RogerWilco]

No he did not.

Re:14Pb for 170k employees...

[me_mi_mo]

Mod parent down.

Grandparent is correct.

Re:14Pb for 170k employees...

[PremiumCarrion]

My calculations suggest grandparent is only out by a factor of ~1.07

My assumptions are as follows:
Pb == 10^15 bytes
170k people == 170 000 people

Therefore 14 000 000 000 000 000 / 170 000 = 82 352 941 176 (rounded)

i.e under 83GB per user

Re:14Pb for 170k employees...

[gardyloo]

I'd call it a good start.

Re:14Pb for 170k employees...

[Daniel Phillips]

Missed the "k", my bad. Not that that should justify some of the responses...

Re:14Pb for 170k employees...

[ToasterMonkey]

Big surprise, NOBODY on slashdot knows what a SAN is. Return your geek cards!

At Last!

[Zymergy]

Someone can install a FULL install of Windows Vista!

Re:At Last!

[kraemate]

Pfft. I am going to need this to install all the packages in the 12 cds of Debian Sarge.

Re:At Last!

[Pingmaster]

and yet, still they cannot find a machine with enough RAM to run it...

Shouldn't this be written somewhere?

[rm999]

SAN = Storage area network

Re:Shouldn't this be written somewhere?

[OverlordQ]

This is slashdot, if you dont know what SAN stands for, please turn in your geek card and report to Digg.

Re:Shouldn't this be written somewhere?

[Eideewt]

Why thank you. I was trying to figure out what a Storage Attached Network might be.

Re:Shouldn't this be written somewhere?

[Smurfeur]

I thought that SAN = "Call of Cthulhu"'s sanity.

Does it count as geeky ?

Re:Shouldn't this be written somewhere?

[__aailrp9629]

Sanity, right?

Re:Shouldn't this be written somewhere?

[mgblst]

What is this network you speak of? And storage, I am guessing you mean cupboard space?

Re:Shouldn't this be written somewhere?

[ari wins]

Crap. :(

I'm going to miss you fuckers.

...and why does the article say "Pbytes", "Tbytes"

[Joce640k]

...and why does the article say "Pbytes", "Tbytes", etc.

The abbreviated units are "PB" and "TB".

See: http://en.wikipedia.org/wiki/Petabyte

That's nothing...

[Jah-Wren Ryel]

"ByteandSwitch is searching the World's Biggest SANs, and has compiled a list of 5 candidate with networks supports 10+ Petabytes of active storage. What? That's nothing. I've got 100 petabytes just for my pr0n collection!

Re:That's nothing...

[Anonymous Coward]

yeah I've had a look. How you managed to find so much midget pr0n is beyond me...

Re:That's nothing...

[Lxy]

Wow... that's a lot of petafiles *rimshot*

Re:Finally ...

[Zymergy]

Australian library may catalogue Internet porn http://news.bbc.co.uk/2/hi/asia-pacific/2221489.stm You may Need to give them a call....

I didn't say the disks were full...

[Joce640k]

I didn't say the disks were full, just that the storage available per person in the average office is more than that.

Does the whole 14Pb go in a single room? That might be impressive.

Re:I didn't say the disks were full...

[Enderandrew]

For purposes of disaster recovery, I sure hope not.

Sooo...

[Tastecicles]

My home entertainment server at 3.3TB RAID6 isn't even in the running then?

Bugger.

Re:Sooo...

[farkus888]

haha, I remember it was only a few short years ago I was hot shit because my personal file server had 500 gig of storage in it. most storage in a single computer of anyone I knew. I had a gig of ram and one of the fancy p4 processors with twice the cache of all the wussy ones. man was I cool. of course now I am older and don't have nearly the discretionary computer money I used to, and I am still running that exact same hardware having wet dreams about joining the 64bit real dual core revolution.

Re:Sooo...

[clarkkent09]

You mean your porn server?

Re:Sooo...

[EmagGeek]

I am running way behind! My home video server only has 1920GB of storage via 8x320GB drives in a Raid-5... but I know what you mean. In college, I put four 1.6GB drives in my box, and did a Linux Stripe to 6.4GB, and that was hot shit back then. We stored lots of CDs in that new-fangled MP3 format on there.

Re:Sooo...

[Tastecicles]

'Ave you been peekin' through my window?

Re:Sooo...

[Chineseyes]

Unfortunately your "entertainment server" is not among the winners for biggest SANs but it IS in the running for "copyright offender of the year". Our lawyers will be contacting you with your prize soon. Sincerely, Steven Marks Executive Vice President and General Counsel, RIAA

Re:Sooo...

[Tastecicles]

Dear Mr. Marks,
                              Apologies for the prompt reply, but I feel it falls within my diplomatic obligation to inform you that **AA has no jurisdiction within 2274 miles of the location of my home media storage. I would also like to inform you that said server contains material which I OWN in optical or magnetic tape format which falls within my consumer rights to format shift to view or listen on devices of MY choosing.

So, fsck off. :p

Let's hope CERN's data can be zipped...

[Joce640k]

Let's hope CERN's data can be zipped...if not, they'll be in trouble pretty quickly.

Remember when you got your first copy of Napster and ADSL? That's how serious...!

Just SANs... so what?

[Duncan3]

Kinda like saying the worlds fastestest runner that likes swiss cheese best. This isn't a list of fastest, largest, most used, etc. Just just some PR spin for SANs. Nothing wrong with that, but still.

Large Hadron Collider (CERN) will need 15 PB/yr

[infolib]

At least according to this article.

It's planned to go running next spring, but it's already at 3.5 PB. (The old LEP collider working in the same tunnel produced quite a bit of data too).

Details?

[clarkkent09]

Ah, go on tell us. We won't tell anybody

They forgot about ...

[chris_sawtell]

Google, the WayBack Machine, to say nothing of the 1.5 million machine bot-net we've been hearing about recently.

Re:They forgot about ...

[swordgeek]

You apparently don't have the slightest clue what a SAN is.

security, resilience, risk, etc

[kefa]

From my experience most FTSE 100 companies in the UK have multi-petabytes of storage so I'm assuming that the article is referring to a single consolidated SAN and not disparate SAN islands. Although it is interesting to examine the limits of scalability for such an environment on theorectical grounds, a more interesting question would be to understand the reasons why organisations would want to consolidate such vast quantities of data within a single SAN system.

Surely there are other important considerations such as security, resilience (yes most SANs are dual fabric - but do you not need more if you are putting an entire organisational egg in one basket?) and risk which must be balanced againsts the need to have consolidated access to the entire organisation's storage through a single interconnected SAN?

Re:security, resilience, risk, etc

[statemachine]

Zoning, Virtual SANs (VSANs), and Inter-VSAN Routing (IVR) solve that problem. And why would you need more than a dual fabric setup? Do you like buying expensive HBAs for your servers? Just what kind of hostile environment are you deploying into?

Do tell. I'm curious.

Re:security, resilience, risk, etc

[kefa]

These are all logical methods of isolation and do not enable you to escape the the impact of physical infrastructure changes. Suppose you need to replace your SAN infrastructure to upgrade - do you really want your entire infrastructure to be dependent on a single fabric while you are carrying out your changes? Likewise I would never want to be dependent on only two copies of your data. If you are working on one copy do you really only want to be protected by a single remaining copy. I wouldn't necessarily advocate a SAN system that uses more than 2 fabrics but when your entire organisation sits on a single SAN surely this poses a significant physical risk.

Re:security, resilience, risk, etc

[statemachine]

And to that I ask: How paranoid are you, and how much money do you have?

You also talk about copies of data as if a disk went bad, you'd lose the data. These storage arrays have multiple redundancies (RAIDs of VDisks which are RAIDs themselves) as well as having live replication capability to remote sites -- at which point you likely have a copy (or copies) of an entire datacenter in a different geographic location that is running as a hot spare.

Within a datacenter, you would not have more than dual fabrics. Your fabrics' switches will also be redundantly connected within themselves. And if you're killing an entire fabric with an upgrade, you're doing it wrong.

You'll also have service contracts with lockers of disks, switches, linecards, etc., *on site* with field technicians from the vendors on-call 24/7.

Fibre Channel installations are not like some small company's Ethernet LAN.

Re:security, resilience, risk, etc

[Bandman]

Aside from working in a facility with a pre-existing setup like this, how does one go about educating themselves on how SANs of this magnitude work?

Re:security, resilience, risk, etc

[afidel]

Huh? Unless you have unwisely maxed out your switches before planning the upgrade you would simply interconnect the new physical switches into the fabric through ISL's and then move host over to the new distribution layer switches. You would be vulnerable for the length of time it takes to move a cable from one distribution switch to the new one and only for that single host which is most likely a member of a cluster. As far as the two copies comment, no you would likely have very frequent snapshots within the storage pool at each site which are going to be stored in different cabinets if things are correctly designed. Beyond that many very large companies will have three or more redundant sites, though it gets terribly expensive beyond three from what I have seen and is largely unnecessary as you can get great geographical dispersion with three sites.

Re:security, resilience, risk, etc

[kefa]

This assumes that the new switches are of the same manufacturer and generation. Many SAN 'upgrades' involve changing from one manufacturer to the latest flavour the day (probably as a result of some deal made on the golf course); in this instance you would have to ensure that the SAN is running in 'interoperability' mode which is not generally recommended and can result in reduced SAN functionality - not the sort of state you want your mission critical business to be in.

Also fabric merge and other configuration commands can be momentarily disruptive to a fabric. You would be suprised by the number of organisations that still run single-connect hosts. Obviously here the correct solution is to fix this, but with the best will in the world this often doesn't happen.

As for the copies of data - again vast majority of FTSE 100 companies do not have fully redundant sites. Even a single redundant site is too expensive for most companies - it is cheaper to take the hit on shareholder value (although often the risk that organisations take on a daily basis is not exposed to shareholders) rather than invest in full site redundancy.

Don't get me wrong: most companies have multiple datacentres (e.g. UK1, UK2, etc) that would appear on the surface to provide redundancy, but the quantity of redundant infrastructure in these confusingly titled datacentres is usually vanishingly small. (the banks might be excepted here...)

Re:security, resilience, risk, etc

[kefa]

It's difficult aside from a few public whitepaper produced by each of the appropriate vendors. However these whitepapers are often no more than bragging rights to boast about 'how scalable our solution is' rather than providing any real information.

Re:security, resilience, risk, etc

[statemachine]

Now you're making the argument that these companies need redundancy but are too cheap to spring for it, or the existing topology won't allow it because of the limitations of the design. No one can help that.

Fabric merges and other configuration commands can have their potential disruptions limited by using zoning and VSANs. But no one can help you if you decide to log on and reboot the switch.

And yes I do know about those multi-vendor switch fabrics. While no one vendor seems to follow the standards the same way, these differences are mapped out and are well understood by each vendor. If you're just dumping a new vendor's switch on the fabric without research and a plan, then no one can help that.

It seems the situations you describe are either brought about by unwillingness or being unable to spend the correct amount of money, or just a poorly designed SAN.

Re:security, resilience, risk, etc

[statemachine]

Of course they can easily have more than 2 fabrics. They can have 3, 7, 12 redundant fabrics, if they were that paranoid. I never said it wasn't possible.

But you're just picking and choosing and making up strawmen now. If you have three fabrics, and one goes down because you just botched the upgrade, and the other goes down because a circuit breaker blows, well, now you're back to one fabric and vulnerable again. You're also totally flipping insane if you're worried about this, because the power company can have an extended outage or a maintenance person can short circuit the building with a new power line cut-in for a UPS. So, now you're saying you need three separate power companies, with three separate connections, and three separate UPS systems capable of running the whole building? What's the risk/return ratio on this?

Just as a simplistic example, director switches have dual power supplies. When one fails, it's reduced to running on one. Do you now want three redundant power supplies to guard against the infinitesimal risk (assuming you've designed properly) that the other will fail? You've just upped the cost, which will probably be more than 50%.

If you had read some of my other posts in a parallel thread, you'd see that I mentioned having redundant datacenters in different geographical locations.

You say that SAN design should only be directed by business need, and I've never said it shouldn't. However, your argument is a vendor's dream scenario, and not dictated by sound logic.

Re:security, resilience, risk, etc

[statemachine]

Huh? I never slammed any SAN islands. You posted about redundancy with more than a dual fabric. If you are talking about SAN islands, you need to get your terminology straight. You also need to understand that if you have a SAN island, and there isn't more than one path (like through a dual fabric setup), like you say, you're taking down your entire BU anyway as you have no redundancy.

SAN islands can be dual fabric, as well as single fabric. I'm not sure you're grokking my posts.

Internal Revenue Service

[Dekortage]

A few years ago, I remember reading an article about the IRS (the government tax division) that had seven or eight regional data centers around the U.S. -- each with many petabytes of storage to store current and historical tax data on hundreds of millions of Americans, corporations, etc. I can't find the article now but it seems like *it* should make the list... maybe even top the list.

Discount Web host with scalable SAN

[rjamestaylor]

Recently I found a discount online web hosting company
with an unlikely name that offers a scalable,
distributable SAN, called an HDSAN
(High Density Storage Area Network),
for its customers:

SlumLordHosting.com

Re:Discount Web host with scalable SAN

[solakov]

LOL, you totally got me on this. I was expecting to read something great, and I was surprised by something so great it even defied time itself! Awesome.

Re:Discount Web host with scalable SAN

[rjamestaylor]

one of the best "gotta have been there to get it" sites I've ever seen. It's either pure brilliance or a complete WTF.

Excellently executed (no, I had no hand in it!).

Largest DISCLOSED SANs

[mihalis]

I have no idea how much disk space my firm has, but I did hear an apocryphal tale of installing multiple truckfuls of disks every week pretty much indefinitely (now, of course, older smaller disks are also being removed, but even if it's one for one the service life of enterprise disks means the total is continuously growing). But the firm and the total space can't be disclosed. I'm not trying to make any claims - it could well be smaller than the five mentioned, but the point is nobody knows. I'm sure lots of other firms have very big SANs too.

Re:Largest DISCLOSED SANs

[wlandman]

Is your firm Goldman Sachs as per your resume on your website?

Re:Largest DISCLOSED SANs

[mihalis]

Is your firm Goldman Sachs as per your resume on your website?

Not since they let me go in 1998

Re:Largest DISCLOSED SANs

[mihalis]

know who you are and what firm you're talking about. If you pre-declared your PHP variables or turned off notices in php.ini you'd cut your disk use by 2/3rds.

I've never used PHP personally in my life.

Re:...and why does the article say "Pbytes", "Tbyt

[imsabbel]

Because EVERY SINGLE FUCKING story with "TB" and "GB" causes arguments in the way of "this has to be "...bits", the number is too large for bytes" or vice versa even here.
To avoid missunderstandings, 4 additionals bytes (B) dont seem that much of a price.

5 biggest sans?

[aapold]

• Personally, I would have to go with...
• Taih Kki
• Akebono Tar
• Chiyonofuji Mitsugu
• Kitanoumi Toshimitsu
• and uh... E. Honda

LDS Church

[Anonymous Coward]

I swear the LDS church has 20 Petabytes for geneology information.

What I want to know

[teslatug]

How do they do backups (especially online ones) and restores?

Re:What I want to know

[demi]

I suspect in at least some of these cases, the SAN is not continuous, and they're actually doing backups to the SAN. Part of the storage mentioned will be the online portion of a hierarchical storage manager or similar.

Re:What I want to know

[Phishcast]

There are many ways to do online backups of large amounts of storage. One common way is with storage based snapshots. You quiesce your application (i.e. put your database into hot backup mode), take your logical snapshot, and present that snapshot to a backup server. To the backup server, it looks like you're backing up local disk, but in actuality you're getting a point in time copy of your application/database.

Re:What I want to know

[The Second Horseman]

You can also be doing scheduled replication at the disk array level - from array to array, and back up from the secondary storage. Also, you have dedicated network interfaces for any backup to tape that isn't going exclusively over the storage network. Finally, there are some really massive tape libraries with robot loaders (that move quickly enough to kill anyone inside the unit when it comes on - detecting debris / obstacles becomes an important task). There are tape library systems that can support numbers like 400 drives, and 40,000 to 50,000 tapes.

Re:mod parent up

[j-cloth]

I would, but I'm a grammar nazi and then != than

I think we need to coin a few new terms

[Teilo]

Pr0ntab: A score, equal to the amount of time in tenths of a minute, that elapses from the moment a news article is posted to the first comment relating said article to a person's porn collection or viewing habits.

Pr0ntible: The statistical likelihood that any given article will have a low Pr0ntab score, where 1.0 is the highest score, and 0 the lowest.

Pr0ntabulary: A time sensitive, categorical table of subject matter, where each category is assigned a Pr0ntible, and said table is organized in descending order by Pr0ntible.

Example: On today's Pr0ntabulary, the Storage category ranks near the top.

This site is unbelievable ...

[freaker_TuC]

... Thanks for raping my computer, messing with my eyes and giving me 5 extra braintumors on top looking to that site!

heeeeellllooooo 1990! We're back!

Re:This site is unbelievable ...

[rjamestaylor]

I though it was more "Heaven's Garish", myself.

Mmmmm... Heaven's Garish. I think that's the name of that style web (non-)design I'll use from henceforth.

Re:Pebibytes

[guruevi]

I've never heard of anything expressed in Pebi, Tebi or Gibi nor Mebi. A Petabyte is still 1024 x 1 Terabyte which is 1024 x 1 Gigabyte which is 1024 x 1 Megabyte which is 1024 x 1 Kilobyte which is 1024 x 1 Byte which is 8 bit. As soon as you have a 10-bit based computer, you can express your stuff in *bibytes

Gootube?

[hedkandee]

With all those (crappy bitrate admittedly) videos they must have a fair chunk of data lying around

Call of Cthulhu players know...

[Jim in Buffalo]

Call of Cthulhu players know that your maximum SAN can never be more than 99 minus your Cthulhu Mythos Knowledge score.

Re:...and why does the article say "Pbytes", "Tbyt

[TheSkyIsPurple]

> 4 additionals bytes (B) dont seem that much of a price.

Sorry, had to say it... but are you really sure that "ytes" is beings represented on my machine in bytes? My browser might actually be translating into a unicode representation (or some other storage) that uses at least 2 Bytes per character. =-)

Re:...and why does the article say "Pbytes", "Tbyt

[raddan]

Because everybody around here thinks that means "peanut butter" and "turkey butter", respectively.

I used to work at the NASA facility mentioned....

[jCaT]

Back in 1998, the amount of storage they had was pretty impressive too. It took rooms and rooms to do it, but if I'm remembering this correctly they had about 5 terabytes of disk online and close to 5 petabytes of tape robots. It was a pretty slick automated system- everyone had accounts on a main fileserver, and files that had not been accessed in a certain amount of time were written to tape. If you were to try to grab a file that was on tape, it would fire up the robot, transfer it off the tape, and give it to you- all seamlessly. It was a little slow of course but you never had to do anything- it all appeared as though the files were in your home directory the whole time.

this list is quickly going to be irrelevant

[xcjohn]

As the next round of NSF machines come online, this list will quickly be outdated. The least of these machines will have at least 1PB of *scratch* space. Tape libraries are going to be completely necessary rather quickly. SDSC's 19PB might seem like a lot, but we're constantly expanding that to deal with how quickly users dump data into it.

NSA SAN

[shis-ka-bob]

I guess that the NSA counter would be stored that on the NSA SAN. Hmmmm, NSA is an anagram of SAN. A SAN is an anagram of NASA, the government is everywhere in this. Perhaps all SANS have a back door for the NSA, some sort of _NSAKEY. Hey, _NSAKEY is an anagram of SNAKEY, if you don't think that something that is snakey isn't evil, you have better review the history of Adam and Eve. Maybe all SANs are controlled by the NSA. I think we all need a personal Gauss box (aka tinfoil hat) to prevent the NSA from monitoring our brainwaves from the now ubiquitous SANs that surround us and that share their data with the NSA's master SAN.

In their pockets

[LeeMeador]

I know ... it's not a SAN. But a PB isn't what it used to be.

When a large university graduates (say 8000 people) you have to figure that in that stadium, auditorium, whatever each graduate has 20GB in their pocket between the phone and the music player plus, say, 3 family members with video recorder or cameras between 1 and 10 GB. Let's say it averages 2 GB.

(20 GB + 2 GB * 3) * 8000 graduates = 208 TB

I know its a quick estimate but that's a lot of storage and that's just what they are carrying around in their pockets.

CERN?

[mikelang]

CERN has 8 petabytes on just a CASTOR filesystem... Not to mention all the data on EGEE grid. Poor guys, they only quoted those _inferior_ commercial solutions :-). Forgot that scientists have more guts... And of course: don't you remember that Google has a SAN also?

Re:Pebibytes

[rukkyg]

Try reading the side of a disk drive. 300 GB on a drive is 300,000,000,000 bytes. When you plug it in to your computer you're short quite a few Gibibytes.

FFS, what he described isn't even a SAN

[ToasterMonkey]

Stop bitching about the numbers and go look up fibre channel.