Slashdot Mirror
IBM's High Performance File System
HoosierPeschke writes "BetaNews is running a story about IBM's new file system, General Parallel File System (GPFS). The short and skinny is that the new file system attained a 102 Gigabyte per second transfer rate. The size of the file system is also astonishing at 1.6 petabytes (petabyte == 1,024 terabytes). IBM has up a page with more information and specs on the system.."
Are there open source drivers for this FS that can perhaps be integrated into Linux or the *BSD projects?
Is this stuff available in a fashion where we might see it ported for use on standard x86 hardware? Is it GPL'd? I want this in my living room!
-1 Uncomfortable Truth
Will this mean that you can share storage more easily, maybe. It certainly seems to reduce sharks/ESS into an expensive interface for attaching discs (but there again there just a load of discs with a AIX box or 2 and SSA adapters to conenct the discs anyhow).
:).
Given the managment/maintenance levels of discs wil be more intergrated and distrubutable with this I cant help but think that OS/features and the trend in (and rightly so) resiliance,easy and sharing resources approach towards what Plan 9 was setout to be.
The more we move on the more we seem to get towards the lego-type appraoch to IT were you can just buy another box of bricks and add on and keep your older bricks instead of throwing the whole lot out and/or hacksawing the end of a brick of and gluing it onto the side of....
Storage wise this is a nice step forwards and having worked on AIX and its many filesystems and managment tools and the ease of getting the job done with the option to get clever if you wish (you chose and not forced) this looks funky albeit its RAID for SAN's in a way.
What I realy want is a FS that will propergate automaticaly and resiliantly in a way that accomodates network diversaty already and I still come down to me wanting, what is all intent a filesystem sat on a database sat on a p2p network, alas atm performance would suck, least today but you know how long code takes to get right and how fast hardware moves - remember alot of code in windows XP has origins to when it was written on a humble 386 cpu if not lower.
What this does show is how netowrk/storage interfaces have moved forward and I/O requests dont hammer CPU's as much as they used to, getting there
they have 104 servers... that's almost 1GB/s/server ... that's a lot.. and they have 4 raid controlers per server.. that means each raid controler does around 250 mb/s.. (which normal for a high end raid controler) and they are connected with a 10gb/s interconnect (probably infiniband or 10G ethernet). So the whole thing is not that hard to do if you use your servers properly.
But they have 1000 clients.. so its only 100MB/s/client.. so 1Gbps/s/client.. so the clients are probably gigabit ethernet... Otherwise they could do much more... I've seen other cluster file systems do 600MB/s/client, its not that impressive. It only shows that IBM has a huge budget and they can afford lots of hardware.
This is like saying, NASA builds huge rocket for many many billions of dollars. Its just of matter of cash, not of great technical prowness.
If we do a dollar count.. lets say 10k$/server * 104 = 1M$ + 25k$/storage controler w/ disks * 416 = 10M$ + 2k$/client * 1000 = 2M$, plus the switches etc... give me 30M$ and I can do the same thing.
Btw, I work on a cluster filesystem, that performance is not that hard to achieve if you have that kind of hardware.
We used GPFS in our production environment for about 9 months in 2004/2005. We chose it specifically because it allowed several machines to share the file system (like NFS) but with file locking. It was also supposed to be very fault tolerant with no single point of failure. We set it up using a fiberchannel SAN.
Unfortunately we had a lot of problems with it. For one, performance was quite bad in ceratin cases... doing an ls in a large directory would take a very long time. Doing finds would take a very long time. Once you had a specific file you wanted, opening and reading it was reasonable (though all disk ops were still on the slow side), but multi file operations lagged on the level of 10s of seconds or more. I think it was having to issue network checks to every machine in the set for each file or something.
Also, the CPU usage was very high across all our machines, primarly from lock manager communications. It really taxed the system. And perhaps worst of all, it would caused crashes sometimes. A single machine in the set would die (usually a GPFS assert), and though that didn't break the set permanently, a multi-minute freeze on all disk reads would take place until the set determined the machine was unavailable. We spoke with IBM about all this stuff... provided debugging output and everything, we used the latest patches. But we never got the issues resolved. It was a very rough few months indeed. I probably averaged 4 hours sleep per night.
When I say "slow" what am I comparing it to? In the end we switched to NFS and we came up with a somewhat clever way to avoid the need for file locking. NFS used the same SAN hardware, but had a single point of failure: the head server. We doubled up there with warm failover. The load on all servers dropped dramatically (I'm talking from ~40 load to ~.1 load). Disk operations were orders of magnitude faster. And we've not had a single NFS related lockup or failure in the past year and a half *knocks on wood*.
Anyways -- GPFS probably has some good uses. But I would not recommend it for a very high-volume (lots of files, lots of traffic) mission critical situation. Unless they've made some major improvements.
Cheers.