This year's Supercomputing 2003 Bandwidth Challenge netted some cool results in this area including 23+gbps inside the US, 7.56gbps between the US and Japan, and 8.96gbps throughput to a remote network filesystem using GPFS. There are even some pretty graphs. My former co-workers at SDSC were involved in a lot of this work.
Actually, as it says on the page the one we built has a GBE card. I didn't include it in the "base" description because there are significant applications for this server where 10MB/s is sufficient.
During one long afternoon when we were fighting with the kernel on the server, we came up with another cool application:
How does adding 1tb of disk to a TiVo sound?
It wouldn't be easy - you'd have to go the "do-it-yourself-ethernet" route and find some way to get NFS to work, and even then it might not be fast enough. Sure would be cool to have a few thousand TV shows, though.:)
These discussions actually came up while we were building this thing. We have an IBM HPSS for these purposes but I suspect that the average small business or academic department doesn't have one lying around.
1tb AIT2 libraries are going for about US$7000, making them substantially more expensive than the server per gigabyte, but several posters correctly point out that there is a difference between online and offline backups. There is more than one kind of failure - you could have boxes like these in ten different cities and just 'rsync' them every night, but if that new sysadmin you just hired deletes the master copy, all of the other copies will get deleted that night. Archival storage *is* an important consideration, and it's one that gets even more difficult to address if you build it in the 4tb configuration.
We're looking at this in much the same way. My original thought was that a system like this would be most useful for research that needed lots of workspace without the massive cost of central NFS or SAN disk. It's quite likely that many of these uses would not even require NFS, but would instead be an application running locally on the machine with the disk.
The phrase we decided to use was "medium availablity". We have an Auspex for our critical filesystems - it's big, it's expensive, and it never breaks. But there are applications where cost and size are more important than 99.999% uptime.
- Tom G.
I put the page on staff.sdsc.edu, which is normally only used by a small number of people to get info that's not of interest to anyone outside of SDSC. Since it's low traffic, it does DNS lookups for each connection.
Had I known how much interest there would be, I would have put it on another server. In any case, the server is no longer doing lookups and all is now well.:)
Slashdot Mirror
This year's Supercomputing 2003 Bandwidth Challenge netted some cool results in this area including 23+gbps inside the US, 7.56gbps between the US and Japan, and 8.96gbps throughput to a remote network filesystem using GPFS. There are even some pretty graphs. My former co-workers at SDSC were involved in a lot of this work.
Actually, as it says on the page the one we built has a GBE card. I didn't include it in the "base" description because there are significant applications for this server where 10MB/s is sufficient.
During one long afternoon when we were fighting with the kernel on the server, we came up with another cool application: How does adding 1tb of disk to a TiVo sound? It wouldn't be easy - you'd have to go the "do-it-yourself-ethernet" route and find some way to get NFS to work, and even then it might not be fast enough. Sure would be cool to have a few thousand TV shows, though. :)
These discussions actually came up while we were building this thing. We have an IBM HPSS for these purposes but I suspect that the average small business or academic department doesn't have one lying around. 1tb AIT2 libraries are going for about US$7000, making them substantially more expensive than the server per gigabyte, but several posters correctly point out that there is a difference between online and offline backups. There is more than one kind of failure - you could have boxes like these in ten different cities and just 'rsync' them every night, but if that new sysadmin you just hired deletes the master copy, all of the other copies will get deleted that night. Archival storage *is* an important consideration, and it's one that gets even more difficult to address if you build it in the 4tb configuration.
We're looking at this in much the same way. My original thought was that a system like this would be most useful for research that needed lots of workspace without the massive cost of central NFS or SAN disk. It's quite likely that many of these uses would not even require NFS, but would instead be an application running locally on the machine with the disk. The phrase we decided to use was "medium availablity". We have an Auspex for our critical filesystems - it's big, it's expensive, and it never breaks. But there are applications where cost and size are more important than 99.999% uptime. - Tom G.
My fault.
I put the page on staff.sdsc.edu, which is normally only used by a small number of people to get info that's not of interest to anyone outside of SDSC. Since it's low traffic, it does DNS lookups for each connection.
Had I known how much interest there would be, I would have put it on another server. In any case, the server is no longer doing lookups and all is now well.
- Tom G.