It's not like Solaris with ZFS is any different in this regard. To use snapshotting or any other zfs feature on a physical disk, you have to add the raw block device to the zfs pool. Just like you have to add a physical disk to LVM before you can use LVM snapshots on Linux. You can't just snapshot arbitrary block devices on either OS, holistic layering violations or not.
I agree with just about everything you say. CSP + HVDC seems to be one of the few renewable energy ideas that have the potential to be scaled up to the truly massive scale that would be needed if we ever want to get rid of fossil fuels, be it for global warming, resource depletion (peak oil etc.), "national security", or whatever reasons.
As for the current political favorite, biofuels, I fear that its a disaster waiting to happen. Competition for arable land with food, poor efficiency (photosynthesis ~0.1% vs. ~30% for CSP, more than two orders of magnitude!), soil erosion, aquifer depletion, you name it. From an US perspective, http://energybulletin.net/28610.html sums it up.
Concentrating solar power works comparatively better in areas with little cloud cover, since they are entirely dependent on direct radiation, vs. normal solar cells which at least get some output from diffuse light.
High-availability solutions don't have to be complicated and expensive. Starfish is the perfect example of such a simple and low-cost system. In fact, it is THE reason we wrote Starfish: To provide an in-expensive, fault-tolerant, highly available clustered storage platform that works from the smallest website to the largest storage network. We've based the technology on the assumption that having expensive hardware/software is the wrong way to go about solving the problem.
Oh, I absolutely agree. But I wasn't advocating any gold-plated SAN solution either. A standard rackmount server with hotswappable fans and power supplies is a few thousand, the daisy-chainable external enclosures (again, with redundant power supplies and fans) likewise. With 750 GB SATA drives about 1/2 of the hardware budget goes to the drives, which is about the same as you get with smaller nodes that you advocate. Some study (IBM, IIRC) showed that about 80 % of server failures are due to failing power supplies, fans, or hd:s. With all of these redundant, even a single server can be quite reliable. And if that isn't enough, you can mirror it to another box using DRBD or GNBD. And the software is mature, which unfortunately can't be said of most parallel FS:s (case in point, our $zillion cray is down at the moment due to Lustre problems), and free. Your website says Starfish is $12000 per 10 TB, which is more than the hardware itself..
Of course, at some point a bunch of big mirrored servers and playing with automount becomes pretty tedious to maintain.
I guess my main point is that with current modestly priced gear (standard rackmount servers and external enclosures), you can go to pretty big systems before a "real" parallel FS becomes necessary. I guess in many cases the limiting factor will be the network BW, 2 GbE (most rackmount servers come with two gigabit ethernet interfaces, so you can bond them) is not that much for ~150 TB storage. 10 GbE would help, of course.
However, with a stable, mature, free and high performance parallel FS the balance would shift to much less storage per server (perhaps even using just internal storage). I'm just not convinced such a thing exists yet, however much I'd like to see it.
As others have mentioned, HA solutions are complicated and expensive. Unless you really need it, you probably don't want to go down that route.
But with 2 TB currently and scaling to perhaps a few hundred TB in the future, the obvious simple solution is to just buy bigger servers. With modern gear you can really connect a frightening amount of storage to a single server at modest cost. Say a rackmount box with space for 12 drives, then SAS card(s) with external connector(s), so you can chain together multiple enclosures. Taking Dell as an example (just what I quickly found with google (http://www.dell.com/downloads/global/power/ps3q06 -20050311-Kadam-OE.pdf), I'm sure other vendors sell similar gear), you can daisy-chain 3 enclosures per connector, and a SAS card has 2 connectors. With 2 cards per server (about what you can fit in a 2U box?) you then get 12 external enclosures of 15 drives each, for a total of 192 drives. With 750 GB SATA drives you then have 144 TB raw storage per server.
When needs grow beyond one server, clever use of automount maps lets you manage the namespace for multiple servers easier than doing it all by hand.
As for Lustre, it's really a specialized solution for HPC, made for multiple compute nodes striping to the storage nodes at full speed using a collective IO API like MPI-IO.
For utility scale systems they seems to be more cost efficient than big arrays of solar cells. The downside is that they require direct solar radiation so they are very inefficient on a cloudy day.
The track record of non-shared memory supercomputers is terrible. There's a long history of dead ends, from the ILLIAC IV to the BBN Butterfly to the NCube to the Connection Machine.
The vast majority of supercomputers today have distributed memory, from basic PC clusters to the highest end stuff available such as IBM blue gene or Cray XT4.
At some point, you have to go to non-shared memory, but that doesn't have to happen until you hit maybe 16 CPUs sharing a few gigabytes of memory, which is about when the cache interconnects start to choke and speed of light lag to the far side of the RAM starts to hurt. That might even be pushed harder; there's been talk of 80 CPUs in a shared memory configuration. That's optimistic. But we know 16 will work; SGI had that years ago.
Current SGI gear supports a maximum of 512 sockets, i.e. 2048 threads, all cache coherent.
That being said, considering all the problems with shared state programming, I think the future belongs to some form of message passing, e.g. actor model, CSP, join calculus etc. Those can be implemented without CC.
much more like what Mozilla has done with Firefox and Thunderbird; spinning one huge piece of bloat into several smaller tools that do their job effectively.
On my Linux box, firefox + thunderbird at the moment use about 62+36 ~ 100 MB memory (RSS), about twice as much as mozilla ever used. On windows the situation is a little better but not much, ff+tb still use about 2x as much memory as seamonkey with browser and mail windows open.
The claim that ff & tb are somehow less bloated than mozilla/seamonkey is a myth. Though seamonkey still looks like netscape 4.x, i.e. shit, so at least ff & tb have done something right.
Cray is actually just about to launch a next generation MTA, the XMT. Interestingly, the processors plug into Opteron sockets and except for the processors themselves all the other hw is from the XT4 (Cray's Opteron-based supercomputer).
Have you tried bacula? I've heard stories of people migrating from amanda to it, although probably less so these days now that amanda supports spanning many tapes.
And my pet peeve, neither amanda, bacula or any commercial program I know of supports extended attributes (ACL:s, SELinux labels). #"@%&
Legacy 2D X drivers (EXA, XAA) can only provide 2D acceleration. OpenGL 3d drivers can provide 2D AND 3D acceleration. OpenGL 3d drivers can provide faster 2D acceleration then what the legacy 2D drivers can do. (due to the nature of the hardware GPU, not so much the drivers)
What about those cards for which there is no open source hw accelerated OpenGL driver? How fast is 2d rendered via the Mesa software renderer?
AFAICS the WD5000YS tested by StorageReview, has a 5 year warranty, it's certified for RAID use, and has a 1.2 million hour MTBF at 100% duty cycle. And at least over here, it sells for the same price as the ordinary one, so IMHO the choice is pretty simple.
Since I run simulations that routinely take two months, that 30% is a big difference.
So do I, but us HPC users are few and far between. There are many situations, even where Fortran is used, where a 30 % worse performance than the top of the line commercial compilers is good enough.
The only advantage of current Fortran over C is that the vector processing unit of modern CPUs is better utilised, thanks to Fortran semantics.
The reason why Fortran often vectorizes better is not that Fortran has array ops (in reality, modern Fortran compilers actually scalarize array expressions before handing the code over to the middle end tree optimizers), but due to different aliasing rules than C. This was the main reason why Fortran 77 (with no array ops) often was an order of magnitude or so faster than C on old vector supercomputers.
Sure C can get the same kind of thing with const and restrict, but googling reveals that ironically one of the very few places where it's actually used is in the runtime library of the GNU Fortran compiler.
So, COMMON blocks have been removed from the language?
No, they're still there although "modern Fortran" programming style doesn't use them.
But more to the point, what are you babbling about? You were never allowed to access a variable both via a procedure argument and via a common block in the same procedure.
Compare the efficiency of GCC at auto-vectorising FORTRAN (which has a primitive vector type) and C (which doesn't), if you don't believe me.
Typically this is due to aliasing, not because "Fortran has an primitive vector type and C hasn't". For that matter, the GCC Fortran frontend scalarizes array expression (i.e. converts them into equivalent loops) before handing them over to the middle end.
Had they left the Novell Linux Desktop name and replaced the SUSE Linux Enterprise Server with Novell Linux Server or Novell Linux Enterprise Server, wouldn't they have been able to distinguish the community versions against the enterprise versions much easier?
Yes, but then they would have thrown out the brand SUSE had built around their enterprise versions.
Most people probably never knew what "Novell Linux Desktop" was.
Seems my understanding of modern piracy is rather different.
These days, pirates operate with small speedboats. As they can't load much stuff into those boats, they take over the entire ship instead, paint some new name on it, sail it to some port that isn't too picky about documents and procedures, sell the cargo and perhaps the ship too.
If you look at the hulls of big sailing vessels they don't have keels or anything like that, and their draft is about the same as a machine powered vessel of the same size. The point is that when loaded with cargo, the center of gravity was low enough to prevent capsizing and the draft of the loaded hull was big enough that the hull itself provided enough area to prevent excessive skewing. That's a rather different scenario compared to a modern yacht, which is mostly empty inside and has a very different hull shape.
When they were not carrying cargo they were quite heavily ballasted.
And yes, they were able to sail upwind, though of course with square rigging that is pretty tedious compared to the bermuda rigging you see on modern yachts.
Slashdot Mirror
It's not like Solaris with ZFS is any different in this regard. To use snapshotting or any other zfs feature on a physical disk, you have to add the raw block device to the zfs pool. Just like you have to add a physical disk to LVM before you can use LVM snapshots on Linux. You can't just snapshot arbitrary block devices on either OS, holistic layering violations or not.
alias yum='yum -t -y'
Problem solved.
I agree with just about everything you say. CSP + HVDC seems to be one of the few renewable energy ideas that have the potential to be scaled up to the truly massive scale that would be needed if we ever want to get rid of fossil fuels, be it for global warming, resource depletion (peak oil etc.), "national security", or whatever reasons.
Here's an interesting large scale csp+hvdc study:
http://www.trecers.net/
http://www.trec-uk.org.uk/reports.htm
As for the current political favorite, biofuels, I fear that its a disaster waiting to happen. Competition for arable land with food, poor efficiency (photosynthesis ~0.1% vs. ~30% for CSP, more than two orders of magnitude!), soil erosion, aquifer depletion, you name it. From an US perspective, http://energybulletin.net/28610.html sums it up.
Concentrating solar power works comparatively better in areas with little cloud cover, since they are entirely dependent on direct radiation, vs. normal solar cells which at least get some output from diffuse light.
Sure.
LVM snapshots are on a per logical volume basis, so you can just mount the snapshot partition on another mountpoint and copy files from there.
High-availability solutions don't have to be complicated and expensive. Starfish is the perfect example of such a simple and low-cost system. In fact, it is THE reason we wrote Starfish: To provide an in-expensive, fault-tolerant, highly available clustered storage platform that works from the smallest website to the largest storage network. We've based the technology on the assumption that having expensive hardware/software is the wrong way to go about solving the problem.
Oh, I absolutely agree. But I wasn't advocating any gold-plated SAN solution either. A standard rackmount server with hotswappable fans and power supplies is a few thousand, the daisy-chainable external enclosures (again, with redundant power supplies and fans) likewise. With 750 GB SATA drives about 1/2 of the hardware budget goes to the drives, which is about the same as you get with smaller nodes that you advocate. Some study (IBM, IIRC) showed that about 80 % of server failures are due to failing power supplies, fans, or hd:s. With all of these redundant, even a single server can be quite reliable. And if that isn't enough, you can mirror it to another box using DRBD or GNBD. And the software is mature, which unfortunately can't be said of most parallel FS:s (case in point, our $zillion cray is down at the moment due to Lustre problems), and free. Your website says Starfish is $12000 per 10 TB, which is more than the hardware itself..
Of course, at some point a bunch of big mirrored servers and playing with automount becomes pretty tedious to maintain.
I guess my main point is that with current modestly priced gear (standard rackmount servers and external enclosures), you can go to pretty big systems before a "real" parallel FS becomes necessary. I guess in many cases the limiting factor will be the network BW, 2 GbE (most rackmount servers come with two gigabit ethernet interfaces, so you can bond them) is not that much for ~150 TB storage. 10 GbE would help, of course.
However, with a stable, mature, free and high performance parallel FS the balance would shift to much less storage per server (perhaps even using just internal storage). I'm just not convinced such a thing exists yet, however much I'd like to see it.
As others have mentioned, HA solutions are complicated and expensive. Unless you really need it, you probably don't want to go down that route.
6 -20050311-Kadam-OE.pdf), I'm sure other vendors sell similar gear), you can daisy-chain 3 enclosures per connector, and a SAS card has 2 connectors. With 2 cards per server (about what you can fit in a 2U box?) you then get 12 external enclosures of 15 drives each, for a total of 192 drives. With 750 GB SATA drives you then have 144 TB raw storage per server.
But with 2 TB currently and scaling to perhaps a few hundred TB in the future, the obvious simple solution is to just buy bigger servers. With modern gear you can really connect a frightening amount of storage to a single server at modest cost. Say a rackmount box with space for 12 drives, then SAS card(s) with external connector(s), so you can chain together multiple enclosures. Taking Dell as an example (just what I quickly found with google (http://www.dell.com/downloads/global/power/ps3q0
When needs grow beyond one server, clever use of automount maps lets you manage the namespace for multiple servers easier than doing it all by hand.
As for Lustre, it's really a specialized solution for HPC, made for multiple compute nodes striping to the storage nodes at full speed using a collective IO API like MPI-IO.
This is called concentrating solar power (CSP). See e.g. http://en.wikipedia.org/wiki/Solar_thermal_energy
For utility scale systems they seems to be more cost efficient than big arrays of solar cells. The downside is that they require direct solar radiation so they are very inefficient on a cloudy day.
The track record of non-shared memory supercomputers is terrible. There's a long history of dead ends, from the ILLIAC IV to the BBN Butterfly to the NCube to the Connection Machine.
The vast majority of supercomputers today have distributed memory, from basic PC clusters to the highest end stuff available such as IBM blue gene or Cray XT4.
At some point, you have to go to non-shared memory, but that doesn't have to happen until you hit maybe 16 CPUs sharing a few gigabytes of memory, which is about when the cache interconnects start to choke and speed of light lag to the far side of the RAM starts to hurt. That might even be pushed harder; there's been talk of 80 CPUs in a shared memory configuration. That's optimistic. But we know 16 will work; SGI had that years ago.
Current SGI gear supports a maximum of 512 sockets, i.e. 2048 threads, all cache coherent.
That being said, considering all the problems with shared state programming, I think the future belongs to some form of message passing, e.g. actor model, CSP, join calculus etc. Those can be implemented without CC.
Actually, Erlang is an implementation of the "Actor model", somewhat different from CSP.
much more like what Mozilla has done with Firefox and Thunderbird; spinning one huge piece of bloat into several smaller tools that do their job effectively.
On my Linux box, firefox + thunderbird at the moment use about 62+36 ~ 100 MB memory (RSS), about twice as much as mozilla ever used. On windows the situation is a little better but not much, ff+tb still use about 2x as much memory as seamonkey with browser and mail windows open.
The claim that ff & tb are somehow less bloated than mozilla/seamonkey is a myth. Though seamonkey still looks like netscape 4.x, i.e. shit, so at least ff & tb have done something right.
Cray is actually just about to launch a next generation MTA, the XMT. Interestingly, the processors plug into Opteron sockets and except for the processors themselves all the other hw is from the XT4 (Cray's Opteron-based supercomputer).
Good post.
Have you tried bacula? I've heard stories of people migrating from amanda to it, although probably less so these days now that
amanda supports spanning many tapes.
And my pet peeve, neither amanda, bacula or any commercial program I know of supports extended attributes (ACL:s, SELinux labels). #"@%&
Actually, the 9250 is the fasted fully supported ATI card under Linux.
In real life, the 8500 is faster since the 9200/9250 are cut down budget versions of the 8500.
Yes, they will save on resistance losses by about 1/3. (120/380)
(1/3)**2 = 1/9 actually.
Legacy 2D X drivers (EXA, XAA) can only provide 2D acceleration.
OpenGL 3d drivers can provide 2D AND 3D acceleration.
OpenGL 3d drivers can provide faster 2D acceleration then what the legacy 2D drivers can do. (due to the nature of the hardware GPU, not so much the drivers)
What about those cards for which there is no open source hw accelerated OpenGL driver? How fast is 2d rendered via the Mesa software renderer?
AFAICS the WD5000YS tested by StorageReview, has a 5 year warranty, it's certified for RAID use, and has a 1.2 million hour MTBF at 100% duty cycle. And at least over here, it sells for the same price as the ordinary one, so IMHO the choice is pretty simple.
Since I run simulations that routinely take two months, that 30% is a big difference.
So do I, but us HPC users are few and far between. There are many situations, even where Fortran is used, where a 30 % worse performance than the top of the line commercial compilers is good enough.
The only advantage of current Fortran over C is that the vector processing unit of modern CPUs is better utilised, thanks to Fortran semantics.
The reason why Fortran often vectorizes better is not that Fortran has array ops (in reality, modern Fortran compilers actually scalarize array expressions before handing the code over to the middle end tree optimizers), but due to different aliasing rules than C. This was the main reason why Fortran 77 (with no array ops) often was an order of magnitude or so faster than C on old vector supercomputers.
Sure C can get the same kind of thing with const and restrict, but googling reveals that ironically one of the very few places where it's actually used is in the runtime library of the GNU Fortran compiler.
So, COMMON blocks have been removed from the language?
No, they're still there although "modern Fortran" programming style doesn't use them.
But more to the point, what are you babbling about? You were never allowed to access a variable both via a procedure argument and via a common block in the same procedure.
how slow your G77/GFortran programs are; they're great tools, but they're primarily Fortran Lints
The Polyhedron Fortran benchmarks puts gfortran at about 30 % slower than the fastest compilers (ifort or pathf90 depending on platform).
Compare the efficiency of GCC at auto-vectorising FORTRAN (which has a primitive vector type) and C (which doesn't), if you don't believe me.
Typically this is due to aliasing, not because "Fortran has an primitive vector type and C hasn't". For that matter, the GCC Fortran frontend scalarizes array expression (i.e. converts them into equivalent loops) before handing them over to the middle end.
Had they left the Novell Linux Desktop name and replaced the SUSE Linux Enterprise Server with Novell Linux Server or Novell Linux Enterprise Server, wouldn't they have been able to distinguish the community versions against the enterprise versions much easier?
Yes, but then they would have thrown out the brand SUSE had built around their enterprise versions.
Most people probably never knew what "Novell Linux Desktop" was.
Seems my understanding of modern piracy is rather different.
These days, pirates operate with small speedboats. As they can't load much stuff into those boats, they take over the entire ship instead, paint some new name on it, sail it to some port that isn't too picky about documents and procedures, sell the cargo and perhaps the ship too.
If you look at the hulls of big sailing vessels they don't have keels or anything like that, and their draft is about the same as a machine powered vessel of the same size. The point is that when loaded with cargo, the center of gravity was low enough to prevent capsizing and the draft of the loaded hull was big enough that the hull itself provided enough area to prevent excessive skewing. That's a rather different scenario compared to a modern yacht, which is mostly empty inside and has a very different hull shape.
When they were not carrying cargo they were quite heavily ballasted.
And yes, they were able to sail upwind, though of course with square rigging that is pretty tedious compared to the bermuda rigging you see on modern yachts.