what you buy depends mostly on how much you want to spend on your interconnect, which in turn depends on your applications. You can spend >50% of your cash on the interconnect - but do you need to?
are your apps parameter study serial jobs? (interconnect doesn't matter much - just use gigE)
already written MPI apps? (few large messages? many small?)
OpenMP only? (you need large SMP nodes)
do they need large bandwith or low latency or both?
Infiniband or gigabit ethernet are your main options. IB is low latency, and probably even more cost effective then gigE, but you may need the gigE anyway for a maintainance network (netbooting, NFS etc.). gigE usually comes with the motherboard, but you still need to budget for a fat tree of switches to connect it all. Myrinet's too pricy and (I think) slower then IB, but might be simpler to connect and has more mature MPI implementations for it.
Watch out for big vendor cluster software people - they may not actually know what they're doing.... not naming any names. What big vendor actually did (for the cluster next door to ours) was make it all slower!
IMHO you don't need that serial maintainance network crap they try to sell you, or even IPMI or similar. these Xeon/P4/Athlon64/Opteron clusters should be reliable enough that it's a waste of money. Our 264 node (528 Xeon) machine is fine without it.
If you want real bang for your buck then avoid the large expensive gigabit ethernet switches - they usually have limited backplane bandwith anyway. We use 2D mesh networking made from a stack of 24port gigE switches and had the fastest machine in Canada for a while... our networking is now way simpler than the hypercube-like topology on that page, but every node is still a router, and it works really well.
OSCAR is a great install system for a cluster. Do it yourself - it's the only way you'll ever be able to maintain the machine in the long term anyway... Just buy the hardware from anyone who gives you the best deal and looks like they'll be around for 5 years to replace nodes as they die.
Drop us a line if you want more dodgy advice:-)
Re:About scientific computing and fortran...
on
GSL 1.0 Released
·
· Score: 1
...
especially since high quality stable fortran 95 compilers are available for most platforms (but no free ones:( ).
Doesn't Intel's free (as in beer) ifc compiler do f95 (and OpenMP)?
Trimaran NOT a Catamaran - can you count to 3?
on
High-Tech Hydrofoil
·
· Score: 2
Yellow Pages Endeavour is a planing vehicle which is VERY different to a hydroplane based craft. One skims the water and the other lifts as much as possible entirely out of it. Wetted surface area on both is TINY, and so both are theoretically fast, but the hydroplane has got to be able to handle at least some waves whereas YPE would no doubt disintegrate. YPE ran for (and got) the speed record in a depth of <1m of water after all. So under real conditions Hydropetre is more promising - especially for the 24hr distance record which it looks like they're going for.
Big cats and tris are WILD, so one that's largely out of the water(!) is definitely the regime of those French nutter who are brave enough to push these things hard (and Grant Dalton of course:)
The cats in The Race were routinely at 35 knots so I don't think 45 is out of the question for a radical craft. This has been under development for a long time after all.
Hey, didn't many of the Formula40 cats and tris have some sort of hydroplane appendages? They were way cool... Are there pics of them anywhere? I only have one Becken calendar...:-/
What sorts of applications are you running? I've seen a MUCH bigger difference
for multithreaded applications (on the order of 25% or more)
I'm running C + MPI scientific codes. The MPI (lam mostly) calls map through to shared mem operations. My codes aren't super-heavy on the communications though - maybe 5% or 10% of the time is in comms in a mix of many small operations and some big bandwidth limited ops - so yeah - it's possible that an app which used GB/s of bandwidth between CPUs would hit more limitations in the Linux SMP implementation compared to Tru64.
As always it's a good idea to try your specific app and different OS's before buying the machine.
To avoid the troll calling: Linux and *BSD are good on x86 but in SMP they don't
compare to Alpha's official OS.
On my measurements they do compare. It's about a
5% hit for Linux vs. Tru64, and another 5% hit for the Compaq Tru64 compiler being better integrated into the OS than the Compaq Linux/Alpha compilers. So that's about 10% slower overall which isn't bad at all... It's true that gcc/egcs and it's math library is pretty lame on Alpha compared to Compaq compilers, but the Compaq compilers are free-ish so it's not a problem.
This is running MPI codes on 4-way Alpha/ev67/667 es40's with 8G mem and 2.2.16 Linux - so 2.4 Linux would be better - leaving only about a 5% difference between Linux and Tru64. Most of that is in page colouring (people guess) which Linux doesn't support.
The big thing that the Linux/Alpha version of the Compaq compilers doesn't support is OpenMP style SMP programming - so you'd have to do this by hand with threads or MPI or buy a different compiler.
My advice is to go with SMP Athlons as these are
pretty much a match for Alpha in Floating Point these days - primarily 'cos Alpha MHz's haven't grown much for years. Athlons are also way cheaper so you can buy many more of them:-)
AFAIK Alphas haven't used any sort of an L3 since 21164 (ev56's). ev6 and ev7 have only L1 and L2 - just like Pentium2/3 etc. The Linux boot message on our es40's (4-proc Alpha) using Tsunami/Clipper chipsets mentions nothing of L3.
... and before anyone says that a stack'o'smaller boxes won't work - the guy said that his CFD app wouldn't run on linux, NOT that it required a large shared mem machine. Hence distributed mem systems may be an option. Even a pile of es40 Alphas would give you better price/performance than a single big shared mem box... not anywhere near the price/performance of linux/athlon, but hey.
Ahem, if you'd done your benchmarks then you would know that Linux/x86 ARE better. Certainly from a price/performance point of view, and ALMOST from an ultimate performance point of view as well.
This assumes a) fast Athlons (not necessarily duals) b) distributed memory (aka MPI) apps are ok.
Linux and gcc aren't superior to proprietary OS's (such as IRIX, Tru64 on SGI and Compaq/Alpha hardware) or their excellent C compilers, but it doesn't matter 'cos MIPS, Alpha, SPARC and HP MHz just haven't kept up with the x86 crowd.
Of course if you need large shared mem boxes then SGI and Compaq et al are your only choice, and boy do you pay for it...
My Astro/CFD benchmarks show that Alpha is WAAAY faster than SGI. In fact even my sub $2k Athlon 700 home linux machine is heaps quicker than a SGI/R12k/300/8M cpu. I don't expect that HP is that much better.
Hence get your CFD vendor to port their code to Linux/x86 and buy yourself a stack of Athlon (thunderbird) 950's. These are nearly as quick as the fastest Alphas (ev67/667/8M)... For the money you'll get about 4 times the performance out of your purchase if you can go with Athlons... If your CFD vendor won't port then find another vendor!
If you have to go with the expensive hardware then look on the SPEC website (www.spec.org) to see how they compare - SPEC is essentially biased towards the scientific and CFD apps you are running. Look for details on each part of SPEC to see which is closest. Also look at Compaq gs320 hardware - it might be the fastest of the 'big iron'
If Athlons came in duals or quads they'd be the only choice. We're still forced to look at Intels (which are about 30% slower per MHz and cost more than Athlons) just 'cos they come in duals:-/
And once again, try your app on the real hardware before handing over that sort of cash.
yes, temperature gradients are huge, and there are vibrations etc. But the basic point is that glass is NOT a good material for strength. Use the absolute minimum of (heavy) glass and get all the strength and stiffness from carbon fibre or a similar material that is designed for strength and light weight. Hence a thin mirror bonded to a stiff and strong backplate, NOT a thick mirror.
If you think differential expansion is a problem then deliberately put grooves in the mirror (like the gaps in concrete pavements) to let mirror segments move apart. As long as the stiff backplate remains parabolic you have no problems.
It just seems like a correct material for the job issue to me... ???
BTW, mirrors on earth ARE limited by their deformability under gravity simply because they rotate and point at different angles to gravity, and so deform in a complicated way. Hence people now avoid single large mirrors and use segmented mirrors like Keck.
... The
article doesn't mention what they would do to handle the stresses and strains that the
very thin mirror would undergo. A glass mirror that thin is going to flop all over the
place unless you can control the mirror shape,...
And exactly what is going to make it flop around?? This is in Space remember - ie. vacuum, no wind, no gravity,... nothing to make the mirror deform apart from temperature gradients and the telescope's own tiny pointing movements. The mirror has only got to maintain its own shape which can be accurately set before launch/install.
The main reason big telescope mirrors are hard to build (apart from lack of money!:) is that they deform under their own weight - no such problemo up there dude.
If you're using tcsh then make sure autologout is unset. What might be happening is that the tcsh session from which you're firing up your remote xterms is logging out, and thus leaving ssh nowhere to connect to. So it hangs. The 300s thing sounds a lot like this. All my rxvt sessions stay up for days on end over ssh and have never died unless my PPPoE IP dynamically changes.
Oh, I guess that's another thing - how often does DHCP/pump(?) change your IP address? If that changes regularly (which it's allowed to do) then ssh would hang...
Where is the simap (secure SSL imap) support for pine though? Netscape has it... and it's dead easy to implement at the sever end with SSLeay or OpenSSL. The text-based clients are lacking though!:-/
Rather than speed up the OS loading, get rid of DOS BIOS and save yourself time that way!
The Linux BIOS Project was in this/. article just a week ago. It's not ready for prime time yet, but does all the BIOS step in about 0.1 seconds! Combine this with the right Linux distribution (debian has a fast (binary) service startup program - do others?), and a carful choice of which services you run, and Linux would boot REALLY fast!:)
At lest one of clueless French S.O.B.'s who had got caught ('cos of their own blatant arrogant stupidity) for that NZ lark got sent (eventually) to a french tourist island to serve her prison term and then (big surprise) got sent home early. Do you call that punsihment??? You have no clue about what Greenpeace try to do and no respect for human life.
I don't always agree with their media stunt methods (nor their junk mail) but they do more good than a dozen secret services of any nations you'd care to mention.
Does this sort of community remind anyone else of "Shockwave Rider" or is it just me?
Anything so far removed from reality that actually needs to create something so artificial as Disneyville practicaly defines a civilisation that is doomed. Good luck USA, you'll need it.
I could competently admin Irix for years, had programmed about a squintillion lines of basic (sad but true - everyone learns somewhere), C, Fortrash, C++, Java,... written web sites, blah blah blah... All before I'd ever seriously seen the inside of a computer.
Being happy with software and OS's (ie. Linux) has NOTHING to do with hardware. If Jon gets freaked by scary trashed hardware then I'm all with him. There is NO NEED to know what bizzaro decades old SCSI controllers look like in order to have a clue.
The first time I installed an OS I was freaked 'cos I was dealing with hardware - I mean cylinders, heads, sync frequencies,... what sort of baroque crap is that for this day and age anyway? But you get used to it and then forget it and get on with the real task - writing programs.
Anyway, Jon didn't write a Linux article, but it's still funny:)
Slashdot Mirror
it's ~10x faster on specfp_rate than the itanium2 altix.
specfp turns out to be a good representation of the codes we typically run...
Infiniband or gigabit ethernet are your main options. IB is low latency, and probably even more cost effective then gigE, but you may need the gigE anyway for a maintainance network (netbooting, NFS etc.). gigE usually comes with the motherboard, but you still need to budget for a fat tree of switches to connect it all. Myrinet's too pricy and (I think) slower then IB, but might be simpler to connect and has more mature MPI implementations for it.
Watch out for big vendor cluster software people - they may not actually know what they're doing.... not naming any names. What big vendor actually did (for the cluster next door to ours) was make it all slower!
IMHO you don't need that serial maintainance network crap they try to sell you, or even IPMI or similar. these Xeon/P4/Athlon64/Opteron clusters should be reliable enough that it's a waste of money. Our 264 node (528 Xeon) machine is fine without it.
If you want real bang for your buck then avoid the large expensive gigabit ethernet switches - they usually have limited backplane bandwith anyway. We use 2D mesh networking made from a stack of 24port gigE switches and had the fastest machine in Canada for a while... our networking is now way simpler than the hypercube-like topology on that page, but every node is still a router, and it works really well.
OSCAR is a great install system for a cluster. Do it yourself - it's the only way you'll ever be able to maintain the machine in the long term anyway... Just buy the hardware from anyone who gives you the best deal and looks like they'll be around for 5 years to replace nodes as they die.
Drop us a line if you want more dodgy advice
Doesn't Intel's free (as in beer) ifc compiler do f95 (and OpenMP)?
Yellow Pages Endeavour is a planing vehicle which is VERY different to a hydroplane based craft. One skims the water and the other lifts as much as possible entirely out of it. Wetted surface area on both is TINY, and so both are theoretically fast, but the hydroplane has got to be able to handle at least some waves whereas YPE would no doubt disintegrate. YPE ran for (and got) the speed record in a depth of <1m of water after all. So under real conditions Hydropetre is more promising - especially for the 24hr distance record which it looks like they're going for.
Big cats and tris are WILD, so one that's largely out of the water(!) is definitely the regime of those French nutter who are brave enough to push these things hard (and Grant Dalton of course :)
The cats in The Race were routinely at 35 knots so I don't think 45 is out of the question for a radical craft. This has been under development for a long time after all.
Hey, didn't many of the Formula40 cats and tris have some sort of hydroplane appendages? They were way cool... Are there pics of them anywhere? I only have one Becken calendar... :-/
I'm running C + MPI scientific codes. The MPI (lam mostly) calls map through to shared mem operations. My codes aren't super-heavy on the communications though - maybe 5% or 10% of the time is in comms in a mix of many small operations and some big bandwidth limited ops - so yeah - it's possible that an app which used GB/s of bandwidth between CPUs would hit more limitations in the Linux SMP implementation compared to Tru64.
As always it's a good idea to try your specific app and different OS's before buying the machine.
On my measurements they do compare. It's about a 5% hit for Linux vs. Tru64, and another 5% hit for the Compaq Tru64 compiler being better integrated into the OS than the Compaq Linux/Alpha compilers. So that's about 10% slower overall which isn't bad at all... It's true that gcc/egcs and it's math library is pretty lame on Alpha compared to Compaq compilers, but the Compaq compilers are free-ish so it's not a problem.
This is running MPI codes on 4-way Alpha/ev67/667 es40's with 8G mem and 2.2.16 Linux - so 2.4 Linux would be better - leaving only about a 5% difference between Linux and Tru64. Most of that is in page colouring (people guess) which Linux doesn't support.
The big thing that the Linux/Alpha version of the Compaq compilers doesn't support is OpenMP style SMP programming - so you'd have to do this by hand with threads or MPI or buy a different compiler.
My advice is to go with SMP Athlons as these are pretty much a match for Alpha in Floating Point these days - primarily 'cos Alpha MHz's haven't grown much for years. Athlons are also way cheaper so you can buy many more of them :-)
AFAIK Alphas haven't used any sort of an L3 since 21164 (ev56's). ev6 and ev7 have only L1 and L2 - just like Pentium2/3 etc. The Linux boot message on our es40's (4-proc Alpha) using Tsunami/Clipper chipsets mentions nothing of L3.
oops, I forgot to add --->
... and before anyone says that a stack'o'smaller boxes won't work - the guy said that his CFD app wouldn't run on linux, NOT that it required a large shared mem machine. Hence distributed mem systems may be an option. Even a pile of es40 Alphas would give you better price/performance than a single big shared mem box... not anywhere near the price/performance of linux/athlon, but hey.
Ahem, if you'd done your benchmarks then you would know that Linux/x86 ARE better. Certainly from a price/performance point of view, and ALMOST from an ultimate performance point of view as well.
This assumes a) fast Athlons (not necessarily duals) b) distributed memory (aka MPI) apps are ok.
Linux and gcc aren't superior to proprietary OS's (such as IRIX, Tru64 on SGI and Compaq/Alpha hardware) or their excellent C compilers, but it doesn't matter 'cos MIPS, Alpha, SPARC and HP MHz just haven't kept up with the x86 crowd.
Of course if you need large shared mem boxes then SGI and Compaq et al are your only choice, and boy do you pay for it...
My Astro/CFD benchmarks show that Alpha is WAAAY faster than SGI. In fact even my sub $2k Athlon 700 home linux machine is heaps quicker than a SGI/R12k/300/8M cpu. I don't expect that HP is that much better.
:-/
Hence get your CFD vendor to port their code to Linux/x86 and buy yourself a stack of Athlon (thunderbird) 950's. These are nearly as quick as the fastest Alphas (ev67/667/8M)... For the money you'll get about 4 times the performance out of your purchase if you can go with Athlons... If your CFD vendor won't port then find another vendor!
If you have to go with the expensive hardware then look on the SPEC website (www.spec.org) to see how they compare - SPEC is essentially biased towards the scientific and CFD apps you are running. Look for details on each part of SPEC to see which is closest. Also look at Compaq gs320 hardware - it might be the fastest of the 'big iron'
If Athlons came in duals or quads they'd be the only choice. We're still forced to look at Intels (which are about 30% slower per MHz and cost more than Athlons) just 'cos they come in duals
And once again, try your app on the real hardware before handing over that sort of cash.
yes, temperature gradients are huge, and there are vibrations etc. But the basic point is that glass is NOT a good material for strength. Use the absolute minimum of (heavy) glass and get all the strength and stiffness from carbon fibre or a similar material that is designed for strength and light weight. Hence a thin mirror bonded to a stiff and strong backplate, NOT a thick mirror.
If you think differential expansion is a problem then deliberately put grooves in the mirror (like the gaps in concrete pavements) to let mirror segments move apart. As long as the stiff backplate remains parabolic you have no problems.
It just seems like a correct material for the job issue to me... ???
BTW, mirrors on earth ARE limited by their deformability under gravity simply because they rotate and point at different angles to gravity, and so deform in a complicated way. Hence people now avoid single large mirrors and use segmented mirrors like Keck.
And exactly what is going to make it flop around?? This is in Space remember - ie. vacuum, no wind, no gravity, ... nothing to make the mirror deform apart from temperature gradients and the telescope's own tiny pointing movements. The mirror has only got to maintain its own shape which can be accurately set before launch/install.
The main reason big telescope mirrors are hard to build (apart from lack of money! :) is that they deform under their own weight - no such problemo up there dude.
If you're using tcsh then make sure autologout is unset. What might be happening is that the tcsh session from which you're firing up your remote xterms is logging out, and thus leaving ssh nowhere to connect to. So it hangs. The 300s thing sounds a lot like this. All my rxvt sessions stay up for days on end over ssh and have never died unless my PPPoE IP dynamically changes.
Oh, I guess that's another thing - how often does DHCP/pump(?) change your IP address? If that changes regularly (which it's allowed to do) then ssh would hang...
Where is the simap (secure SSL imap) support for pine though? Netscape has it... and it's dead easy to implement at the sever end with SSLeay or OpenSSL. The text-based clients are lacking though! :-/
The Linux BIOS Project was in this /. article just a week ago. It's not ready for prime time yet, but does all the BIOS step in about 0.1 seconds! Combine this with the right Linux distribution (debian has a fast (binary) service startup program - do others?), and a carful choice of which services you run, and Linux would boot REALLY fast! :)
... pathetic Greenpeace ship.
Yeah right. Someone DIED on that ship buddy.
At lest one of clueless French S.O.B.'s who had got caught ('cos of their own blatant arrogant stupidity) for that NZ lark got sent (eventually) to a french tourist island to serve her prison term and then (big surprise) got sent home early. Do you call that punsihment??? You have no clue about what Greenpeace try to do and no respect for human life.
I don't always agree with their media stunt methods (nor their junk mail) but they do more good than a dozen secret services of any nations you'd care to mention.
Does this sort of community remind anyone else
of "Shockwave Rider" or is it just me?
Anything so far removed from reality that actually needs to create something so artificial as Disneyville practicaly defines a civilisation that is doomed. Good luck USA, you'll need it.
I could competently admin Irix for years, had programmed about a squintillion lines of basic (sad but true - everyone learns somewhere), C, Fortrash, C++, Java, ... written web sites, blah blah blah... All before I'd ever seriously seen the inside of a computer.
... what sort of baroque crap is that for this day and age anyway? But you get used to it and then forget it and get on with the real task - writing programs.
:)
Being happy with software and OS's (ie. Linux) has NOTHING to do with hardware. If Jon gets freaked by scary trashed hardware then I'm all with him. There is NO NEED to know what bizzaro decades old SCSI controllers look like in order to have a clue.
The first time I installed an OS I was freaked 'cos I was dealing with hardware - I mean cylinders, heads, sync frequencies,
Anyway, Jon didn't write a Linux article, but it's still funny