Slashdot Mirror

1. Does it run Linux?

2. Imagine a Beowulf cluster of these!
Building a Beowulf Cluster in just 13 steps

How many cluster nodes per cm^3?

You might look at http://limulus.basement-superc... for a concrete example of the sort of system you are talking about. Also, http://www.beowulf.org/ is the home of the community of people who build compute clusters from any old hardware and run open source software on it.

It's a FAQ there, but you really should be asking this on the beowulf list, after skimming the list archives for any of the eight and a half million answers (in gory detail) that have been posted there in response over the years. Slashdot has plenty of nerds and I'm sure a lot of cluster geeks (who are likely on the beowulf list) but the beowulf list is sort of distilled cluster geekery/wisdom.

http://www.beowulf.org/mailman/listinfo/beowulf

rgb (Google "rgb duke beowulf" if you like -- I used to help answer this question once a month a few years ago on list, although I'm too busy and less active now.)

So Seymour Cray should have traveled to the future, scooped up a pallet of Droid phones and then created a beowulf cluster?

I'm imagining a Beowulf cluster of these. No, really, I am. I have a personal project that's going to eventually need a lot of parallel processing power. This thing has a dual core processor and comes with its own external power brick. They seem perfect for a cookie sheet Beowulf. Since they're aiming for retailing this for around $180, you can easily hit $200/per node once you add a stick of RAM. That looks cheaper than any other solution once you've added a power supply.

What most people, including the parent of this thread, don't understand is that NASA and other federal R&D facilities do is fuel our economy.


Many people here on /. work in the IT field. Well you can thank NASA for the Beowulf Cluster. NASA also worked with industry to make cordless drills, CAT Scans, digital thermometers, welder's goggles and thousands of other products.

Don't take my word for it.
http://www.beowulf.org/overview/history.html
http://space.about.com/od/toolsequipment/ss/apollospinoffs.htm
http://er.jsc.nasa.gov/seh/spinoff.html

Engage brain before moving mouth.

Have found nothing about this in the internet except a highly speculative, non-authoritative post in the Beowulf forum. Guess we'll have to wait for more details.

I'm not sure this all that big of a risk since computers have been around for a couple of decades now. It sounds like they just set up a Beowulf Cluster and didn't even use all the nodes possible.

"Beowulf Clusters are scalable performance clusters based on commodity hardware, on a private system network, with open source software (Linux) infrastructure. The designer can improve performance proportionally with added machines. The commodity hardware can be any of a number of mass-market, stand-alone compute nodes as simple as two networked computers each running Linux and sharing a file system or as complex as 1024 nodes with a high-speed, low-latency network." as quoted from http://www.beowulf.org/overview/index.html .

Electricity costs money too, and it keeps costing money even after the initial capital purchase (or lack-there-of). If you're building a cluster powerful enough to be comparable with even a couple low-mid end modern PCs you're not going to be running for too long before you hit the point where the power savings makes the modern PCs less expensive. A more thorough explanation can be found in these two posts from a beowulf mailing list:

http://www.beowulf.org/archive/2003-March/009658.h tml
http://www.beowulf.org/archive/2003-March/009662.h tml

Electricity costs money too, and it keeps costing money even after the initial capital purchase (or lack-there-of). If you're building a cluster powerful enough to be comparable with even a couple low-mid end modern PCs you're not going to be running for too long before you hit the point where the power savings makes the modern PCs less expensive. A more thorough explanation can be found in these two posts from a beowulf mailing list:

http://www.beowulf.org/archive/2003-March/009658.h tml
http://www.beowulf.org/archive/2003-March/009662.h tml

Y e a h.

Ever heard of the BriQ? (Picture) It's an entire PowerPC computer in a 5.25" drive form factor. Very cool. I'm not entirely sure if it is still available though as the link I have bookmarked isn't coming up. Either way, I figured it warranted mentioning.

"SPEC is subject to all kinds of problems."

All benchmarks are subject to problems. If you are testing for the CPU, somehow you must control for variables of compiler, OS, system architecture, and the amount of time and expertise of the tester.

Uh, how exactly do you get "GCC, is said to generate code that less well optimised than Intel's" from "GCC, is said to generate code that less well optimised for x86"?

Sigh. read: "Dell's own figures were calculated using different compilers and host operating system: Windows XP Pro, Intel's own C++ and Fortran compilers, and the MicroQuill SmartHeap Library 6.01. Secondly, the compiler used by VeriTest, GCC, is said to generate code that less well optimised for x86." QED less well than intels compiler in the previous sentence

GCC for PowerPC is not as mature: "The gcc scheduler is not really designed ideally for a processor like the 970 and the Power4...that was one of the things that we're continuing to work on to try to get the best performance out of the processor."

GCC on intel is far more mature with a long history, read a little of the history: "...When Intel released the Pentium some of their team produced a version of gcc with enhancements which gave 30% speed improvements on some benchmarks..."

Look at these redhat GCC 3.3/4.0 benchmarks. Notice how the 2-way PPC970 is twice as fast as the 4-way P4 on many tests and at close to par on the others. Now this is not the end all, am I'm sure you could come up with a different test that shows the P4 beating the G5, but certianly the G5 is not a "peice of crap".

You arguement about standardizing compilers is equivalent...

Standardizing of compilers is scientific method. Ideally you'd do a bank of tests, and unroll the variables: Standard compilers, standard OS, standard CPUs. Or you could tune each system to the max and then compare, that was LinPack and you didn't like that one either.

Hmm, does this appear to be vector processing done by a compiler?

Exactly my point! Intels compiler does auto-vectorization. GCC doesn't. If you test C code, P4+intel against GCC+G5, you are crippling the G5 by leaving out the altivec unit, which is a more capacble vector unit than SSE2

"hand coding...becomes completely out of reach for humans"

Hand coding is still done frequently on high performance algorithms:

• "I've also recently started hand coding the low level math kernels... P4, this gives a ~30% boost to performance on this particular MILC code."
• "However, once that level of optimization becomes necessary it's generally just easiest to hand-code the instructions, rather than jumping through a bunch of hoops to try to trick the compiler into doing what you want."
• "In some cases, complete vectorization is not possible and you may want to include hand coded SIMD instructions for the best possible performance"
• "Altivec requires hand-coding to exploit"

The people in TFA are part of a larger group (see http://www.gpgpu.org/) that thinks about how to use graphics cards for a wide variety of math problems. Here's an abstract from one of their papers:

In our experiments we compare the execution on a midclass GPU (NVIDIA GeForce FX 5700LE) with a high-end CPU (Pentium 4 3.2GHz). The results show that to achieve high speedup with the GPU you need to: (1) format the vectors into two-dimensional arrays; (2) process large data arrays; and (3) perform a considerable amount of operations per data element.

I'll repeat a cool link posted by somebody else: http://www.cs.unc.edu/~ibr/projects/paranoia/ - this is a program, originally written in the 80s, to characterize the performance and idiosyncracies of a floating-point processor. Recent work at UNC Chapel Hill has been done on Windows platforms. (Twenty years ago, UNC Chapel Hill was one of the hotbeds of computer graphics development that eventually gave us Shrek and The Incredibles.)

is a very fluff term, however both of our clusters have hard disks in all of the nodes as local caches for very large data sets. The next one we get will almost certainly have mid- to high-end GPUs as well, since a lot of the kind of things we do can be run (at least in part) in a GPU a lot faster than on a CPU.

I did not say that added stability was actual but if you want to see some references to chip set related problems okay.

http://support.microsoft.com/kb/q270715/
http://www.overclockers.com/tips59/
http://www.tech-report.com/news_reply.x/929/
http://www.beowulf.org/archive/2002-October/008407 .html
http://pcbuyersguide.com/hardware/motherboards/VIA -Problems.html

I do not think that any of these problems are current but they did or do exist. While Intel motherboards are almost never the fastest or most feature rich they do tend to be super stable. I like AMD. I plan on using AMD in my next server and I use use AMD at home for most of my boxes. I do not think that AMD is any less stable than Intel but that the combination of an Intel CPU and motherboard has a track record of stability that AMD + a non AMD chipset lacks. If you read my post you would understand that I am saying AMD needs to make a chipset and motherboard so it like Intel can be a one stop shop. There motherboards should be reference boards. Not the fastest, not the most cutting edge, but super stable.

There is a section of the server market where stablity is the number one metric and an Intel CPU on an Intel Motherboard has earned the reputation as the ultimate in stablity in the x86 market. If you want to see really stable look past the x86 to the Sparc and Power lines. For stability they make the x86 look like... Well the x86.

Big Iron:

BigTux Shows Linux Scales To 64-Way

My current test system has 16 CPUS:

zeus0:~ # tail -15 /proc/cpuinfo
processor : 15
vendor : GenuineIntel
arch : IA-64
family : Itanium 2

(yes, it is Itanium!! Anyone got a 16-way Opteron box? Anyone? Buhler? I thought not...)

And, of course, we all know about Linux clustering:

Beowulf Clusters
Single System Image Clusters for Linux

Ignoring the oddity of Oracle being in that group, none of the rest of the members actually make a scaleable Linux box, just ones that compete with them. The slant is obvious.

- Necron69

Listen up kid,

http://www.beowulf.org/

Beowulf and Windows

and

Beowulf Cluster Computing with Windows

A Beowulf cluster of these things would be awesome.

This RAID array is just meh.

Of course, you'd have to completely rewrite an operating system to use one of these babies.

http://www.beowulf.org/overview/index.html

OMG no body got this joke? *sigh* Either no one got this joke or did not want to acknoledge that it was a very good joke. Atleast the mods knew what you were talking about. Oh and for the people who still cant figure out the answer its a Beowulf Cluster

Don't bother. The bug he was remembering was fixed at least a decade ago. Since the bug has been fixed, and read() and write() work on block devices, cp will work just as well as dd.

Despite you're desire not to, you most certainly managed to be one anyway - adding a new wierd layer of superstitions on a old legend once grounded in buggy old OS's.

With a non-buggy OS that can read block devices correctly 'cp' will work as well as 'dd'. Or were you talking about a SunOS 1.X system?

from beowulf.org

Note that 'dd' is pretty much the same as
cp /dev/hda1 /dev/hdb1
The reason old timers recommend 'dd' is a long-forgotten semantic bug in
old UNIX systems that could be worked around by reading only whole
blocks from raw devices. This has now turned into superstition.

yeah right, do you know anything about beowulf (not beowolf)? i think not.. please read http://beowulf.org/overview/faq.html before saying such things..
beowulf clusters works great but with software written for them! you can't just run apache, tomcat, postgresql or oracle on such cluster and expect they would run faster. they propably won't. maybe mosix is an answer but frankly i don't think any cluster system is adequate for this job.
if you want high availability and high performance at the same time you won't run away from load balancers and redudant setup. sorry, no bonus.

"A Beowulf Cluster of these!!!"

What? No one has posted that already?

• Jörg Schilling, cdrtools
  
• Donald Becker, linux ethernet drivers, Beowulf
  
• thekonst, centericq (a console IM client)
  
• Alan Cox, linux kernel guru (I hate that word, but it fits), including being the primary maintainer of the 2.2 tree
  
• Paul Vixie, Vixie cron, BIND, ISC

"The run clusters of small machines, that's what it sells today and that's where Microsoft is putting their eyes."

yes because it's well known that linux does not cluster very well and neither does the Mac. It's smart of MS to go after a market where the competition hasn't demonstrated that it can outscale MS without even trying.

Yea right.

Or, just "cp /dev/hda /dev/hdc".

And yes, I have heard about 'dd'. cp works just fine

Mustard gas

Corporate site for Fluorinert

doesn't this CTO of cray remind you of someone?
"There IS no Linux in high-performance clusters."

"There IS no Americans in Iraq."

OMG! It's the former Iraqi mis-Informed-ation minister!

Especially when 2004 has been dubbed the year of the penguin, it's wreckless to claim that Linux can't be used in HPC's.
Hell, just look at the current top500 list. There's no Cray in the top 10 but there are two Linux based clusters there (and one based on OSX [FreeBSB based]).

Here's a few:
NCSA's IA32 Linux cluster
NCSA's IA32 Linux cluster
Space Simulator Clust at Los Alamos (SS51G based; makes me proud as I have a SS51G too)
Beowulf - used in many Linux clustering projects
Linux clusters at Los Alamos (they seem to have more than one)
Virginia Tech's Supercomputer X

If you have seen all the jokes, but you still don't know what a beowulf cluster is, then this site is for you. It has all you need to know about it.

Imagine a Beowulf cluster of these.

The original links went to NASA/GSFC, but the Beowulf project central site has moved.

Imagine a beowulf cluster of these...

It's hard to imagine that this is cost-competitive with Linux clustering. Macs aren't cheap.

(the same thing happpened with Japan and the PS2 I think, or it might have been a urban legend in that case)

No, it was not; although Japan wasn't exactly the culprit.

What would you expect to happen? (assuming you have large file support)

Other people on beowulf.org also report successes.

"Note that 'dd' is pretty much the same as
cp /dev/hda1 /dev/hdb1 The reason old timers recommend 'dd' is a long-forgotten semantic bug in old UNIX systems that could be worked around by reading only whole blocks from raw devices. This has now turned into superstition."

Speaking of the good old days, here is an "old" comment by Robert G. Brown denouncing Liquid Nitrogen cooling as dangerous and a waste of time.

I like the idea of CO2 cooling though...

Since the article doesn't really have to do with grid computing. Here are some real Grid Computing links.

Globus Toolkit
LSF
openPBS
gridengine
OSCAR
ROCK MPP
maui

and last but not least: beowulf cluster
---

here and here are probably good places to look.

From the "Related links" Slashbox to the right of the article: Beowulf cluster. Go educate thyself.

And your mom said to tell you to watch what you say, potty mouth. Heh.

This was recently discussed on the Beowulf list.

And it's not as simple as hardware/software prices/support. There's some critical stuff that cannot and does not show up in the specs, and it's not cheap.
I agree. Perhaps, after all Linux can not seem to run very top-end systems. Worse yet, you will not find it in the enterprise systems As to the Desktop, Well skip that as well

You can solve yesterday's problems on tomorrow's computers quite cheaply.
Same thing here as well (with out the sarcasm). Tomorrow's problems are being solved on todays computers due to their low cost. Otherwise, we would be waiting till the costs of the computers were less than the costs of the problem.

The more imaginative may notice a link here; essentially, NASA needed good networking in their Beowulf nodes, so they tweaked the drivers.

I have hot swapable drive support. HP is working on this for w2k but does dell have this?

Mac OS X has had this since version 10.0. It's called FireWire(tm) brand IEEE 1394 serial bus technology. FireWire storage devices are hot-swappable, and there's no reason a hardware manufacturer can't make a cage that holds a couple dozen FireWire drives.

Every version of the Microsoft Windows OS since Windows 98 and Windows 2000 can speak IEEE 1394 as well.

Int64

The i386 architecture can already handle 64-bit long long variables straightforwardly, just not natively. The x86-64 architecture processors (AMD Athlon 64; AMD Opteron) will fix this deficiency. (Yes, I know Dell doesn't currently sell AMD CPU based machines.)

I have a scalable server that has supperior clustering software that NT and Linux lack. With up to 128 processors I can have one fast mutha.

Can you imagine a Beowulf cluster of PC blades? If not, what clustering feature does the Solaris OE have that the GNU/Linux OE lacks?

Linux has serious VM problems

I've been told Linux 2.6 will take steps toward improving the kernel's scheduler and VM.

If a chip fails you can have an engineer from Sun with a replacement part be at your office within a matter of hours if your a gold member!

Dell has only begun to compete with Sun. How do you know Dell won't introduce an analogous "gold" program for its machines?

here

MacSlash has a brief note how NASA's JPL has put together a cluster of 33 XServes that was able to achieve 1/5 teraflop

I think you meant:

MacSlash has a brief note how NASA's JPL has put together a cluster of 33 XServes that was able to achieve 1/5 teraflop. Apple.

Well, I don't know how to use CLIC, but I set up a cluster using some surplus P2s, RH 7.3, and openMosix.

For kernel 2.4.18, openMosix consists of two kernel patches - a kernel patch to actually do the clustering, and a kernel patch for administration. You patch your kernel, change a few settings (all documented on the site), reboot (gasp!), and you're done.

You can get more fancy with net installs, but you don't really need to for a small number of nodes. Anyway, I just used a Fast Ethernet switch - the boxes all came with Ethernet cards, luckily for me.

Mine is a pretty craptacular cluster, but hey, it's pretty cool to have one. The cluster behaves like one powerful computer in the sense that processes on one machine will migrate to another. However, in order to achieve any noticeable affect, the program has to take advantage of multiple CPUs. If you're going to program your own, look into forking, message-passing interface (MPI), and parallel virtual machine (PVM). Sorry, I don't have the addresses for them on hand, I think www.beowulf.org links to them.

The guys on the Beowulf mailing list dealt with this a while ago. The general consensus was HT would be effective in applications which have, for instance, threads which do I/O and others which do computation, as well as programs in which memory latency is an issue. Additionally, it was postulated that HT would enable simultaneous int and fp operations. Generally, they found compute bound problems received no benefit -- there is physically no significant mathematical capability added to Hyperthreading CPUs.

The thread begins here. There are some posts which aren't in the thread, but are relevant. Find them here

The guys on the Beowulf mailing list dealt with this a while ago. The general consensus was HT would be effective in applications which have, for instance, threads which do I/O and others which do computation, as well as programs in which memory latency is an issue. Additionally, it was postulated that HT would enable simultaneous int and fp operations. Generally, they found compute bound problems received no benefit -- there is physically no significant mathematical capability added to Hyperthreading CPUs.

The thread begins here. There are some posts which aren't in the thread, but are relevant. Find them here