Mini-ITX Clustering

NormalVisual writes "Add this cluster to the list of fun stuff you can do with those tiny little Mini-ITX motherboards. I especially like the bit about the peak 200W power dissipation. Look Ma, no fans!! You may now begin with the obligatory Beowulf comments...."

Comments

Imagine..

[hookedup]

A beowulf cluster of these? There, done... and it felt good!

Re:Imagine..

[Anonymous Coward]

Just a single one of these. It'd be like me on a Friday night!

Re:Imagine..

[Anonymous Coward]

why imagine... just make one...

you can't complain now because you don't have a vast underground bunker to build your massive cluster in.

..although I've never quite figured out just what to do with such a thing if it were to be built. Could you run seti@home on a Beowulf cluster?

Re:Imagine..

[iminplaya]

Too Many Users

Evidently they didn't cluster enough...

Re:Imagine..

[Mr. Bling Bling]

Yo what up?

Would it be possible to set up a clusta of these in a stretch Escalade? If so, how much would it cost and can I get some iced out (real diamonds, not no zircon encrusted shiat) 1U or smaller cases for the nodes in the clusta? Anybody willing to set something up for me. I gotst cash fo it.

Re:Imagine..

[Anonymous Coward]

Maybe, but there would be no point. Much better to have them each run SETI independently.

Re:Imagine..

[SEWilco]

I might be the originator of this phrase, so I would be qualified to point out that the proper phrasing requires the informative link:

Imagine a Beowulf cluster of these.

The original links went to NASA/GSFC, but the Beowulf project central site has moved.

Re:Imagine..

[Anonymous Coward]

Use it to run anything that you normally would. But remember that you will probably have to roll your own software to take advantage of a cluster.

Re:Imagine..

[pair-a-noyd]

Imagine,
a beowulf cluster of beowulf cluster comments!

Re:Imagine..

[Cypherus]

Screw Beowulf Clusters...Open Mosix Clusters are where it's at! http://openmosix.sourceforge.net

Re:Imagine..

[Digi-John]

OpenMosix is pretty nice... I have a cluster in at my school, it works pretty well. It is 8 PI and PII nodes, I run SETI on it and get about 3 units done a day. Info and pictures at this site, check it out.

Re:Imagine..

[Doomdark]

Evidently they didn't cluster enough...

After Slashdot flash crowd came, it was all just a big Cluster. :-/

Imagine...

[Chmarr]

... a beowulf cluster of obligatory beowulf cluster comments.

Re:Imagine...

[Anonymous Coward]

John Lennon surely would not have been happy!

Re:Imagine...

[Anonymous Coward]

A clusterKnoppix of these things...

Moshe would be shocked.

Re:Imagine...

[Anonymous Coward]

...a new, original joke. Now imagine another one, because that last one wasn't that funny.

You mean... a Beowulf cluster of new, original jokes?

Re:Imagine...

[FurryFeet]

In Soviet Russia, shit does you!.

Sorry. I'll go kill myself now.

Re:Imagine...

[Anonymous Coward]

In Soviet Russia, unfunny joke imagines YOU!!!

Re:Imagine...

[will234]

hello i will intrested

Re:Imagine...

[Anonymous Coward]

In Soviet Russia, Beowulf clusters YOU!

Re:Imagine...

[Adriax]

a beowulf cluster of obligatory beowulf cluster comments.

Isn't that the definition of Slashdot?

Floating point performance

[October_30th]

I thought about this some time ago.

I decided against a mini-ITX cluster because the floating point performance (why else would you build a cluster?) of VIA CPUs is just abyssmal.

Is there any reason why there are no P4 or AMD mini-ITX mobos around?

Re:Floating point performance

[wed128]

i would imagine they run too hot for such a small form factor...this is just a guess, so treat it as such.

Re:Floating point performance

[Anonymous Coward]

the power consumption of the desktop p4/amd chips would kind of defeat the purpose of building one from these

Re:Floating point performance

[Short Circuit]

Not to mention that mini-ITX is VIA-proprietary technology. At least, I think it is.

And VIA markets their own line of CPUs for use in that scenario.

However, I wouldn't mind seeing Pentium-M or mobile Athlons placed on mini-ITX boards.

Re:Floating point performance

[J3zmund]

They might be on their way. Here's a 1.7 GHz Pentium M.

Re:Floating point performance

[0x1337]

The reason why you don't see any Mini-ITX mobos around the Athlon, is power consumption. I recently built a mini-ATX computer around a T-Bird (1gHz, should have picked something less of an oven), and the mini-ATX power supply crapped out on me, making me buy a REAL ATX powersupply. Gah, still can;t find a 300 WAtt mini-ATX supply.

Btw, you're wrong - there ARE P4-based mini-ITX mobos.

Re:Floating point performance

[stratjakt]

Mini-ITX really isnt THAT much smaller than some FlexATX boards, most notably shuttle's offerings..

You just have to find a way to dissipate the heat, the heatpipe setup in Shuttle's latest line of barebones is pretty clever.

As for the Mini-ITX cluster, it's kind of a joke. You may as well just cluster old 486 boards, it'd be cheaper, they can be had for a buck or so..

You'd probably have to cluster a dozen of them together to equal one 3.6 p4.

Re:Floating point performance

[-tji]

There are P4 Mini-ITX systems available: Pentium 4

But, most mini-itx systems are very small in size, and strive for quiet or silent operation. So, there are obvious problems with the P4's heat/power requirements. Perhaps a better solution is the Pentium-M in a mini-itx form factor. It has pretty good performance, at a low power/heat level: Pentium M. But, most of the Pentium-M boards are intended for industrial or OEM use, so they are hard to find in retail, and are pretty expensive.

Re:Floating point performance

[niko9]

How about Fujitsu's mini-tx form factor for the Pentium M proc. Runs passive (huge heatsink, but passive nonetheless) and uses less electrons.

Coudn;t find a link though, sorry.

Re:Floating point performance

[IWorkForMorons]

I've seen AMD and P4 Shuttle systems that are that size. They may not reach the under 12 CM size boards that VIA are starting to make, but mini boards are available for AMD and P4.

This was found after 2 seconds with Google.

Re:Floating point performance

[October_30th]

Sounds excellent.

In fact, a Pentium M platform would be a perfect choice as long as the mobile Athlon mobos are impossible to find.

Does anyone have a link?

Re:Floating point performance

[F34nor]

HEAT.

Re:Floating point performance

[Unoti]

For SETI, I'd need about 33 of my Pentium 133's to equal the performance of one of my AMD 2500's.

Re:Floating point performance

[hawkbug]

Using a t-bird will do that to you for sure. Consider using an XP 1500+ based on the thoroughbred core... and your energy consumption will greatly decrease.

Re:Floating point performance

[Anonymous Coward]

But Pentium M's aren't cost-effective...

Re:Floating point performance

[a20vertigo]

There are supposedly some Pentium M boards around, as well as 4s... in fact, if you look at Mini-ITX.com's store, they're selling a P4 mini-itx board. If only it's one slot was AGP and not PCI, that would make a hell of a small little gaming box...

Re:Floating point performance

[mi]

the floating point performance (why else would you build a cluster?)

• To crack encryption?
• To compile big projects?
• To compress huge files?

The floating point is just a convenience. Almost any algorithm can be modified to work with fixed point precision -- and without loss of performance.

Of course, many people will insist, they need FP to be able count dollars and cents -- they don't even think of counting cents (or any other fractions of the dollar) with integers, for example.

These are, usually, the same people, who have troubles defining bit...

Re:Floating point performance

[a20vertigo]

By Sandra's floating-point benchmark, the 1ghz VIA Ezra CPU couldn't match my old AMD K6/2-550... and that wasn't even that fast of a chip! Or that hot of one, either.

Re:Floating point performance

[F34nor]

He had a beefy external DC to DC Power supply.

Re:Floating point performance

[ducman]

Interesting timing on this. I was just looking at a mini-ITX mobo on tiger direct this morning. I'd been looking for SATA drives to add to my home storage server. But they had a mini-ITX board with 366 mhz Celeron and on-board IDE raid for about 50 bucks! That's less than most SATA controllers. So now I'm thinking I might buy, say 4 of those boards, put two, 250 Gb IDE drives on each, and have a fully mirrored terabyte cluster.

Re:Floating point performance

[struan]

There are Pentium-M boards.

I'm not aware of any Athlon-based boards, but mostly because I'm satisfied with my Via-based M10000 board.

Re:Floating point performance

[J3zmund]

Here is a Pentium M powered Mini-ITX board.

Re:Floating point performance

[steveha]

the floating point performance [...] of VIA CPUs is just abyssmal.

Older C3 cores run the FPU at half the clock rate. If you get the fanless 600 MHz EPIA motherboard, the FPU will be running at 300 MHz.

The newer, Nehemiah core C3 chips run the FPU at full clock speed. Any C3 newer than Nehemiah should run the FPU at full speed.

He used the VIA EPIA V8000A motherboard with an Eden core CPU. From what I found on google (here), the Eden core does run the FPU at full clock speed.

In any event, he said the cluster has more processing power than a four-P4 SMP system, while taking less electricity to run. And it will be quieter and more reliable. I'd like to see actual benchmarks, but it seems like it makes enough sense.

I read about a cluster of PocketPCs, and that didn't make practical sense. It was just a fun project.

steveha

Re:Floating point performance

[steveha]

he said the cluster has more processing power than a four-P4 SMP system

Whoops, I made a mistake. He actually said his 12-node VIA cluster has more power than "four 2.4 GHz pentium 4 mcahines used in parallel". Not SMP!

Sorry about the mistake.

steveha

Re:Floating point performance

[aonaran]

Really?? Where is it, I just searced for Mini-itx and miniitx on tigerdirect.com and .ca and found nothing.

Re:Floating point performance

[Mr. McGibby]

the floating point performance (why else would you build a cluster?)

* To crack encryption?
* To compile big projects?
* To compress huge files?

How about scientific computing? That's really the big thing that keeps cluster computing alive. Cracking encryption is the only thing on that list that makes sense. The other stuff shows your lack of knowledge of other disciplines by the fact that you think these are computationally expensive tasks.

Re:Floating point performance

[dabadab]

There's one thing that makes VIA CPUs very interesting performance-wise: the xcrypt instruction. Using it the VIA CPUs just beat - and beat badly - anything else in certain task.

Check out Theo de Raadt's little benchmark:
http://marc.theaimsgroup.com/?l=openbsd-misc&m=107 577297024182&w=2

Re:Floating point performance

[ducman]

http://www.tigerdirect.com/applications/SearchTool s/item-details.asp?EdpNo=86698&Sku=MBM-TS20-C3 66

$49.95 with Celeron 366

Re:Floating point performance

[mangu]

The floating point is just a convenience. Almost any algorithm can be modified to work with fixed point precision -- and without loss of performance.

But at a significantly higher development and debugging cost. Why go for integer adaptation, if a P4 can do four FP operations in one clock, using SSE2? I have tested my 2.4GHGz P4 at 6 gigaflops, in a practical application doing matrix inversion. The theoretical maximum for my machine would be 9.6 Gflops. If you RTFA, you'll see they mention 3.6 Gflops performance for their cluster, about 60% of my single-processor system. I see no point at all in building that cluster.

Re:Floating point performance

[Anonymous Coward]

The theoretical maximum for my machine would be 9.6 Gflops.

I'm interested. Do you have to use assembly to get this, or can you plunk down some C code that reaches this?

Re:Floating point performance

[merlin_jim]

I decided against a mini-ITX cluster because the floating point performance (why else would you build a cluster?) of VIA CPUs is just abyssmal.

Not only does the Nehemiah core (which wasn't used in this cluster) have full speed FPUs (unlike the half-speed Ezra core), but it has hardware acceleration support for RSA and keygen operations.

Is there any reason why there are no P4 or AMD mini-ITX mobos around?

There are a few versions with P4 Mobiles. One of the big reasons is that the low cost and low size of the mini-ITX depends on not using a ZIF socket, but having the CPU actually soldered in. The other reason is heat. If you're building a small form factor computer but have to put three fans in and plan for 2-3 liters of airspace, that kind of defeats the point...

Re:Floating point performance

[Anonymous Coward]

It's pretty much up to how good the compiler is.

If you use the Intel compiler and its vectorization stuff then you could easily do it in C. It will generate the SSE instructions.

Most compilers like GCC won't give you maximum performance without resorting to asm.

Re:Floating point performance

[mi]

Mars is not made any closer to Earth by the revelation, that Alpha Centauri is really far...

How about scientific computing?

This is why you might need the FP performance. I was answering a totally different question -- what would you do without the good floating point performance.

The other stuff shows your lack of knowledge of other disciplines by the fact that you think these are computationally expensive tasks.

Thank you, thank you.

Would you, please, demonstrate, how I can rebuild a project of 3000+ files, modified by 100+ developers (ccache helps, but still)? Or compress a 32Gb database dump? Granted, these tasks are nothing compared with, say, protein folding, but they are computationally expensive still.

Re:Floating point performance

[magarity]

floating point performance ... of VIA CPUs is just abyssmal

Nevermind the individual CPU performance; I thought the biggest slowdown in clusters was inter-CPU communication. This setup is using the built in 100base-t and connecting all the nodes plus the controller to the same 16 port unmanged switch. While a nice project just to do it, doesn't it really want gigabit and a segmented layout?

Re:Floating point performance

[dustymugs]

there are p4 mini-itx boards available in the us at http://store.ituner.com/ituner/viaepiap4itx.html

It may not support 800 fsb... but its a p4....

Re:Floating point performance

[Anonymous Coward]

I decided against a mini-ITX cluster because the floating point performance (why else would you build a cluster?) of VIA CPUs is just abyssmal.

Hmm, high availability and/or load balancing come to mind.

As others have said, the Nehemiah core based boards are really slick little widgets.

Re:Floating point performance

[merlin_jim]

He used the VIA EPIA V8000A motherboard with an Eden core CPU. From what I found on google (here), the Eden core does run the FPU at full clock speed.

I have the VIA EPIA 8000 (not sure what the V and A modifiers mean), with an Ezra core. FYI, Eden isn't a core, it's an initiative. The VIA Eden is aka VIA EPIA 5000, and was the first fanless Mini-ITX. Eden was the development product moniker, and came to refer to the motherboard that was first produced from that initiative. It can also refer to any C3 CPU made to run fanless.

Back onto the original topic; my EPIA 8000 with an Ezra core runs the FPU at half clock. This document on the differences between the Ezra/Ezra-T and Nehemiah cores indicates that one of the fundamental differences between the two is the full speed FPU. So I doubt that the article you quoted is accurate...

Just some more info... Nehemiah was manufactured at 933 MHz, 1 GHz, and speeds up to 2 GHz are planned. The Ezra was manufactured at 533 MHz and 800 MHz in its first run; the 533 is also known as the Eden. The Ezra-T (the second run of the Ezra) was made at 600 MHz (aka Eden), 800 MHz, 933 MHz, and 1 GHz.

Re:Floating point performance

[addaon]

If the heatsink is massive, and it's made of aluminum, it probably makes up a significant number of the atoms in the computer. As a result, the Pentium M mini-itx board probably uses more electrons. It also, purely coincidentally, uses more electricity than the Nehemiah boards.

Here's your link, by the way.

Re:Floating point performance

[addaon]

See, that's how I used to think. G4 at 800MHz... 4 fp operations in parallel with altivec... 3.2GFlop goodness. But of course, why stop there? With various tuning, you can get up to 32-way parallel integer math (although going beyond 16, admittedly, sucks). 3.2 GFlop is nice, but 25.6 G-ops ain't too shabby.

Re:Floating point performance

[Enahs]

And modded "interesting" at that...you don't even need a benchmark to understand why a 1GHz Ezra isn't as fast as an AMD K6/2-550. On an Ezra, the FPU runs at half the speed of the CPU. The Nehemiah(sp?) doesn't have that handicap. Even so, instructions like cmov haven't been implemented.

Here's an old review. The VIA processors aren't built for speed; they're built for low power consumption. In that department, they're great. They're also relatively cool, temperature-wise.

I've got a machine based on a 1GHz Ezra; it's really not as bad as it sounds. Not a stellar machine, but it's got better performance than the K6-2 it replaced (but not a whole helluva lot better, especially when there's lots of floating-point math involved, for the reason stated above) and it doesn't require insane cooling techniques.

Re:Floating point performance

[afidel]

Gigabit would be good for reducing latency (bandwidth probably isn't a problem) but with only 12 nodes you probably don't need to do any segmentation as most switching fabrics can handle that at full speed. What I want to know is why no one I've seen has used a Cisco 6513 with gigabit modules for clustering, it's switching fabric is more than up to the task.

Re:Floating point performance

[gasmasher]

You had me all excited. The link is actually a microATX Socket 370, not a miniITX.

Re:Floating point performance

[bhtooefr]

AMD? The CPU is too big. As for Intel, Commell makes P4 (LV-670 - no Prescott, but still...), P4-M (LV-670M), and P-M boards (LV-671, available in MP (Mini-PCI) and MA (Mini-AGP) variants).

Re:Floating point performance

[bhtooefr]

Funny you mention cracking encryption. Just think - a beowulf of C3s could encrypt better than the same number of P4s (any speed) or AXPs (any speed) and maybe even A64s. It's because an encryption engine is on the C3 (C5XL cores or newer, and it gets better at C5P)

Re:Floating point performance

[bhtooefr]

Here's what the modifiers mean (don't know about the A suffix, though), and what boards have them:

No prefix: has two IDE connectors
VIA EPIA 500 (533MHz Ezra Eden)
VIA EPIA 8000 (800MHz Ezra C3)

V (Value) prefix: has one IDE connector, one FDD connector, and lower quality components (compared to the original)
VIA EPIA V5000 (533MHz Ezra Eden)
VIA EPIA V8000 (800MHz Ezra C3)

M (Multimedia) or ME (Multimedia Eden) prefix: has two IDE connectors, one FDD connector, and MUCH higher quality components
VIA EPIA ME6000 (600MHz? Ezra Eden)
VIA EPIA M9000 (933MHz Ezra C3)
VIA EPIA M10000 (1.0GHz Ezra C3)
VIA EPIA M10000 (sometimes is marketed as M10000B/2/N - 1.0GHz Nehemiah C3)

CL prefix: similar to M, but has less multimedia and more networking. Allows for upgrades other than PCI. Performance also seems to be better.
VIA EPIA CL10000 (1.0GHz Nehemiah C3)
possibly others

TC (Thin Client) prefix: has onboard DC-DC converter, has the new CL upgrade interface, otherwise think V series, except it performs better than an M series:
VIA EPIA M10000 (1.0GHz Nehemiah C3)
possibly others

Re:Floating point performance

[bhtooefr]

Keep in mind, this site is trying to pass it off as the EPIA P4 that VIA had internally developed. No such thing was released, and this is a Mini-ITX board with a VIA chipset.

Re:Floating point performance

[QuantumFTL]

The floating point is just a convenience. Almost any algorithm can be modified to work with fixed point precision -- and without loss of performance.

Apparently you've never done any numerical computing, especially of the scientific variety. In an astrophysics simulation, for instance, the density of a field may span over 20 orders of magnitude, hardly reasonable to do with fixed point arithmetic.

Not to mention that many iterative algorithms can oscillate wildly in the presence of numerical error.

It is true that there are many other uses for a cluster besides numerical computing, however the idea that any floating point algorithm can be converted to fixed point could not be more wrong.

Disclaimer: My research at Cornell University is high performance clustered numerical computing.

Cheers,
Justin

Re:Floating point performance

[steveha]

Thank you for the corrections.

I wonder what he was thinking -- why he didn't get Nehemiah core motherboards. For that matter, I wonder why he used microdrives instead of just getting extra RAM and having the motherboards do net boots?

steveha

Re:Floating point performance

[Hoser McMoose]

The floating point performance of Nehemiah still stinks, it just stinks somewhat less than the FP performance of previous processors. The built-in cryptography units in quite nifty though, albeit for only rather limited purposes.

As for the P4 and AthlonXP chips in Mini-ITX, the issue is primarily to do with cost, not power/heat. Just compare the size of a typical Mini-ITX system to a laptop, and now go see how many mobile P4 and AthlonXP-M laptops there are out there. Some mobile P4 chips have a TDP of up to 70W and they still manage to stick them in a laptop, so sticking a low-power AthlonXP-M with a TDP of only 25W in a Mini-ITX case would be easy, but it would cost money.

Re:Floating point performance

[mi]

20 orders of magnitude, hardly reasonable to do with fixed point arithmetic.

You only have so many bits to work with (and to store the numbers). So you can only have so many distinct numbers -- however you choose to represent them (as integers, or as FP). A 128-bit integer will cover the variations of densities you mention comfortably. Just as you are probably using "long double" currently for these to have precision. If you don't care for the precision of mantissa (sp?), you may as well adopt logarithmic scale and use "short" integers for exponent values only :-)

the idea that any floating point algorithm can be converted to fixed point could not be more wrong.

I only claimed that almost any algorithm. In some others it just does not make sense, because it is inconvenient and unnatural. My point was simply, that it is possible (and even desirable) in more cases than many realize...

Re:Floating point performance

[Anonymous Coward]

Actually there are P4 mini-itx boards around.

It's just that they cost more and are low in quanity.

Most mini-itx boards are VIA because they are cheaper and the CPU is quiet.

Re:Floating point performance

[tietokone-olmi]

Dude, the 800mhz VIA chips have the 3dNOW! extensions in them. Remember the AMD K6-2 and K6-3? The ones with the completely non-pipelined x87-style FPU that, quite frankly, sucked? But they ended up having a fully pipelined, 2-cycle latency packed 2-way SIMD floating-point thingy in 3dNOW! that could totally issue a pfadd/pfmul pair on every damn cycle in parallel. Remember how the optimized Voodoo2 driver for a K6 absolutely blew anything else out of the water?

Anyway. What I'm trying to say is that you generally wouldn't provide a SIMD floating-point implementation if your underlying core is rubbish. Still, the site states that the cluster is probably equivalent to a significantly costlier cluster of 5 or 6 2.4ghz P4 boxes, and with the credentials the builder's got I'd say he's done some proper benchmarking already. That said, it wouldn't be hard to convert whatever the hell he's using for linear algebra to use 3dNOW! loops for the matrix multiplications, seeing as that the builders of one experimental cluster (made of Athlon Thunderbirds, IIRC) already wrote a patch to cantrememberthename to that effect.

Re:Floating point performance

[Anonymous Coward]

Sucker.

Plain old SSE does 1 or 4 operations per instruction (i.e. 1+1 operands for the scalar variants, 4+4 operands for the vector variants). SSE2 simply adds a packed double-precision floating point type and operations to do stuff with them (i.e. again, 1+1 operands with scalar doubles and 2+2 with vectors of 2 doubles each). Along with the more versions of the old 64-bits-wide MMX instructions that chomp 128-bits-wide XMM registers instead, which is nice.

Besides... I'd very much like to see the real-world application (a linear algebra benchmark would do as well) that would give you even a significant fraction of that "9.6 gigaflops theoretical maximum". Given the P4's terribly long pipelines and the associated vulnerability to data dependency stalls and memory load latency, you'll be hard pressed to find the instruction path that gets you even one G-unit's worth of floating point operations per second. ... and you do remember, don't you, that some matrices can be inverted with nothing more than a transpose?-) A special case to be sure, but...

Re:Floating point performance

[tietokone-olmi]

If you're going to compute the kind of numbers one would run across in an astrophysics simulation, it would sound reasonable that you'd also be using some kind of a rational number library (GNU MP comes to mind), especially with the concern for numerical error you talk about. Though I can't be sure -- I guess the performance tradeoff would be rather large. Anyway, from a quick read of some of the multi-precision floating-point number code in GNU libgmp, I'd conclude that it doesn't do all that much with the FPU. ICBW tho.

Not that I disagree with you on the fixed-point matter. The original poster must have been brought up on old 486s or early Pentium 1s and the canonical affine texture-mapping loop or something :-)

Re:Floating point performance

[tietokone-olmi]

Then there's anything at all that requires a not-quite-trivial amount of computation per item but that generates absolutely huge datasets. Or just medium datasets that need to be gone over several times, as with some iterative clustering algorithms. I'm sure there are other examples where the given data is not naturally in a floating-point format.

In any case, the cluster is pretty neat. Apart from the obvious hack value I'd think that it would be quite nice having something like that to develop your parallel processing applications on, considering the low power requirement and heat dissipation (which I'd expect would help with reliability, though really I haven't a clue), provided that you kept your development datasets small enough to fit on one non-RAIDed 160-gig volume.

Re:Floating point performance

[merlin_jim]

I wonder what he was thinking -- why he didn't get Nehemiah core motherboards. For that matter, I wonder why he used microdrives instead of just getting extra RAM and having the motherboards do net boots?

Check out my recent posts. In one I discuss the viability of PXE booting and the comparative cost savings, plus a proposed network topology to make the computer more efficient. Remember, in a cluster, the fundamental limit isn't the computing power available but the bandwidth to combine it. In another few I suggest Live CD distributions that are already configured to do it.

As I said before, if only I had a few grand :)

Re:Floating point performance

[QuantumFTL]

I only claimed that almost any algorithm.

Uh... If you mean rediculously simple algorithms that people use for non-intensive stuff, perhaps. Anyone who's using a cluster to run a numerically intensive program is most likely using serious numerical algorithms.

Try reading Numerical Recipes, I think you'll see that most numerical algorithms are sensitive even to Floating Point error enough that degrading it even further would be quite disasterous.

Or maybe you are still thinking about adding dollars and cents?

Re:Floating point performance

[mi]

most numerical algorithms are sensitive even to Floating Point error enough that degrading it even further would be quite disasterous.

You did not get it. You are looking at the bit (and byte, and word) as a number. I suggest you look at it as a unit of information. With 64 bits you can only have 2^64 distinct possibilities. If you choose to treat them as numbers -- fine, you only have 2^64 distinct numbers.

You may split the 64 bits to use some of them to represent the mantissa and some as the exponent. Or -- use all of them to represent the integer number of the smallest units in your application's domain.

The second method, actually, gives you better precision in a controllable (by you) fashion. If the difference between the smallest and the biggest quantity of those minimal units in your application exceeds 64 orders of binary magnitude, than 64 bits is not enough for you -- regardless of whether you use floating or fixed point. You either lose precision (FP) or overflow (int).

The reason to use FP may be because it is more convenient to think in terms of standard units, rather than the minimal units of the application (its precision). Also, many CPUs have special features allowing to do FP computations really quickly. But it is possible to go without them.

Or maybe you are still thinking about adding dollars and cents?

That was just an example. If 1/10 of a cent is the precision you require, than treat the integer number 361 as 36.1 cents or $0.361. Likewise in (almost?) any other domain, the minimal unit can be taken as the integer 1 and other quantities can also be integers.

Re:Floating point performance

[agent0range_]

Is there any reason why there are no P4 or AMD mini-ITX mobos around?

None, really.

One need only follow a link on mini-itx.com to find P4 mini-itx motherboards. They're a little more expensive than VIA's products, though.

http://www.commell.com.tw/Product/SBC.htm#Indust ri al%20Embedded%20MB

Re:Floating point performance

[QuantumFTL]

For our applications, the numerical error in normal 32 bit IEEE Floating Point is good enough... Our processing power is still a bit limited, and there are other sources of error in the program, mostly due to having imperfect boundary conditions.

I understand what it's like to want to overoptimize everything..> iw as brought up on a 68030 processor, and I did fixed point myself for a lot of things as I didn't really care about accuracy (it was just a video game!)

Then I started using real math in my code! :)

Cheers,
Justin

Re:Floating point performance

[QuantumFTL]

Ugh. I was hoping to avoid a long discourse in the theory of computational arithmetic however you leave me no choice.

Point by point:

You did not get it. You are looking at the bit (and byte, and word) as a number. I suggest you look at it as a unit of information. With 64 bits you can only have 2^64 distinct possibilities. If you choose to treat them as numbers -- fine, you only have 2^64 distinct numbers.

Okay FYI I do remember first year discrete math, I have experience with inner details of computer architecturs, I understand two's compliment representation, IEEE floating format, how ALUs work for integer operations etc... I have read Shannon's information theory paper etc. What you are saying doesn't change the reality of the situation though, because you are looking at numbers without regard for the staggering dynamic range in real life scientific computations.

You may split the 64 bits to use some of them to represent the mantissa and some as the exponent. Or -- use all of them to represent the integer number of the smallest units in your application's domain.

Yes and both approaches have very different merits. For most numerically intensive programs, there are very specific requirements for numerical precision. All numerical programs are necessarily approximations to infinite precision arithmetic, and the following constraints are the norm:

• Additions/Subtractions of similar magnitudes must be accurate
• Multiplications/Divisions of very different magnitudes must be accurate.
• Square roots, trigonomic operations, and exponentiation must be accurate.

Now if you examine fixed point arithmetic, you will find that the first criterion is met, however the second and third are not met nearly as well as they are in FP of the same number of bits.

The nature of scientific operations is such that it makes sense to think about numerical calculations in terms of significant digits, due to the nature of error propagation in numerical arithmetic.

The second method, actually, gives you better precision in a controllable (by you) fashion. If the difference between the smallest and the biggest quantity of those minimal units in your application exceeds 64 orders of binary magnitude, than 64 bits is not enough for you -- regardless of whether you use floating or fixed point. You either lose precision (FP) or overflow (int).

It most certainly does not give better precision for the same number of bits used. Think of FP as a lossy compression algorithm. It allows the use of orders of magnitude less number of bits because it alters the density distribution of the representable numbers to meet the above specificiations.

Also you should note that many applications do not have a "basic unit". For instance, what is the "basic unit" of length? What if your "basic unit" of lenght is of a radicaly different exponent than your "basic unit" of energy or "basic unit" of time? In physics applications these basic units are near infinitesimal... we're talking 10^-51 or smaller! Then add to that that astrophysics simulations tend to work on scales that are 10 orders of magnitude greater than 1, you're talking about a dynamic range that is clearly rediculous!

The reason to use FP may be because it is more convenient to think in terms of standard units, rather than the minimal units of the application (its precision). Also, many CPUs have special features allowing to do FP computations really quickly. But it is possible to go without them.

The problem here is that you are thinking in terms of absolute error. In most cases it is the *relative* error that is important, not the absolute. Because of the exponential notation, relative error is minimized for any given number of bits used to represent numbers.

Another issue that you fail to mention is that integer overflow is rediculously easy to run into when using several multiplications in a row.

Re:Floating point performance

[mangu]

some matrices can be inverted with nothing more than a transpose

No, I tested it with a random matrix, as in

void matmuls()
{
int i, j, k, c1, c2, c3;
char opa = 'N', opb = 'T';
float alfa, beta, s, *AT, *BT, *CT;

AT = calloc(N * N, sizeof(float));
BT = calloc(N * N, sizeof(float));
CT = calloc(N * N, sizeof(float));

for (i = 0; i < N * N; i++) {
AT[i] = (float)rand() / (float)RAND_MAX;
BT[i] = (float)rand() / (float)RAND_MAX;
}

c1 = N;
c2 = 1;
c3 = 0;
alfa = 1.0;
beta = 0.0;
gettimeofday(&tv, &tz);
bs = tv.tv_sec;
bu = tv.tv_usec;
sgemm_(&opa, &opb, &c1, &c1, &c1, &alfa, AT, &c1, BT, &c1, &beta, CT, &c1);

gettimeofday(&tv, &tz);
du = tv.tv_usec - bu;
ds = tv.tv_sec - bs;
}

Re:Floating point performance

[alcyon73]

the fpu performance is not too bad, really. I've run some simple tests on the mini clluster and here are some numbers. I have a little test program for parallel that does 6 floating point math operatins inside a loop that iterates 2 billion times. I can run a process on 1-12 cpus, or I can even run multiple copies of the same thing on all the cpus and have just made a little run to show you what I mean. here is the breakdown. on 1 cpu it runs in 386 seconds on 2 cpus it runs in 193 seconds on 4 cpus it runs in 96 seconds on 6 cpus it runs in 64 seconds on 8 cpus it runs in 48 seconds on 10 cpus it runs in 38 seconds on 12 cpus it runs in 32 seconds That is how long it is taking to perform 12 billion calculations. In contrast, a similar non-parallel program running on my single cpu 2.4 GHZ p4 machine takes 265 seconds to run. You can readily see that at around 6 cpu's you are getting into having a pretty fast machine. At 12 cpus it is faster than any production pc you can buy. It is just that simple. This stuff really works. Glen

Re:Floating point performance

[Shadow51]

This will support any 533FSB cpu and it's an ITX! http://www.ibt.ca/mb850.htm

Re:Floating point performance

[Anonymous Coward]

As pointed out by another poster, there are rounding errors in Floating Point instructions.

Unless the project doesn't require a high precision for the final result, I doubt anybody would use floating point extensively.

(Or perhaps it's just me -- I've never trusted floating point data types for anything besides simple arithmetic operations.)

Re:Floating point performance

[alcyon73]

Yes, it has more power than several p4's and is quiet, cooler, and I hope reliable... But it is also fun to build a computer that is faster than anything I could afford to buy ready-made.

Re:Floating point performance

[Doomdark]

The other stuff shows your lack of knowledge of other disciplines by the fact that you think these are computationally expensive tasks.

Hardly so; all three are definitely computationally expensive tasks, unless your definition of expensive means "runs for days on work station level machines". Compiling big projects written in languages like C or heaven forbid, C++ IS still a computationally heavy task. Perhaps not HEAVY heavy as in days of churning, but still minutes on current work station machines. That is, you can do it without multi-CPU/clustering, but it'd be nice bonus to scale... and can fairly easily be parallelized.

As to compression; that too is computationally intensive, but unfortunately for most commonly used algorithms, doesn't scale well. Compressing multiple files can be made to work, though. As with compilation, nothing like simulating nuclear reactions, timewise, but still utilizing CPUs nicely.

Re:Floating point performance

[mi]

Think of FP as a lossy compression algorithm. It allows the use of orders of magnitude less number of bits because it alters the density distribution of the representable numbers to meet the above specificiations.

This makes sense. With integers the density is uniform, which is impediment in some cases, but of help in others. [Any attempt to quantify the number of cases in each group is silly and will reveal nothing, but the attempter's personal bias. With my bias, I'll insist you are underestimating the number of cases, where such uniform distribution of density is useful and desirable.]

However, unless you carefully choose the basic unit, you don't have control over the precision distribution. If most of your computations involve quantities on far edges of (you claimed 20 orders of (decimal?) magnitude) -- you are less precise than you may realize and the (carefull) use of integers may improve your results.

Also you should note that many applications do not have a "basic unit".

Of course, they all have basic units! Usually, it will depend on the application's desired precision.

For instance, what is the "basic unit" of length?

Depends on the application. In yours, it is, probably, some fraction of light year.

What if your "basic unit" of length is of a radicaly different exponent than your "basic unit" of energy or "basic unit" of time?

Who cares? Even if my program operates internally on units as horrible as, say "pounds per square inch" (a.k.a. PSI) -- so be it. If 4.5 Newtons is my basic unit of force I want and the 0.025 meter is as precise as I want the length to be -- fine.

I'll leave the problem of how many bits it takes to achieve multiplication of four numbers of values from 0 to 10^30, base unit 1.

Wait, we started with 20 orders of magnitude. Is it 30 now? Fine, the 128 integers (long long) will be able to store that. But I don't believe, the tasks where so wide-ranging amounts of the same thing are common place (my bias?). Whether you are using floating or fixed point, you are not going to do this easily -- you'll risk losing precision dramaticly, or overflowing. Whichever it is, it is, probably, better to consider modifying the algorithm.

Also, the modern processors (at least -- Intel's) "cheat". Their FPU's internal precision is 80 bit by default (64 significant bits) -- if I'm reading ``icc -help'' output correctly. So they can "promote" the numbers to higher precision when applying precision-losing operations. So, floating point might win. :-)

In most cases it is the *relative* error that is important, not the absolute.

Very valid point. However, sometimes (often?) carefully picking the basic unit and the number of bits it is possible to avoid all computational imprecisions, simply by having more bits left at your disposal, whereas the blind use of the floating will mask them and further compound all other sources of errors (measurements, estimates, &c.)

And I don't even want to think about fixed point division by numbers very close to 0.

You don't need to think about it, because 1 is as close as you can get to zero with integers...

it requires that the basic unit be no larger than the smallest representable number in the floating point system

No it just has to match the smallest reasonably needed by the application -- something, it'll never need a half of. And I urge you to pick such units carefully even if you stick with floating point, because otherwise, even the smallest number in the floating point system might not be small enough at some point, and you will waste a few teracents of taxpayers' money :-)

That being said, I don't think, anyone else reads this, but us. It is hard to justify continuing the thread. Thanks for your input!

Re:Floating point performance

[QuantumFTL]

Wow... here we go again, my old comments in bold, parent in italics, and new comments plain old text. Good luck!

Think of FP as a lossy compression algorithm. It allows the use of orders of magnitude less number of bits because it alters the density distribution of the representable numbers to meet the above specificiations.

This makes sense. With integers the density is uniform, which is impediment in some cases, but of help in others. [Any attempt to quantify the number of cases in each group is silly and will reveal nothing, but the attempter's personal bias. With my bias, I'll insist you are underestimating the number of cases, where such uniform distribution of density is useful and desirable.] However, unless you carefully choose the basic unit, you don't have control over the precision distribution. If most of your computations involve quantities on far edges of (you claimed 20 orders of (decimal?) magnitude) -- you are less precise than you may realize and the (carefull) use of integers may improve your results.

My bias, having to do with this story and the great^n grandfather post is towards numerical cluster computing. Now I might be narrow sighted but having looked at the types of things which are generally done on supercomputing clusters (from books, web sites, and of course observing the supercomputing cluster which I work on, which is #50 in the world, so not exactly small) I see very few applications that do not have the requirements my previous post describes. Lots of multiplication and division by wildly different orders of magnitude... this can be found in almost all branches of science... only discrete math seems to have little of this.

Also it should be noted that through the use of interval arithmetic, you can quickly find the maximum bounds on the error in a given floating point program. This is something that, to my knowlege, fixed point arithmetic cannot support. So, the amount of error is actually quite evident to the experienced numerical computer, and not "more than I realize". Any kind of numerical processing requires people to be careful, fixed point most of all (because most languages don't have fixed point infix operator support, and all shifts etc must be done manually).

Also you should note that many applications do not have a "basic unit". Of course, they all have basic units! Usually, it will depend on the application's desired precision.

A basic unit is an absolute number, however most error bounds must be given as relative error, and if the range of inputs on the program is unknown, then it is impossible to derive a "basic unit". What's more, the numer is generally meaningless in the types of constrained numerical systems which I described. The granularity is usually only meaningful in terms of significant digits, not absolute differential units.

For instance, what is the "basic unit" of length? Depends on the application. In yours, it is, probably, some fraction of light year.

The unit length in our simulation is, I believe, the radius of the neutron star. This can be as little as 10-100 kilometers. It should also be noted that the "base unit" represents the minimum discernable difference in measurements, and therefore must be much much much smaller than that for accurate calculations after many many multiplications.

What if your "basic unit" of length is of a radicaly different exponent than your "basic unit" of energy or "basic unit" of time? Who cares? Even if my program operates internally on units as horrible as, say "pounds per square inch" (a.k.a. PSI) -- so be it. If 4.5 Newtons is my basic unit of force I want and the 0.025 meter is as precise as I want the length to be -- fine.

What do you mean who cares? The quintessential question here is the varying orders of magnitude involved in calcuations! not to mention the fact that to multiply fixed point numbers together you will have to find a common base unit,

Re:Floating point performance

[NateTech]

Hmm... yeah I read it...

Yadda yadda yadda, big numbers, yadda yadda yadda, floatin-point rocks!, yadda yadda yadda, NASA, yadda yadda yadda, neutron star.

Translation: Tax dollars being pissed away on the 50th largest supercomputer and a bunch of theoretical math.

Nice work if you can get it. Some of us have had a few rough years of just trying to get by out here in that "real world" that gives you all that entropy. Sorry if we're bothering you.

Good lord, who are you people and why is 40% of my paycheck being ripped out of my pocket in order to pay your salaries again, please? (Yes I know that NASA's budget is a drop in the bucket in the overall Federal budget, but as you said, it doesn't make Mars any closer to know that Alpha Centauri is so far away! Money wasted is money lost, and if time = money the converse must be true. Great minds wasting time is more of a tradegy than me doing it, for certain.)

Could someone explain in layman's terms the real scientific gains that studying neutron stars at great expense to taxpayers brings that will affect "life on planet Earth" significantly -- in my lifetime or the next two to three generations?

Feel free to throw in any information you might have about why we all pay for Shuttle to be re-flown and ISS to produce virtually zero real science. (The real-world answer to that is probably "Public Relations", I know. And lotteries are taxes for people who are bad at math too.)

Nevermind, that's a topic change. Forget it. In all seriousness, your posts were an interesting read, but I find theoretical studies of things like this a giant waste of resources and so-called brainpower.

The dude at MIT who just created the liquid lens thing -- now that's an engineer I'd like to meet. His creation will directly affect people's quality of living.

Somehow I doubt your 50th largest supercomputer cluster will ever create enough value returned to society to pay for the electricity it uses, let alone the costs of building it, maintaining it, or coding for it. What a waste.

Feel free to educate me -- I'd love to know what I'm truly paying for other than someone who has enough math skills to argue about floating-point calculations on useless data that doesn't lead to any more useful information than to prop up a self-perpetuating PhD paper mill.

Bring it on -- tell me how your supercomputer I paid for is changing my life for the better. Bottom line.

Re:Floating point performance

[Carl T]

Just bored. Slightly less so after having read it, but OTOH now I have less time to do whatever the hell I was supposed to do before 3 pm.

Re:Floating point performance

[Anonymous Coward]

For N of what? 3? I have very different experience -- once you need calculations with a lot of data (no chance to keep everything in caches) using SSE2 results in not so impressive speedups, in order of only some 10%.

BTW From the engineering standpoint P4 is totally unsucessfull processor. Pentium M (Centrino) which produces much less heat, uses much less power, and runs on only 1.6 GHz is for real life calculations certainly faster than that P4 2.4 GHz with all this big fans making all that noise to cool the supid thing off and provide it enough power.

P4 is designed only to market "a lot of gigahertz". And it worked for them -- there's still not much people knowing what's really going on.

Re:Floating point performance

[QuantumFTL]

If you want to see what your tax dollars (and more likely) corporate dollars are doing (the cluster is mostly funded by a donation from intel) check out A list of research done at the cluster. I'm sure that you can find many of those things to be "useful". If you cannot, perhaps you should read up on the science behind those efforts, much of it is very relevant to solving real world engineering problems, and advancing the state of the art.

I would suggest that before you criticize a large research effort that you take a bit of your precious time to research it yourself first.

Also, we happen to live in a democracy, so feel free to vote for candidates that are not interested in promiting science, technology, and a better understanding of the universe. You do vote, right? And of course if you feel that our government doesn't represent you and never will, you can feel free to choose another country of residence...

Speaking of which, I seem to remember this weird network project that was funded by money spent on high energy particle physics. You know, that area of science that is so far from practical that it doesn't produce anything usable... I seem to remember this project being called the World Wide Web... A curiosity at best, we'll never see a return on our investment!

Re:Floating point performance

[some1somewhere]

I have wondered why there are no P4 or AMD mini-ITX mbs around myself.

I can imagine lots of uses for those, as they would basically be full-fledged systems apart from the limited expansion capabilities (but not everyone wants that anyway).

Re:Floating point performance

[NateTech]

Hey fair enough -- your original posts kept using theoretical physics examples and said nothing of the practical applications that are obviously being studied in your list. And the term "NASA" elicits thoughts of deep-space study, not the more practical applications of the cluster you linked to. When you sent the link and it said "Cornell" the lightbulb came on.

Perhaps the real issue is that the deep-space threoretical calculations you mentioned are the ones that are a waste of time on the cluster -- while most of the projects listed on the web link do seem to have a practical "here on Earth" application.

Thanks for taking your time to post a link. You never said WHICH cluster you worked on, that I saw -- so it was a little difficult for me to "use some of my precious time".

I appreciate the response you sent. Thanks. (And yes, I do vote -- and if I saw a cluster being used for ONLY theoretical calculations of things that had no practical value I would probably write a few letters asking why to Congressional folks and include examples of publically-funded clusters that were NOT doing so. No need to kill the funding, just re-allocate the resources to things that have a reasonable application.)

In so far as your comments about the WWW - CERN didn't really come out of my tax dollars anyway, did it? Wrong country?

And HTTP was a huge idea, but the add-ons to HTTP since it was created are mostly an abomination, seeking to turn HTTP into a page-layout language instead of an information-transfer and referral language... the original scientist had a practical solution to a documentation problem.

Using todays Web as an example of how his work "created" that is very incorrect and assumes he wrote furture versions -- the VAST majority of the work on web standards wasn't done by that one person nor was tax money used to fund it. He had the "brilliant" idea, the rest of the world developed it at zero cost out of my pocket to him. His original research was CHEAP and his application was PRACTICAL. Both are good to strive for when using taxpayer/society's resources.

Re:Floating point performance

[mangu]

For N of what? 3?

No, 500.

Re:Floating point performance

[alcyon73]

Let's clarify this. The EPIA has the C3 cpu which runs the FPU at system clock. The EPIA 800 has the eden processor and runs the fpu at 1/2 clock. The EPIA V 8000 A has the C-3 and runs the fpu at full clock. This info comes from the spec sheets on the VIA web site btw. For floating point, the C3 is fine. The Mini Cluster is using the EPIA V 8000 A motherboard, so FPU performance is fine and more than adequate.

Re:Floating point performance

[alcyon73]

Gigabit would be faster, but by the time you add gigabit cards you end up with a lot more cost and a LOT more power use (the machine has 12 nodes). For just running the parallel programs fast ethernet seems fine. MPI is not inordinately chatty on the network. If you do a lot of file I/O you can run into bottlenecks there unless you buffer to memory, or have your file I/O servers on a separate network.

Re:Floating point performance

[alcyon73]

I'd like to know what benchmark you used to yield 6 GFLP on a single 2.4 GHz p4 ? I'd believe 6 GFLP peak, but not sustained. I might accept close to a GFLP sustained. If you are basing that on a theoretical maximum of 6 GFLP, then one could argue that the mini cluster is good for 9.6 GFLP minimum at 12 nodes. Try this test: use your favorite compiler and write a program that does 6 floating point calculations inside a loop that iterates 12 billion times, that comes out to 12 billion calculations. I'm betting your 2.4 GHz p4 is gonna take several minutes to do it. the mini cluster does it in about 30 seconds.

Re:Floating point performance

[alcyon73]

you might try a 1U power supply, it will lstill be big, but at least will only be 1 inch high

Pointy-Haired Boss

[Vexler]

Just imagine Dilbert's boss asking him for a Beowulf cluster.

Kind of like that strip where he (the boss) wanted to have a SQL database in lime.

Re:Pointy-Haired Boss

[ComradeX13]

I think it was mauve. ...Yes, I am pathetic. And I have a photographic memory.

Re:Pointy-Haired Boss

[Magus424]

Actually, he though Mauve had the most RAM :)

Inexpensive for testing purposes,

[Space cowboy]

... but that's about all it'll be useful for. A Nehemiah CPU is really weedy by todays standards, even the 1GHz one is about the same as a 600MH P3. So, he's got 12 of them, which is probably less CPU power than an average dual P4 motherboard...

Still, you can get some stats on how the clustering works, what's the best algorithm for dispersing problems, and these boards are cheap, but that's about the only advantage I can see...

Simon

Re:Inexpensive for testing purposes,

[addaon]

I agree, but that's actually a very interesting use. It also lets you play around with network topologies, and interconnects, and such. And of course, these boards do have one PCI slot, as well as the standard assortment of serial and parallel, so the hardware people can have fun too. For real number crunching? Not a chance. For doing a $2000 prototype, in 15 nodes, of a $50000 50-node cluster? I can't really think of a more flexible, more convenient, or more affordable option. For doing a $1000, 6-node flexible network simulator, purely for education? Also more than worth it, with few other options around.

Re:Inexpensive for testing purposes,

[Pidder]

There are no dual boards for normal P4s since they can't runt in SMP mode. You have to buy Xeons and they arn't exactely cheap. Dual AMD Athlons (the MP model or a modded XP) are your only option for a cheap dual desktop.

Re:Inexpensive for testing purposes,

[Gr8Apes]

Actually wouldn't this be awesome to test multi-process/multi-threaded applications for cheap?

Re:Inexpensive for testing purposes,

[jepaton]

A beowolf of mini-itx boards is probably the cheapest way to get bragging rights. As a practical way of fast and cheap parallel computation they are not.

However, I have purchased three (V10000 boards) thus far and intend to add more to my network as low power (as in Watts) servers.

I worked out that given the power of 10.78W (source: mini-itx.com's power comparison tool) for the V series (probably the one with the slowest CPU in the series, board only), I could save a fortune on electricity compared to a more regular computer.

The electricity company sells electricity at the rate of 0.63 ($1.18) per watt per year. Compared with a standard PC of 100W, I can regain the purchase costs (in savings) of the board and memory within two to three years.

Also, I found rack mount chassis available cheaper than one for a regular sized case. This influenced my decision a little - who doesn't want a network of rack mounted computers?

Overall, because of the low price and low power the mini-itx boards are a no brainer if and only if the CPU power of each computer isn't important.

Jonathan

Re:Inexpensive for testing purposes,

[Brooks Davis]

I actually have a small cluster of similar mini-itx boards (though in 1U chassis) for testing changes on our 160 node FreeBSD cluster. It's especially helpful as our main cluster is 1000 miles away so having a local cluster to use for crash tests is very helpful. I choose these systems because I've got enough powersucking servers on 24/7 at home. The ones I've got consume around 1/8th the power on a standard dual Xeon node at 1/5th the cost. Sure performance sucks, but who cares. It's there to do infrastructure development like testing OS config changes and hacking on the schedular or monitoring tools without breaking the main system.

-- Brooks

Re:Inexpensive for testing purposes,

[slashbofh]

... but that's about all it'll be useful for. A Nehemiah CPU is really weedy by todays standards, even the 1GHz one is about the same as a 600MH P3. So, he's got 12 of them, which is probably less CPU power than an average dual P4 motherboard...

Why is it that most people think that 1 4GHz system is just as fast as 2 2GHz systems? This is the fallacy that never fails to irritate me. The fact is that for a lot of things, the number of machines matters. It's a pipeline, and a CPU can only do one thing at a time. For many application having multiple CPUs that are slower will give you faster response time than a single fast CPU. Of course, most people here don't get that, give it up when trying to talk to the PHB about it.

Re:Inexpensive for testing purposes,

[Space cowboy]

I don't think they equate the same. I said CPU power, not measured performance. I remember sitting on a UK working group panel debating the Block-Synchronous Parallel computing strategy for highly-parallel systems. I was only there because it was good for my CV :-) But I did learn a reasonable amount, all those years ago...

That said, the only time a cluster of servers will do better than a fast single node is when the task divides well over the cluster. Great for clustered webservers, even distributed databases (in fact most server processes), but pretty damn useless if you're trying to do interactive work, or calculate something which *doesn't* divide well. Anything with time-dependent processing (ie: you need the results of the last step to calculate the current one) will run as slow as your fastest node, minus some for overhead...

This doesn't dispute your point of course, but I think the sense of how you said it over-stated the case for the usefulness of the system.

Simon

Re:Inexpensive for testing purposes,

[Anonymous Coward]

he's got 12 of them, which is probably less CPU power than an average dual P4 motherboard

How this got modded as insightful when not only the article tells what it is equivalent to, but also several comments on this article.

You are unworthy of the mod points you got. Not only should you RTFA, you should read the thread. Hell, there are less than 100 at Score 1 at this point.

Re:Inexpensive for testing purposes,

[Anonymous Coward]

But I did learn a reasonable amount, all those years ago...

Yes...you learned to shoot off your mouth without knowing what you are talking about.

RTFA before saying another thing. Pretty please. With sugar on top.

Re:Inexpensive for testing purposes,

[merlin_jim]

So, he's got 12 of them, which is probably less CPU power than an average dual P4 motherboard...

RTFA... he compares performance to 4-6 P4s. He does clustering for a living so I'm assuming he knows how to measure and compare performance at this scale...

Re:Inexpensive for testing purposes,

[EvilTwinSkippy]

Samba file server.

Samba throws open a hell of a lot of threads. (At least on my network of 200 people.) A cluster with each node posessing an external network port would be able to split the threads across dedicated processors. Not too useful for me, but if someone was trying to serve a few thousand clients at a time, that would be useful.

TMYK

Re:Inexpensive for testing purposes,

[Anonymous Coward]

You read his post right ? He said 12 boards, how could he have said 12 without reading the article ?

RTFP before saying another thing. Pretty please. With sugar on top.

Re:Inexpensive for testing purposes,

[alcyon73]

You could use the mini cluster for serious work. read the numbers I posted on the FPU performance thread. The mini cluster is a fast computer. I would not try to replace the company onyx with a 12 node mini cluster. But I'd sure replace any single cpu box you have with it. Glen

Re:Inexpensive for testing purposes,

[alcyon73]

low cost, reasonable usable computing power, and low power dissipation. they are so inexpensive you just use more of them to get the computing power you need, and still end up with less power dissipation.

Seriously, though...

[Short Circuit]

All things considered, what's the cost-per-tflop of that sort of system. These guys don't require as much cooling, space, or whatever else you care to think about.

Has anyone tried stuffing several into a single 1U chassis? For a sort of cluster of clusters?

Re:Seriously, though...

[drinkypoo]

You could get (maybe) 2-4 boards into a deep 1U box. It would be better to use a ~6U box and put lots of them on their sides. You could make a 12" deep 6U with probably 18 or so of these things in it, without having to have cables coming out the front AND back of each box.

Re:Seriously, though...

[Unoti]

I prefer to not use a box. I get great heat dissipation that way. I've got my diskless nodes on a rack I bought for $3.99 at the container store that's used for drying dishes.

Re:Seriously, though...

[fatgav]

The answer to this is...

Yes! (2) and Yes! (4)

Re:Seriously, though...

[Anonymous Coward]

Or you could just use a single 2.5 Ghz P4 board that would blow all of they away. Cheaper, faster, less heat, less weight, less components...

I see no point in this ITX cluster thingamajig.

Re:Seriously, though...

[addaon]

That's okay. I see no point in allowing anonymous posting, but I recognize that some individuals (I hesitate to call them people) do it 'because they can.'

Re:Seriously, though...

[Anonymous Coward]

you mean like this??

Re:Seriously, though...

[Anonymous Coward]

Just because you can walk around and kick people in the nuts doesn't mean you should. Or even that you have a right.

Re:Seriously, though...

[bhtooefr]

A SIX U? No way do you need that much. As long as you're careful, a 4U gives you PLENTY of space. Giving 3" per board, you can put 6 boards wide. Allowing .5" between boards front to back, you need at least 14" deep. So, a 14" deep 4U will fit 12 of these. Make it 5U if you'd feel more comfortable that way, as these puppies don't put out much heat when there's just one, but when there's 12? Cool 12 like you would a single P4.

Re:Seriously, though...

[alcyon73]

The cost is about 1/4 that of using high end pc's Glen

Re:Seriously, though...

[alcyon73]

that is the remarkable thing. The down side is the reliability of the boards in large clusters is unproven, but the power savings and low initial cost are so attractive I predict that it will just be a matter of time before someone builds a large cluster with these things. I can tell you the boards are electrically fragile and wont take a lot of abuse, so you gotta be careful to not kill them. I've already killed one by accidentally reversing the hard drive cable. I've had another spontaneously flake out when I was stress testing the cluster. But this kind of thing is not unexpected when you put 12 new machines in the same system and power them up. Overall it looks promising.

Re:Seriously, though...

[alcyon73]

The home made rack costs about $120 to fab with all new stuff. and that houses all 12 nodes.

Re:Seriously, though...

[really?]

check this out for a pro solution. http://www.inpc.kiev.ua/janus.htm

Re:Seriously, though...

[Nynaeve]

This is an excellent concept. I never understood why people designed clusters to "stack" like this one. Especially if you use higher-power processors, heat dissipation is an important issue.

I've been trying to think of a simple way to arrange them in "slices", but I never thought of a dish rack. Thanks for the idea! :)

Re:Seriously, though...

[JacobKreutzfeld]

I have a VIA EPIA 6000 running in the kitchen; it boots FreeBSD-5.x diskless over the net and gets its filesystems from my main box. It's silent and zero-maintenance. I would think you could do likewise for a cluster.

A pile of diskless, low-power, low-waste-heat boxes, all booting the same boot image diskless from a main (control) box, all reading/writing their data to NFS shares.

Problem with putting a bunch of mobos in a 1-6U box is replacing them and downtime: to add/remove a mobo, you have to drop the entire cluster. If instead you rigged a rack where boards could be slotted in, connect to power and ethernet on the back, then you could add and remove nodes at will. This would allow you to scale up power or replace dead boards. Seems you could almost have the board slotting in (like a NetApp disk drive) getting net directly to the mini-ITX connectors. I haven't worked out how to do this for power; would be nice if the mobo mfgrs would put fingers on the circuit board edges so they could slot into a card connector (uh, like S-100 bus? :-)

shuttle

[trmj]

My favorite use for those mini-itx boards is making a nice shuttle xpc. Cheap, fast gaming computers that are quite portable as well.

The only problem I've found so far is they ony come with nvidia onboard graphics, but that's what the agp slot is for.

Re:shuttle

[trmj]

bah, mod me stupid. I was thinking atx while reading/typing itx.

Re:shuttle

[Anonymous Coward]

Except that for a cluster, mundane onboard graphics is a huge convenience.

Re:shuttle

[Anonymous Coward]

LOL, moderators == teh morons

Of course you're talking about Flex-ATX and not Mini-ITX.

Those shuttle systems will blow away any Mini-ITX board (or group of boards). 8x AGP, full blown 3+ Ghz P4 or Athlon. Not as small though.

Yes, but

[Anonymous Coward]

Where did they find those telephone modems for Mini-ITX form factor?

Imagine...

[Anixamander]

...a new, original joke. Now imagine another one, because that last one wasn't that funny.

In fact, maybe you just aren't that funny. Except in Soviet Russia.

Shit, now I'm doing it.

What a easy to read page...

[Anonymous Coward]

seesh black on grey......

Re:What a easy to read page...

[Anonymous Coward]

It's not. Use a browser that supports CSS, you fucking faggot.

Imagine!

[omar.sahal]

Imagine a beowulf cluster of these....it would gain consciousness and kill us all.

This with Chess

[SamiousHaze]

You know I seriously wonder if this would be a viable option for Computer chess programs (http://www.chessbase.com/newsdetail.asp?newsid=25 ). It certainly is getting cheap to get massive hardware processing power.

Linux support

[AmandaHugginkiss]

The site appears to be down so I can't read if these clusters support Linux. Does anybody know?

Re:Linux support

[Anonymous Coward]

yes they do.

Re:Linux support

[Anonymous Coward]

That would be funny, except its too bad the site is actually running Windows 2000.

Re:Linux support

[Anonymous Coward]

Mini-ITX machines are just a VIA low-power X86 box... they run linux more or less fine... he's using a BSD variant for this project, though.

Some preliminary performance results

[JimmyQS]

We studied 3 mini beowulf systems a while back, here at University of Central Florida, one of which was a mini-ITX beowulf. Here's some info and preliminary results: http://helios.engr.ucf.edu/beowulf/miniature.phtml

Re:Some preliminary performance results

[Anonymous Coward]

interesting! of course, with the new Nehemia C3 CPU's, fan-less at 1GHz, this research is already skewed to being historical :-|

Re:Some preliminary performance results

[tommck]

You really did a Beowulf cluster? Wow. I could never have imagined that!

Re:Some preliminary performance results

[bhtooefr]

RTFA. Both this study and the linked article use Ezra C3s (appears to be the EPIA 8000 on this study, and the linked article used the V8000 (Value)) at 800MHz. They weren't at all impressed by the floating point performance from the 400MHz FPU. There is no 400MHz C3. The only 800MHz C3 CPUs that would halve the FPU speed would be the Samuel2 and Ezra, and the only 800MHz on an EPIA is the Ezra.

Re:Some preliminary performance results

[alcyon73]

I read the link you posted. It was good reading. I'm currently using EPIA V8000A motherboards and have not experienced any network issues, and the fpu seems fine. I hope to find time to build linpack and get a proper benchmark soon as indications are that I'm getting much more than the 1 GFLP that you experienced with the EPIA 800 series boards. Glen

Re:Some preliminary performance results

[alcyon73]

Did you use EDEN processors or C3 ? the EDEN processor in the EPIA 800 mainboard is EDEN and the fpu clocks at 1/2 of system clock (400 MHz). The C3 processor in the EPIA V8000A is a C3 and clocks at system clock (800 Mz) It is understandable that you only got 1 GFLP. Of course, you get much lower power use with the EPIA 800 than with the EPIA V 8000 A board... Go to the VIA web site and read the specs on C3 and EDEN processors. they are different cpus , designed for different tasks. For a computational cluster, the eden core is not a great choice. It would be okay for a database cluster or a web server, etc. If you want to do some numbers and get okay performance, use the C3 or the M series processors.

Re:Some preliminary performance results

[Anonymous Coward]

That must explain the results. We must have made a mistake when ordering. In any case, I'm told that the new VIA boards are even better, with additional built in firewire. They lent themselves so well to the kind of mobile Beowulf we were trying to design because they have so much hardware included. What I was thinking was that they would be good if one wanted to design some kind of multiple screened "Beowulf Desktop." I can't quite think of an application for it yet. But very little work seems to go into making Beowulf computers practical from an everyday user standpoint, and I think with a little creativity, they could be.

I built a fanless ITX system...

[Kenja]

I built one of these, cost me six times as much for one third the power. Unless you NEED a quiet system, dont bother.

Re:I built a fanless ITX system...

[addaon]

Six times as much as what? My entire mini-itx system was under $500, and most of the cost of that was a solid-state drive large enough for a decent linux distribution... and most of the rest was a touch-screen monitor.

Re:I built a fanless ITX system...

[captainclever]

what solid state drive?

i'm looking to get one for a database machine, any pointers?

Thanks!
RJ

Re:I built a fanless ITX system...

[Anonymous Coward]

i'm looking to get one for a database machine, any pointers?

Here's a pointer: 0x00fe358e.
You're welcome!

Re:I built a fanless ITX system...

[addaon]

What characteristics do you want? There are two major types of solid state drives -- battery-backed ram, and flash ram. Both are expensive and small. Only one is fast.

My requirements were essentially (1) no moving parts, (2) affordable if not cheap, and (3) small. I settles on one of these. Debian is fine on 128MB, with 512MB of ram and no swap. Performance, it should be said, sucks. The next step up, for slightly more performance, much more capacity, and a whole lot more cost, is here; but I wanted to avoid using a case that needed drive bays, plus I haven't pockets that deep.

Neither of those is likely to be what you want for a database system, though. You're probably more in the market for a bunch of ram and a battery, unless your primary concern is reliability. If speed is the goal, you want this, or, for more capacity and more money, this. Note that I haven't used either extensively, and in playing around with the rocket a little, I was surprised just how much of a bottleneck PCI became. Also, the rocket doesn't have a battery... so really, unless you have a board with 8GB of memory, and you just need another 8GB of low latency space, it's not such a great deal today.

If you fit into any of the niches above, solid state is wonderful. It's always more expensive than you think, though. And for any database systems I've dealt with, a disk is without question the way to go, perhaps with more memory on board. But if you want any further tips, I'm glad to help.

Re:I built a fanless ITX system...

[Anonymous Coward]

Not a very good pointer :( It isn't even aligned on a (32- or 64-bit) word boundary!

Re:I built a fanless ITX system...

[EvilTwinSkippy]

What do you need the solid state drive for? Drop in a network bootable rom on the network card and netboot the sucker. The Kernel loads of TFTP and the file system over NFS.

Completely diskless, and cheap.

Re:I built a fanless ITX system...

[merlin_jim]

I have a mini-ITX with traditional hard drive, DC-DC power supply, ultra tiny case (Cubid 2677), DVD drive, and Hauppauge capture card for $450.

The capture card itself was $150, so deduct accordingly if you don't need it...

Re:I built a fanless ITX system...

[statusbar]

Not just quiet, but more reliable too as no fans or moving parts are necessary (if you use flash drives}

--jeff++

Re:I built a fanless ITX system...

[statusbar]

You don't even need a network bootable rom. The bios on epia boards support PXE booting.

--jeff++

he's apparently not running..

[Anonymous Coward]

..his webserver on that

Why 12 nodes?

[El]

Why not 16 nodes, or some other power of 2?

Re:Why 12 nodes?

[Anonymous Coward]

If you read the article, his actual target was about 10 nodes, the extra 2 were to allow for hardware failures, which according to him happen VERY frequently in clusters.

Re:Why 12 nodes?

[alcyon73]

Because I started with three nodes, and kept doubling the number of nodes until I had enough power to do some serious work with. I built a cluster with three nodes to prove it would work, then I went to 6 and saw a good speed increase, and went with 12 and got almsot twice the speed as 6.

Re:Why 12 nodes?

[El]

Next time, would you please start with 4 nodes, just to make us anal retentive computer geeks happy? Thanks.

Re:Why 12 nodes?

[alcyon73]

correct, the original design was 10, and I decided that 12 would be needed. Interestingly enough, I accidentally kiiled one node, and another flaked out during stress testing. I've since replaced the two and it is back at capacity with 12 nodes. It took about 30 minutes to replace them and bring it all back online. As I said in another post, I started wit hthree and kept doubling.... anyone ever read A.C> Clarke "Rondevouz with Rama" ? They did everything in threes...

Cool stuff ...

[Lazy Jones]

This rocks - we were considering something similar for our clustering-R&D needs (for trying out new network file systems, failover solutions etc.), but we decided to go with plain P4 barebones instead. They can be stacked nicely, are relatively quiet and the fast CPUs with HT come in handy when you want good latencies at CPU-intensive tasks (dynamic websites etc.).

Here's a picture of our first 4 boxes. The USB stick seen sticking out from one of the boxes is bootable and an excellent replacement for floppy disks...

Re:Cool stuff ...

[Feyr]

a bootable usb stick? i'd assume your board has bios support for booting on usb?

Re:Cool stuff ...

[Lazy Jones]

> i'd assume your board has bios support for booting on usb?

Yes, I guess that most current BIOSes of the newer boards do, especially the consumer-ish stuff. We just used the stock Shuttle XPC with its FlexATX-board.

Re:Cool stuff ...

[Yokaze]

Even easier, the BIOS has support to boot net via DCHP and TFTP.

Re:Cool stuff ...

[Lazy Jones]

Indeed - we'll probably switch to booting over the 'net when we have more boxes or need to try out kernel patches. Right now we boot each node from its 3,5" HDD, the USB stick was/is used for the installation only...

Re:Cool stuff ...

[fo0bar]

You have the same digital camera I have for my "misc" camera (D-340R or similar on the desk), and you have a baytech power strip on top of the machines. I had to stand back for a moment and figure out if this was a picture that *I* took.

Nevertheless, pretty cool...

Hmmm

[captain_craptacular]

There was no cutting or bending involved. All metal bits were simply cut, drilled, and bolted together using 4-40 hardware.

So what was it? No cutting, or cutting?

Re:Hmmm

[cemcnulty]

There's no trick to it - it's just a simple trick.

FLASH...

[Short Circuit]

Ouch...He's using flash as the HD for the computing nodes. Hope they're set to be mounted read-only.

Maybe he should consider PXE instead.

Re:FLASH...

[Anonymous Coward]

IBM MicroDrives. They have a flash-style interface (which is really just an IDE connection with a different connector), but no wear-levelling issues.

Re:FLASH...

[technomancerX]

"He's using flash as the HD for the computing nodes"

Actually, he's not. IBM Micro Drives are not CF, they just have a CF form factor/interface to be compatible with hand held devices. They are hard drives.

Re:FLASH...

[Anonymous Coward]

Try READING the article.

>The computational nodes have 256 MB RAM, each and
>boot from 340 MB IBM microdrives by means of
>compact flash to IDE adapters.

"microdrives"....

Re:FLASH...

[dabadab]

Despite the name, CF is NOT flash memory. The CompactFlash Association's definition is this:
"CompactFlash(R) is a small, removable mass storage device."

So you are correct in noting that he is actually using HDDs, not flash, but in the same time, he is using CompactFlash (BTW the CF pinout is IDE compatible, so to hook up your CF to your IDE bus all you have to do is to manage to connect the wires of the IDE cable and the power cable to the card)

Re:FLASH...

[Anonymous Coward]

What is wrong with CF mounted rw?

Whilst not clustering...

[Alioth]

Whilst not clustering, a good use for these low power systems would be for web hosts or budget dedicated servers. I'm sure a server room full of these would require much less airconditioning (and power) than the typical servers. Many people require dedicated servers for security (they are the only one on the box) and don't require fast FPU performance.

Re:Whilst not clustering...

[RatBastard]

I use a 500Mhz Via Mini-ITX system as a file server on my home network. It's quiet, doesn't eat much electricity and it can easily handle the (small) number of client machines it serves.

Re:Whilst not clustering...

[alcyon73]

Clustering is a fine use for these. They are cheap to buy and operate. You just use more nodes to get the same power.

Re:Whilst not clustering...

[alcyon73]

cool. I saw the aurora in Alaska last year. It was really awesome.

page is gone

[Anonymous Coward]

crash dotted already. way to go guys!

not even a cluster can stand up to the slashdot effect!

Re:page is gone

[Anonymous Coward]

It's an IIS server, and they're statement on Linux is: we run Windows on ours, we've heard Linux will work, these people know how to do it, these are glaring issues that we've heard about, but we don't know about them for sure, so go back to the projects section.

Enough on the Web Server !!!

[IamGarageGuy 2]

Just hit reload! It seems to be holding up just fine, with the occasional bad hit. Gotta give 'em a break, this is /. after all.

Interesting Parts of the Article Text

[Anonymous Coward]

Introduction

I built a Mini-ITX based massively parallel cluster named PROTEUS. I have 12 nodes using VIA EPIA V8000, 800 MHz motherboards. The little machine is running FreeBSD 4.8, and MPICH 1.2.5.2. Troubles installing and configuring Free BSD and MPICH were few. In fact, there were no major issues with either FreeBSD or MPICH.

The construction is simple and inexpensive. The motherboards were stacked using threaded aluminum standoffs and then mounted on aluminum plates. Two stacks of three motherboards were assembled into each rack. Diagonal stiffeners were fabricated from aluminum angle stock to reduce flexing of the rack assembly.

The controlling node has a 160 GB ATA-133 HDD, and the computational nodes use 340 MB IBM microdrives in compact flash to IDE adapters. For file I/O, the computational nodes mount a partition on the controlling node's hard drive by means of a network file system mount point.

Each motherboard is powered by a Morex DC-DC converter, and the entire cluster is powered by a rather large 12V DC switching power supply.

With the exception of the metalwork, power wiring, and power/reset switching, everything is off the shelf.

At present, the idle power consumption is about 140 Watts (for 12 nodes) with peaks estimated at around 200 Watts. The machine runs cool and quiet. The controlling node has 256 MB RAM , and an 160 GB ATA 133 IDE hard disk drive. The computational nodes have 256 MB RAM, each and boot from 340 MB IBM microdrives by means of compact flash to IDE adapters. The computational nodes mount /usr on the controlling node via NFS, for storage and to allow for a very simple configuration. No official benchmarks have been run, but for simple computational tasks the mini cluster appears to be faster than four 2.4 GHz pentium 4 mcahines used in parallel, at a fraction of the cost and power use.

Power and Cooling

Mini-ITX boards have very low power dissipation as compared to most motherboard/cpu combination in popular use today. This means that a Mini-ITX cluster with as many as 16 nodes won't need special air conditioning. Low power dissipation also means low power use, so you can use a single inexpensive UPS to provide clean AC power for the nodes.

In contrast, a 12-16 node cluster built with Intel or AMD processors will generate enough heat that you will likely need heavy duty air conditioning. Additionally, you will need adequate electrical power to deliver the 2-3 kilowatts peak load that your 12 node PC cluster will require. Plan on having higher than average utility bills if you use PC's...

Hardware Construction

The cluster is built in two nearly identical racks. Each rack has two stacks of three motherboards and dc-dc converters mounted on aluminum standoffs.

The compact flash adapters used to mount the microdrives are also in stacks of three. Each stack of boards is mounted on a 7 inch by 10 inch 0.0625 thick 6061-T6 aluminum plate as are the microdrive stacks. There are seven metal plates in all, in each rack.

Software Configuration

The cluster consists of a controlling node, with a large capacity hard drive, and several computational nodes, each with their own hard disk drive (these hard drives can be smaller).

The software which performs the parallelization (MPI) is installed on the controlling node, and the computational nodes mount a shared directory on the controlling node via NFS.

Communications between the nodes is established via rsh by MPI, and shared files are found via the mounted NFS file system,

The networking is fast ethernet (100 Mbit) and makes use of a fast ethernet switch switch. Gigbit ethernet is faster (and better for fast file I/O) but 100 Mbit ethernet is quite adequate for number crunching.

The version of MPI used is mpich-1.2.5.2

The Operating system for the controlling node and all the computational nodes is FreeBSD MINI 4.8 RELEASE

FreeBSD has moved forward a bit since I began buildi

Silly question, I know, but...

[pegr]

Just what do you do with such a thing? I don't mean obvious commercial uses, but as a home-bound geek, what reason can I use to justify this to my wife?

Re:Silly question, I know, but...

[Anonymous Coward]

You can justify it by saying "Bitch, get back in the kitchen!"

No sex leads to built up frustration, which leads to bursts of anger. Poor guy.

Re:Silly question, I know, but...

[Anonymous Coward]

Easy. its a server for anything in any combination. with a kvm and the pxe boot setup decently you can test things accross any number of operating systems from half the stack while the other half proxies, downloads, archives, tracks logs, serves media etc. Mine runs Gentoo and a few flavors of windows from 4 160 + 4 80 gig drives. its been the perfect playground to learn linux system administration with. In the box as shown here i'm generating less heat than with a high power box doing the same, get total reliability and all with virtually no noise. (yes, thats a UPS in the case)

Test Text.

[F34nor]

I built a Mini-ITX based massively parallel cluster named PROTEUS. I have 12 nodes using VIA EPIA V8000, 800 MHz motherboards. The little machine is running FreeBSD 4.8, and MPICH 1.2.5.2. Troubles installing and configuring Free BSD and MPICH were few. In fact, there were no major issues with either FreeBSD or MPICH.

The construction is simple and inexpensive. The motherboards were stacked using threaded aluminum standoffs and then mounted on aluminum plates. Two stacks of three motherboards were assembled into each rack. Diagonal stiffeners were fabricated from aluminum angle stock to reduce flexing of the rack assembly.

The controlling node has a 160 GB ATA-133 HDD, and the computational nodes use 340 MB IBM microdrives in compact flash to IDE adapters. For file I/O, the computational nodes mount a partition on the controlling node's hard drive by means of a network file system mount point.

Each motherboard is powered by a Morex DC-DC converter, and the entire cluster is powered by a rather large 12V DC switching power supply.

With the exception of the metalwork, power wiring, and power/reset switching, everything is off the shelf.

At present, the idle power consumption is about 140 Watts (for 12 nodes) with peaks estimated at around 200 Watts. The machine runs cool and quiet. The controlling node has 256 MB RAM , and an 160 GB ATA 133 IDE hard disk drive. The computational nodes have 256 MB RAM, each and boot from 340 MB IBM microdrives by means of compact flash to IDE adapters. The computational nodes mount /usr on the controlling node via NFS, for storage and to allow for a very simple configuration. No official benchmarks have been run, but for simple computational tasks the mini cluster appears to be faster than four 2.4 GHz pentium 4 mcahines used in parallel, at a fraction of the cost and power use.

Power and Cooling

Mini-ITX boards have very low power dissipation as compared to most motherboard/cpu combination in popular use today. This means that a Mini-ITX cluster with as many as 16 nodes won't need special air conditioning. Low power dissipation also means low power use, so you can use a single inexpensive UPS to provide clean AC power for the nodes.

In contrast, a 12-16 node cluster built with Intel or AMD processors will generate enough heat that you will likely need heavy duty air conditioning. Additionally, you will need adequate electrical power to deliver the 2-3 kilowatts peak load that your 12 node PC cluster will require. Plan on having higher than average utility bills if you use PC's...

Hardware Construction

The cluster is built in two nearly identical racks. Each rack has two stacks of three motherboards and dc-dc converters mounted on aluminum standoffs.

The compact flash adapters used to mount the microdrives are also in stacks of three. Each stack of boards is mounted on a 7 inch by 10 inch 0.0625 thick 6061-T6 aluminum plate as are the microdrive stacks. There are seven metal plates in all, in each rack.

The top cover plate has the mounting bracket for the 6 on/off/reset switches.

The plate below it is home to the power distribution terminal block. The power delivery cable for each rack is heavy duty 14 gauge stranded wire with pvc insulation. The power cabling from the terminal strip to each of the dc-dc converters is 18 gauge stranded pvc insulated hookup wire. The wiring for the power/reset switches is 24 gauge stranded, pvc insulated wire.

The top rack houses nodes one through six (node one is the controlling node). The bottom plate of the top rack also houses the 160 GB ATA-133 hard disk drive used by the controlling node. All other nodes make use of the IBM microdrives. Node number three has a spare compact flash adapter which can be used to duplicate microdrives for easy node maintenance.

The disk drive and power cabling to the motherboards was dressed as was sanely possible on the back panel. The liberal use of nylon cable ties helps reduce the ten

Let's save space...

[ComradeX13]

I, for one, welcome a beowulf cluster of our new mini-ITX overlords in Soviet Russia.

Re:Let's save space...

[Anonymous Coward]

Hrmm...

Wouldn't that be:

In Soviet Russia, a Beowulf cluster of our new mini-ITX overlords welcomes me?

Is that link hosted on there cluster?

[unixsource]

I hope not. 'There are too many connected users. Please try again later.' Sucks.

Re:Is that link hosted on there cluster?

[unixsource]

I need to read these posts before I hit submit. Sorry for the bad grammar.

Dissipation

[Bleeblah]

I especially like the bit about the peak 200W power dissipation.
Their web server is dissipating smoke and silicon goo!

Re:Dissipation

[alcyon73]

yah it is low power. It is more like 240 watts total absolute max, and arounfd 120 watts idle.

Just because you can...

[caffeinefiend]

Yet another example of why you shouldn't do everything that you can do! These puppies aren't exactly famous for their flop-per-dollar ratio. In truthfully, it would be more efficient ( and cost effective) to make the cluster out of PIIIs. Anyhow, I'm off to go cluster a few toaster ovens, I hear that they offer a great delicious to efficiency ratio. Chris

Re:Just because you can...

[addaon]

Yes, but see my comment here, and its parent comment, for why this is an interesting option, even if not the best performance.

Re:Just because you can...

[enkidu]

Efficiency can take on many meanings depending on what your objective function looks like. Undoubtedly you can get more FLOP for the $. But that isn't why you'd use a setup like this. I could also see a use for this if you were trying to optimize for FLOPs / Watt. Or FLOPs / dB. Or FLOPs / ft^3. This kind of a computing setup seems to be optimized for low-power, low noise, low-maintenance and small space uses. I can definitely envision scenarios where you could optimally arrive at such a setup.

Re:Just because you can...

[EvilTwinSkippy]

Hmmm. Like robotics?

Re:Just because you can...

[kirkjobsluder]

I think that you have to factor in the full cost over time, rather than just the cost of the motherboard/cpu bundle. Once you add in all the extra hardware to keep the whole thing cool, the beefier power supplies, and the additional power cosumption over time, this starts to look like a viable alternative.

Re:Just because you can...

[alcyon73]

That is precisely what it is optimized for. I wanted a cluster with none of the headaches of a room full of computers. This one could be better in terms of servicability, but it is not too bad overall. I can reinstall a ndoe in a couple of minutes, but I have to disassemble the rack to replace a dead motherboard.

those could be handy for ...

[enrico_suave]

that flash mob cluster party coming up.

"outlet? no, no thank you... I'm good for quiet a few hours on this motorcycle battery right here"

or whatnot... of course my favorite use of mini-itx boards is to build PVR's and HTPC's with them...

*shrug*

e.

Re:those could be handy for ...

[bhtooefr]

Your computer must be at least 1.3 GHZ Pentium III/AMD equivalent or better with 256MB of RAM, a 100 Base-T network connection and a CD-ROM.

256MB of RAM? Yep!
100 Base-T network connection? Got that right here!
CD-ROM? Should I put the slimline in, or go with the full size?
1.3GHz Pentium III/AMD equivalent or better? Hmm... we've got a problem. Umm... can you fire up WCPUID and set the multiplier to ten? *inserts Live CD, reboots, and sees VIA C3 Nehemiah 1000MHz* WHAT THE? I SET IT TO 1333MHz! What was that? Oh, a VIA C3 has to be running at 2.2GHz to qualify? SHIT!

Re:those could be handy for ...

[bhtooefr]

Before I'm modded troll, realize that this will probably be VERY FPU intensive, and I was not knocking it's usefulness in a cluster. I was simply referring to the fact that for FPU, it would take about 2.2GHz to be equivalent to a 1.3GHz P3/Athlon.

slashdotted already?

[cetan]

sheesh that didn't take long.

I managed to get it mirrored here:
page 1:
http://www.phule.net/mirrors/mini-itx-cluster.html
page 2:
http://www.phule.net/mirrors/mini-itx-cluster2.htm l
page 3:
http://www.phule.net/mirrors/mini-itx-cluster3.htm l

Re:slashdotted already?

[Anonymous Coward]

Why would it take long? Do you know how much time it takes after a story is posted to Slashdot before ~1000 people start clicking the link? That's right, about 30 seconds.

more information ...

[Lazy Jones]

Oh, I forgot: each of these boxes contains a 2,8GHz P4 Northwood CPU (200/800MHz FSB), 1GB RAM. The Shuttle barebone used is the S75G2 and one of the reasons we chose it was that it has an on-board gigabit ethernet adapter. The CPU cooler that came with it is also very interesting - it uses a rather unique design with a heatpipe ...

Enough on the Web Server !!!-Group participation

[Anonymous Coward]

"Too Many Users"

A cluster of Slashdotters has hit their website.

Cost Ratio

[thomas536]

Does anybody know the price per flop of this setup? I'm curious what types of setups have the best ratio?

Re:Cost Ratio

[thomas536]

Ok, RTFA...

"No official benchmarks..."

Does anybody have some info on making the fastest cluster for the cheapest cost? Especially for somebody just wanting to get their feet wet?

Re:Cost Ratio

[pla]

Does anybody have some info on making the fastest cluster for the cheapest cost? Especially for somebody just wanting to get their feet wet?

Checking Pricewatch just now...

Pick up as many Althon XP 1700 combo MBs ($66) as you want, Throw in 256MB of PC2700 ($26), add a 30GB IDE HDD ($25), 6ft of cat-5 cable ($3, far less if you get a spool and make them yourself), and get a 400W PS ($10).

You'll also need a few switches ($15 for 8-port 10/100 FD) (NOT hubs, you definitely want the full N-to-N FD bandwidth). Now, go to Home Depot and buy a bag of a few hundred nylon standoffs ($10?) and a few dozen metal standoffs of the same size ($5?). I'll assume you have a working monitor and keyboard that you can temporarily stick on each node to set up the bios (ie, to tell it it doesn't have a keyboard or monitor <G>).

Mount the MBs to each other, with as many standoffs between each as it takes to give the CPU fan a bit of clearance. Use the metal standoffs on the grounding holes (Usually the center rear ones).

Lay the entire hideous tower-o'-motherboards on its side (make sure to put it on something reasonably non-conductive and non-static-prone - smooth wood will suffice, carpet or styrofoam or stainless steel would not). Now just hook up the PSs in a neat row behind the MBs, and the HDDs on top of the PSs. You should also run a wire between the PS case, the HDD case, and the shared ground on the MBs (the metal standoffs), just for safety, but you can probably get away with not doing so (I wouldn't recommend skipping it, for just the cost of a few feet of cheap copper wire).

Poof, you have a working cluster for $50-$100 plus $133 per node. You can probably do a tad better due to quantity, and if you can find any of the above used (HDDs or memory pulls), all the better.

Also, one alternative that may save you quite a lot of effort and a few bucks - Instead of one HDD per node, get a second network card with a bootrom ($12) for each machine. Use one NIC exclusively for the clustering app, and the other (the one with the bootrom) for booting, accessing a Samba share, ssh'ing in, and any other incidental network activity.

You might also want to spring for a few box fans (the kind you stick in a window in the summer), since this sucker will keep your computer room nice and toasty (think "one hairdryer on high per 10 nodes).

Re:Cost Ratio

[alcyon73]

It is looking like it will be around $1 per MFLP lets wait and see what the official benchmarks come out to be.

(ot) the sig:

[janbjurstrom]

Hehe, nice one.

Totally unralated: About you're current .sig, "Do not taunt Happy Fun Ball(TM)". Was it you who had that awesome "The instructions SPECIFICALLY SAID ... DO NOT TAUNT HAPPY FUN BALL!" (or something to that effect), I've seen around some time ago?

Did I miss the part

[stratjakt]

Where he give some stats/benchmarks/observations as to how this thing performs, and if it was at all worth it?

Those 800s are so gutless they can just barely playback a DVD, I can't see exactly what application you would give this cluster.

Re:Did I miss the part

[alcyon73]

First of all I used EPIA V8000A mainbords, not the EPIA 800. The performance is more than adequate. Secondsly, there will likely be another article giving the final , offical stats on performance power and cost per mflp since these seem to be the figures people want to see. From my experience using the mini cluster so far, it seems to be fast and effective, and quite usable for serious work. I don't thikn I'd want to put 200 users on it and run the human genome project code on it, but I'd sure not hesitate to use it for parallel type applications that would normally bog down may single cpu boxes.

USB Stick

[Anonymous Coward]

Not to be too off topic here, but what is the brand / model of the USB drive you're using to boot with?

Clickable link...

[Anonymous Coward]

Would it Kill ya? Clicable link

Re:Clickable link...

[Anonymous Coward]

I ran out of href's you insensitive clod!

Re:Clickable link...

[Epistax]

OR you could use FireFox and not give a damn. I just highlighted the text and opened it in a new tab.

While you're creating links for those with IE, maybe you should create text versions of pages for people stuck on Lynx.

Obligatory Kent Brockman...

[Anonymous Coward]

I, for one, welcome our new humor-defining overlord.

heh I got obligatory for ya

[aztektum]

They musta been runnin' their webserver on one!

*ba dum ch*

/.ed all ready

[Anonymous Coward]

Ouch, slashdotted all ready.. oh well, anyone have a copy of the text?

Mini-ITX? Bah! Nano-ITX!!!

[Cpt_Kirks]

I can't wait for the new, smaller nano-itx boards to come out. 4.5" on a side, 1GHZ CPU and draws 7 watts. I got an email from VIA claiming they will be released in April.

MB, slim DVD and laptop HD in a case the size of a large paperback book!

It will make my "K-Mart Toolbox Mini-ITX PVR" look like a full tower in comparison!

Re:Mini-ITX? Bah! Nano-ITX!!!

[Anonymous Coward]

Yes I'm also waiting to get my hands on nano ITX for some projects. I have not seen evidence of a working system on mini-itx.com but I beleive some developers do have them. The missing bit is they have no onboard LAN/ethernet so its not a single board component for a cluster.

Re:Mini-ITX? Bah! Nano-ITX!!!

[Cpt_Kirks]

The boards I'vs seen do have an ethernet port...

Sounds Fun

[RAMMS EIN]

I have been thinking about this lately. I get disgusted by the fanns everywhere (especially since the one in my laptop makes an awful amount of noise sometimes and still doesn't prevent the beast from overheating and shutting down). Aside from being noisy, computers have way more CPU power than I need, and cost more than I am willing to spend. And they suck up a lot of power. (Some might add that they take a lot of space.)

I think all of these could be solved at once. What if someone built low-power, low-noise, and low-cost computer, good enough for running light office applications? I don't mean OpenOffice, but rather lightweight programs that implement the functionality people use _without_ the bloat. My 486 handles email just fine and the WYSIWYG word processors were once satisfied with a first-generation Pentium (and even these were already bloated).

Current PDAs have more than enough processing power to handle those tasks, and I've noticed that company's like gumstix build and sell devices almost like what I have in mind (the gumstix don't seem to have display connectors, though). Hey, these machines could actually be portable and have a really decent battery life (more than a full working day); that would be a killer!

Am I just daydreaming here or are others with me? Maybe you know of devices that do this job? Someone recommended Sharp's Zaurus, which is excellent, but still rather more expensive than what I have in mind.

Re:Sounds Fun

[EvilTwinSkippy]

I'm with you.

I live off my laptop. Fully tweaked I can my Vaio gentoo to run for 3 1/2 hours on a charge. That's fine for a bus ride, but for a long car trip I need a power inverter. I would really like to have something that could last a plane ride, or be like my Clie and just get plugged in and backed up at night.

I don't have extravigent needs. Emacs, TCL, MySQL, occasional OpenOffice. All right, I do bust out Win4Lin to do work with MicroSlog Office and AutoCAD occasionally. I used to run Win98 on a 200Mhz laptop, and that was acceptable.

My present laptop runs at 800Mhz (500 on battery.) It's boatloads more than I need. What would make me happy is more battery life, and top shut up that damn fan.

Re:Sounds Fun

[Anonymous Coward]

I get disgusted by the fanns everywhere (especially since the one in my laptop makes an awful amount of noise sometimes and still doesn't prevent the beast from overheating and shutting down). Aside from being noisy, computers have way more CPU power than I need, and cost more than I am willing to spend. And they suck up a lot of power. (Some might add that they take a lot of space.

How about this instead?

Massively Parallel

[Seanasy]

I built a Mini-ITX based massively parallel cluster named PROTEUS. I have 12 nodes using VIA EPIA V8000, 800 MHz motherboards.

I'd just like to point out that 12 nodes is not "massively parallel."

Re:Massively Parallel

[dbc]

nit picker...

So, let's help him out. Would it be moderately parallel? Slightly parallel? Hmmm... what is opposite of "massive"... light-weight parallel? Hey... there's a buzzword with pointy-hairedness all over it: "light-weight parallel processing"

As for me, I'm massively serial here

Re:Massively Parallel

[imsabbel]

leave it away. Or do you know a cluster that isnt parallel?
12Nodes is as small as clusters can be, so "a small cluster" would be a better description than "a massivly parallel cluster".

(But it really looks cool, and 12v dc via lab psu is cool,too.)

Re:Massively Parallel

[alcyon73]

sure it is massively parallel. You can have a massively parallel cluster of two nodes. It's not a massive cluster. It is massively parallel... it depends totally on being parallel for speed, and it is capable of communicating from the contronlling node to all the computational nodes and between the computational nodes.

I want one!

[Dracolytch]

Hey gang,
I'd really like to build and use my own cluster, as I do have some MPI experience from college. The only question is: What are they good for at home? I just can't justify the expense to myself without figuring out what I could really do with a cluster if I built one.

Ideas saught!

~D

Re:I want one!

[puhuri]

If you just want to experiment MPI, just get some old Pentiums (without monitors), you may get those for free. If you find some brand-name desktop PCs, they are quite quiet compared to no-name minitowers. They will use more power than mini-ITXs, but cost is much lower.

About year ago I got 20 Pentium 133s for 400 euros with 32 MB of memory each and 10BaseT network card. They were just perfect to emulate web clients -- the key was to have identical computers so I could just mirror hard drives.

Re:I want one!

[TCM]

Heck, if you really just want to study the software-side of it why not use VMware on a fat box to simulate a dozen boxes? I heard the VMware 4.5 beta even has PXE support..

Why Microdrives?

[pantycrickets]

Why would he use microdrives with a CF to IDE converter? Why not 2.5" drives? You could probably get larger, faster disks for the same price.

Re:Why Microdrives?

[Gothmolly]

But they'd produce more noise and heat, and cost more power. Besides, nowadays 340 Microdrives are cheap.

Re:Why Microdrives?

[pantycrickets]

But they'd produce more noise and heat, and cost more power.

I didn't really think of the power consumption aspect. That makes sense.

Re:Why Microdrives?

[alcyon73]

I used them because they were small and used less power than ordinary hard drives. Laptop drives would be a good middle-of-the-road compromise on size and power use. The microdrives are pretty slow.. so I used nfs to mount /usr so I would have a nice place to put things. Using drives to boot the nodes made for a much simpler, and more sanitary configuration than using pxeboot. You do get off cheaper using pxeboot (no drives to buy) at a slight penalty in memory and network resoruces.

Re:Why Microdrives?

[toddestan]

Aren't those microdrives not very reliable? I suppose in a cluster they don't get bumped around a lot, but I don't know if they are up to being run for days and days continously. Then again if one fails I suppose it's probably a quick and cheap replacement.

Re:Why Microdrives?

[alcyon73]

The original 6 microdrives have been up for a few months with no problems. They are little bitty drives, so don't work them to death and they will probably last a long time.

A beowulf cluster of FreeBSD machines?

[Anonymous Coward]

I had no idea all this stuff ran on FreeBSD, but apparently it does. A bit of googling turned up an article on a pretty decent size cluster running FreeBSD at aerospace corporation, and other clusters running FreeBSD too. What with Mac OS X being used widely for clusters, and FreeBSD, it sounds like Linux is no longer the only name in the game. So question: do people consider FreeBSD or OS X clusters also to be Beowulf clusters, or is there some other name?

Re:A beowulf cluster of FreeBSD machines?

[alcyon73]

Oh yah. FreeBSD does a fine job for clustering. Linux is a little more popular for clustering due to the fact that places like NASA tooled up for parallel on Linux a long time ago. Now you have an assortment of commercially available tools for parallel computing that are available for Linux. Less so, for FBSD, when it comes to commercial applications, but I've yet to find a properly written tar that I could not extract and build on FBSD. All of the essential clustering tools work on FBSD with no problems.

Re:Nothing on CNN

[Anonymous Coward]

If you haven't noticed, CNN never published any articles that could cause negative images of US.

BBC is independent. Really. CNN is not. REALLY.

Oh my goodness!

[Greedo]

You are someone who actually did imagine a beowulf cluster of those!

Around here, that must make you a god!

Re:Oh my goodness!

[JimmyQS]

Thanks for your appreciation. :) We're all just trying to increase our cool factor. I submitted this page a few months ago and it got rejected. Not sure why.

Eh, that's nothing

[Jonboy X]

Why, just the other day, I chained together two dozen Power Wheels (tm) Mini-ATV's dogsled-style and ran some benchmarks. The cluster generated an awe-inspiring 49.4 foot-pounds of torque, not far off from the 56 ft-lbs generated by a '93 Dodge Neon LSi. The tough part was not so much acquiring all the Power Wheels as training the monkey pilots to hit the accelerator on command...

Re:Eh, that's nothing

[Jonboy X]

Okay, let's try this again...

My previous post on this subject was an example of sarcasm, and the "offtopic" mod was an example of someone obviously not getting the point. I was drawing an analogy between the original article and a ridiculous fictional situation in an attempt to illustrate why the original article was so ridiculous. This is also sometimes referred to as an "allegory". Here's a mapping between elements in my post and the original article, for the less-clueful.

Mini-ATX -> Mini-ATV (henceforth to be referred to as "Clue #1")
Neon -> P4 (tho the article compared the cluster to 4 P4's)
dogsled team -> computer cluster
torque -> computing power
monkeys -> software
ridiculous, pointless comparison -> same thing

C'mon man, if you don't get the joke, leave it for someone who does.

Re:Eh, that's nothing

[gralem]

MOD PARENT UP--That's damn funny!!!

Re:Eh, that's nothing

[Anonymous Coward]

Tsk tsk... you almost had it right, but he '93 Dodge Neon part gave it away. If memory serves, the Neon didn't come out until 94 or 95.

Re:Eh, that's nothing

[Jonboy X]

D'oh, caught me. I meant the '93 Geo Metro LSi, which eventually got itself a fourth cylinder to become the Neon we know and love.

Re:Eh, that's nothing

[bhima]

Jonboy

You obviously lost many of our fellow slashdoters by mentioning objects that do not exist in their mothers' basement.

Apparently original = offtopic

Pity though, it's the funniest comment in the article

Green Destiny

[kvn]

Yes, the Green Destiny system at Los Alamos is a 'cluster of clusters', as you say...

Don't use FP for money

[bluGill]

Perhaps many people would insist on using FP dollars and cents, but those people are fools, and it is very easy to part them with their money. Just make sure all the rounding errors work out in your favor, which isn't hard if you have access to their accounts.

Yeah I know that for small numbers FP has no rounding errors, but that doesn't last long.

Re:Don't use FP for money

[dborelli]

Think "Office Space" people...

Peter Gibbons: Um, the 7-Eleven, right? You take a penny from the tray.
Joanna: From the crippled children?
Peter Gibbons: No, that's the jar. I'm talking about the tray, the pennies for everybody.

I can't help myself...

[fyzix]

In mother Russia, beowulf clusters you!

One use I thought of right away...

[temojen]

It would be quite useful for a university with an undergraduate course in high performance computing to have their own little NoRMA cluster to play with without the space, heat, and power consumption of a supercomputer.

Let the researchers use the real supercomputer, but the undergraduates can still play with message passing parallel algorythms to their hearts content.

microdrive?

[aonaran]

My only question is why microdrives?
Wouldn't laptop drives be more economical? it's not like he'd save any space, they are the same height, just longer than microdrives.

The standard IDE to laptop IDE adapters are dirt cheap and you get more byte for your buck in laptop drives than in microdrives.

Re:microdrive?

[johnw]

Why any kind of drive? These boards include a NetBoot ROM with the in-built NIC so they work very nicely without a local hard disc drive. For a cluster you're surely better off with one (or two if you need redundancy) boot location rather than having to update one boot device per node.

Re:microdrive?

[NormalVisual]

I'd think mostly power consumption.

I wonder too

[Atario]

Well...I'd be able to get major numbers in SETI@Home...um...

Video encoding? (Now, where'd I put that parallel-processing version of AVISynth?)

Rent it out to a university?

Program it to solve chess and leave it going till it does?

Get a decent frame rate in any FPS, once and for all? (Note to self: develop parallel-processing graphics card.)

Re:I wonder too

[merlin_jim]

Video encoding? (Now, where'd I put that parallel-processing version of AVISynth?)

dyne:bolic is a Live CD distribution, very small, that can be PXE boot, with full audio/video capture/editing/processing/streaming capabilities plus the usual suite of tools, a few games and whatnot... and is auto-clustering on a private network.

Stop with the obligitory jokes!

[Anonymous Coward]

I'll cover them all here.

In soviet russia, beowulf cluster is almost as hot as the grits natalie portman is in.

The only question is, can it run Linux, because *BSD is dying.

Yet another cluster...

[cr0sh]

It is cool to look at (although he should have gone with aluminum bar stock or something else for the diagonal brace - the L-stock looks ghetto), but yet again it is a cluster which seems to not have a use!

I mean, nowhere did I see what he plans on using this thing for. Distributed database system? Near real-time POV rendering (yeah, right - not with those underpowered and too-few nodes)? Protein folding calculations? Something else?

This isn't unique - not by a long shot. The original SETI Stompmonster did it cleaner (can't find any links to it, though), and at least it was for SETI@Home crunching.

I have thought many times about building such a machine myself, but I have yet to come up with a good reason to have one (other than to ooh and ahh over). SETI@Home and Folding@Home (is there such a thing), and distributed POV are cool applications of such a machine, but what other real-world, useful applications for the home user exist for a parallel processing machine (that would conceivably justify the expense)? I am not saying they don't exist (hell, the learning experience of setting up such a machine would be worthwhile), but they are definitely few.

I guess I am just the kind of person who looks for a problem before coming up with a solution to solve it (hammer and nails, hammer and nails)...

Re:Yet another cluster...

[Anonymous Coward]

Shame on you, you shouldn't need any more lofty reason than to 'learn'.

Hell I know people who spent $40K on a ^%$#@ university degree and learned less in 4 years than this guy learned on this project, and he did it in less than $4K.

Since when was it a requirement to have a 'lofty purpose' for something before you do it? Isn't learning enough?

Re:Yet another cluster...

[alcyon73]

1) bar stock was not as stiff as angle stock. 2) bar stock would be heavier. 3) It is more about low cost than looks, so it does not matter if the diagonal stiffeners look funny. All they have to do is keep the chassis from falling apart.

Re:Yet another cluster...

[alcyon73]

I concur. I learned a lot from building this beastie. The project was fun, I am getting great results , and I learned a new trick or two, and yes it was all done for really cheap (for a cluster project). Total cost was under $3500 and it could be done a lot cheaper if you wanted to use pxeboot.

Re:Yet another cluster...

[alcyon73]

Oh I had a problem to solve. The problem was how to put a cluster in an efficiency apartment without turning it into a sauna and/or going broke. The mini itx boards turned out to be the only solution that was workable. In particular, the EPIA V8000A board looked like the lowest cost and power use, with best usable performance ratio, so that is what I went with. I was originally going to use the M series boards and use firewire at 400 Mbit for the network, but working support for firewire at full bandwidth was questinable at the time I started the project, and the cost of the M series boards went up a lot, so the EPIA V8000A proved to be the best deal at the time.

HA-Cluster on Mini-itx boards

[DeBaas]

Not as impressive as their performance cluster, but perhaps interesting as well, we build a High Availability cluster more than a year ago based on mini-itx boards: HA-cluster
It was used for demonstration, but the mini-itx machines are still used quite a bit for testing etc.

I have a dream

[HangingChad]

I built a Mini-ITX based massively parallel cluster named PROTEUS. I have 12 nodes using VIA EPIA V8000, 800 MHz motherboards.

I had this vision of builidng a solar powered cluster. But couldn't think of anything useful to do with it at home. If I could get it to render video, then it's useful. Solar power...just because I can. 200 watts isn't that expensive of a battery pack. Could probably calculate the panel size and cost but not until I can think of a reason to build one.

My understanding...

[porkchop_d_clown]

As I understand it, "Beowulf" implies certain kernel changes to allow all the machines to share a common address space.

It's more likely the machines are organized as a grid, using MPI for interprocess communications.

I could, of course, be utterly, abjectly, wrong.

Re:My understanding...

[alcyon73]

that is not my understanding, tho the definition might bechanging slowly over time. My understanding of a beowulf cluster definition is that it is a bunch of off the shelf computers wortking in parallel, and furhter definition of it seems fuzzy. I'm also aware that there is much disagreement, because there are so many ways to build and configure parallel computers

Why this particular set of software / booting?

[merlin_jim]

I mean, those IBM 340 MB microdrives aren't really that cheap... you can get full size hard drives for the same price...

I've always wondered; why not PXE boot something like this? Set your node controller to also do DHCP and you're set.

While you're at it, use the CL version for the controller which has two network cards and build a NATTING firewall into the node controller too. Then you have a plug-in appliance that doesn't interfere with your network topology at all. PXE boot it and the motherboards will only need RAM.

The board he used is available for $99 with proc. A stick of 256 is probably around $20.

The best price froogle would give me on the drives he's using is $60, and they're prone to wear and tear.

Add in the $10 CF-IDE adapter and the drive is %60 of the cost of the motherboard itself...

Hell if you don't want the network bogged down with a bunch of PXE booting nodes all the time, just get cheap CD drives and put dyne:bolic on it, which does automagic clustering...

Personally, if I were to do it, I'd set dynebolic to PXE boot, get a huge stack of motherboards and RAM, and do it that way. Then adding/changing nodes is relatively simple... IIRC, they're even factory set to try PXE booting if no IDE devices are found...

The only other change I would make would be to ditch the 16-port switch... move to 4-ports, connect those to a 4-port with gigabit uplink, and connect that to a gigabit switch. Of course at this point I'm talking about really scaling the cluster up, to a few hundred nodes or so. At that point I'd stop using a mini-ITX board for my node controller and go with a motherboard with a bit more juice behind it, dual procs, RAID 0/1, the whole shebang...

Now if only I had a couple grand burning a hole in my pocket... speaking of which:

motherboard: $100
RAM: $20
DC-DC converter: $30
CF adapter: $10
Microdrive: $60

Total: $220
Total PXE booter: $150
Savings: 30%

So, not counting the costs of cabinets, power rectifier/UPS, wiring, network gear, and labor, you can increase the size of your cluster by %30 for the same cost, just for setting up PXE boot...

Re:Why this particular set of software / booting?

[alcyon73]

Because the config is simple, and the drives are small, and use very low power. Also, nobody has tried it before to my knowledge

Re:Why this particular set of software / booting?

[alcyon73]

Well, you actually save about 25 % using pxeboot on this cluster. I was more interested in simplicity, small size, and low power use than anything.

Re:Nothing on CNN

[Anonymous Coward]

Not true, I read the BBC and I know.

Not that they aren't marxists, however.

Re:Please explain this

[Anonymous Coward]

sure - it's off topic but...

re: Ok. Please someone explain this [bbc.co.uk]. Has BBC really gone bonkers or is it for real?

Pentagon officials have confirmed that Guantanamo detainees may still be kept in detention, even if they are found not guilty by a military tribunal.

Guantanamo is not a part of the United States and the US laws and constitution don't apply there so anyone detained there really has no resource to any entity. For all intents and purposes these people could disappear without a trace and there is nothing under US law or even International Law that could be done to prevent this or to seek compensation/redress for this.

Welcome to the 21st century and welcome to your new overlords...

I'm surprised no one linked to these already

[Anonymous Coward]

http://www.9grid.net/news/index.html
The pictures are cool.

Scary coincidence...

[WeblionX]

"I built a Mini-ITX based massively parallel cluster named PROTEUS."

That's the same name of the new server at school that at least my account and a shared drive happens to be moved to. The HDD size looks about right, too.

Re:Scary coincidence...

[alcyon73]

That is especially scarey , considering I chose the name from the old SciFi movie "Daemon Seed"

Re:Nothing on CNN

[Anonymous Coward]

apparently you (and your parent) have no idea what does "marxist" mean. it's not an offensive word, like, say, "asshole" - it's quite well defined. marxism, same as other politic/economic ways of thinking is structured in a certain way, and I truly don't see any marxist-like propaganda, whatever, on BBC. learn about things you want to talk, thank you.

Waste of solid states

[Nicolas MONNET]

Don't use solid state drives for whole filesystems, just use them as the journal device -- it's possible on both ext3 and ReiserFS. Remember to configure data journalling in.

Re:Waste of solid states

[addaon]

Again, it all depends on what your goals are. Flash makes a pretty poor journaling device, do to it's limited write count, and battery-backed ram is only useful for journaling if you really, really trust that battery. My goals could not be accomplished with any not-solid-state disk, so I made the choice I did.

Beowulf. The REAL deal.

[madpierre]

LO, praise of the prowess of people-kings
of spear-armed Danes, in days long sped,
we have heard, and what honor the athelings won!
Oft Scyld the Scefing from squadroned foes,
from many a tribe, the mead-bench tore,
awing the earls. Since erst he lay
friendless, a foundling, fate repaid him:
for he waxed under welkin, in wealth he throve,
till before him the folk, both far and near,
who house by the whale-path, heard his mandate,
gave him gifts: a good king he!
To him an heir was afterward born,
a son in his halls, whom heaven sent
to favor the folk, feeling their woe
that erst they had lacked an earl for leader
so long a while; the Lord endowed him,
the Wielder of Wonder, with world's renown.
Famed was this Beowulf

Sample from the Project Gutenberg Text of Beowulf.

Why not do yourself a favour and download it. Classic stuff. :)

FPUs of the future? Re:Floating point performance

[ziegast]

Mod me....

Informative:

If you're looking for a small form factor for high-end processors, you will likely find future products using the picoBTX form factor. The motherboard layout provides better cooling for hot processors that mini-ITX can't address. Here's a summary of the BTX form factors from Anandtech.

Interesting:

Has anyone figured out how to use the floating point power in their graphics cards for non-video applicaitons? Those things are becoming powerful that they use their own heat sinks. Just like we had floating point chips for the 486SX series, perhaps it will be more cost-effective and power-effective in the future to separate commodity, low-cost, and low-power I/O processing from floating point processing.

If graphics card developers start thinking of their cards as being more like floating point coprocessors and less like device controllers, they can help drive future floating point computing and leave traditional central processors to manage memory and I/O.

Redundant:

Imagine a beowulf cluster of video cards!

-ez

Disclaimer: I use a 500MHz Celeron on my desk and a 300MHz laptop at home. I'm not a luddite - I just don't utilize a 3D "rich" graphical environment to surf the web, create documents, and manage computers.

I wonder too-Parallel learning.

[Anonymous Coward]

"Video encoding? (Now, where'd I put that parallel-processing version of AVISynth?)"

Poor FPU's make that impossible. However with the economy being soft, these are ideal for people who want to increase their skill set. Without pushing too hard their economic budget.

Have different kind of servers. Study different networking topologies. Help the OSS community with their compiling tasks. Make multiple LiveCD distro's.

If FPU's are improved? Then PovRay, or other parallel tasks. Circuit design, weather modeling, etc.

Please mod parent down...

[imsabbel]

he doesnt know jack about math or algorithms.

btw: tell me just HOW you would compress a huge file with this cluster? distributed BWT and huffman? over 100mbit?

Re:Please mod parent down...

[mi]

he doesnt know jack about math or algorithms.

:-)

btw: tell me just HOW you would compress a huge file with this cluster?

My humble inability to tell you this woudn't allow a person so skilled in "math and algorithms" as to point out others' ignorance of the subjects to conclude, that such compressions are impossible. But, for the sake of argument, I can offer the following thoughts...

Both libz and libbz2 require some initialization and then expect blocks after blocks of data to be compressed and return the results of compression.

It seems quite possible to -- after the initialization completes once -- send the result of initialization to all of the cluster members, and tell them to pick different parts of a file being compressed (over NFS) and send back (or write down over NFS) the results. The results will be out of order (possibly), so the "main" process will have to assemble them at the end. But it is doable. The only potential roadblock is that the library(ies) preserve some state of the stream between invocations. But my -- admittedly quick -- glimpse of the bzip2 suggests, the state is only altered at the beginning, end, or in case of an error.

over 100mbit?

100mbit is often faster than a harddrive. It is not neccessarily the bottleneck at all. If the controlling program does not read the whole file, each block of it will only be read once by a particular cluster member tasked with compressing that block.

And use it for what?

[Sabalon]

Pardon my ignorance... but just what sort of tasks are suited for clusters? Is the software specially written to work in a cluster environment?

I understand that it is to get a lot more performance by having tons of cheap computers rather than one large system. And I know how renderfarms work - each machine gets a frame to chew on.

Is clustered stuff just math written to be processed distributedly as well?

Re:And use it for what?

[alcyon73]

MPI puts the parallel tasking in the hands of the programmer. In essence, you pass a variable to the nodes and all the nodes chew on it and the results are then collected and hopefully assembled into something usable. It is recursive, and a bit indeterminant because you don't know exactly when a particular node is going to finish it's share of the computing tasks. Alternatively uou can bypass MPI and just run individual programs on all the nodes. Some people even go so far as to send source code from the controlling node to the computational nodes where it is then compiled and executed. Another common thing to do is to just use shell scripts to talk to a bunch of computers on the network and send different parts of the job to variaous machines. There are lots of ways to do things in parallel. The definition of "cluster" is pretty fuzzy.

I knew it...

[n0dez]

They are running FreeBSD :P

Re:I knew it...

[alcyon73]

not they, me. and you betcha It's FreeBSD Baby!

I want my SBC !!!!

[gelfling]

Hey I used to have a 10 slot SBC case with 10 2-way Intel 512MB 1GB DASD SBCs in it.

Flops/$$$ = free

[poptones]

As a green geek I can't resist pointing out this merit: with only a 200W power dissipation this would be "home friendly" even in a non air conditioned house during the hot Mississippi summers. And with only a 200W PEAK draw, the entire system could be powered by a single PV panel and one or two storage batteries. Trade the "high quality UPS" for a couple of batteries and a PV panel (or cheaper still if you're in the midwest or near a coastline, a windmill) and you have a cluster that could run without any "store bought" AC at all.

Re:FPUs of the future? Re:Floating point performan

[Hoser McMoose]

Pico-BTX is still a bit bigger than Mini-ITX and it only solves the cooling issue when used in a fairly standard (albeit small) desktop case.

Has anyone figured out how to use the floating point power in their graphics cards for non-video applicaitons?

Actually yes, some people have started doing floating point in graphics cards. Check out what the GP GPU people are doing in this regard. The latest and greatest GPUs from nVidia and ATI actually offers a LOT of theoretical processing power, significantly more than the latest P4 or Opteron chips (or IBM PPC 970/Apple G5 chips for that matter). They also offer a whole boatload of memory bandwidht, another very important consideration for FP work.

There are some downsides though. First off GPUs are essentially vector processors. This is not necessarily a bad thing (the fastest supercomputer in the world, Earth Simulator, uses vector processors), but some code does not lend itself well to vector processing and other code needs to be reworked. Perhaps a bigger concern is that GPUs only handle single-precision floating point. This is perfectly acceptable for the purpose they were designed for, but they could cause problems for more general pupose FP work which sometimes requires double precision FP. There's also the issue of non-ECC memory on the cards.

Still, they are interesting from an academic point of view. Of course, on the other hand, CPUs have started incorporating some vector processing aspects of their own. SSE/SSE2 and Altivec are designed as vector processing units as well.

GO AEGIS!!

[kc8tad]

Go AEGIS (Greg)!! You made it to slashdot man sweet :) ~TAD

The real Beowolf an interesting parallel of merit!

[Vinny the goldfish]

Beowolf - The Epic Poem vs Beowolf the Epic mini-ITX cluster. The epic poem Beowulf describes the most heroic man of the Anglo-Saxon times. The hero, Beowulf, is a seemingly invincible person with all the extraordinary traits required of a hero.(I.e mini-ITX clusters)He is able to use his super-human physical strength and courage to put his people before himself. He encounters hideous monsters and the most ferocious of beasts(microsoft equiped single processor computers) but he never fears the threat of death. His leadership skills are superb and he is even able to boast about all his achievements. Beowulf is the ultimate epic hero who risks his life countless times for immortal glory and for the good of others.(overheating) Beowulf is a hero in the eyes of his fellow men through his amazing physical strength(multi giga-flopidness). He fought in numerous battles and returned victorious from all but his last. In his argument with Unferth, Beowulf explains the reason he "lost" a simple swimming match with his youthful opponent Brecca(p4 processor). Not only had Beowulf been swimming for seven nights, he had also stopped to kill nine sea creatures in the depths of the ocean. Beowulf is also strong enough to kill the monster Grendel, who has been terrorizing the Danes for twelve years, with his bare hands by ripping off his arm. When Beowulf is fighting Grendel's mother, who is seeking revenge on her son's death, he is able to slay her by slashing the monster's neck with a Giant's sword that can only be lifted by a person as strong as Beowulf. When he chops off her head, he carries it from the ocean with ease, but it takes four men to lift and carry it back to Herot mead-hall. This strength is a key trait of Beowulf's heroism. The most heroic of traits within Beowulf is that he is not afraid to die. He set a noble example for all human beings relaying the necessity of brotherhood and friendship. Beowulf is most definitely an epic hero of epic proportions.

in the topic of clustering...

[ShadowRage]

I'm interested in it, can anyone give me any good links on clustering with linux?

Power.

[Absurd Being]

Your P4 uses what, >300W? This cluster has a peak load of 200W. Plus you can do more varieties of hardware interfacing at once. That's a reason to build this cluster, if you don't find that clustering things because you can to be a good enough reason.

Re:Power.

[mangu]

Your P4 uses what, >300W? This cluster has a peak load of 200W.

Well, I just applied my, admittedly imprecise, clamp ammeter to the power cable, and got ~2 amps @ 120 V = 240 W. Which means, 240W/6Gflops = 40W/Gflop. That cluster has 200W/3.6 Gflops = 55.555... W/Gflop. Slightly worse...

I admit that hardware interfacing is getting to be a problem for us hobbyists, since the demise of the ISA bus, but I have been able to get along with the parallel interface. I just hope the USB interface doesn't get too popular too soon!

It's called volume..

[xtal]

When it gets down to the real big numbers, semiconductors are just sand and some big fancy photograph machines. Unfortunately, those are big numbers.

Often it is cheaper to get the much higher end than lower power smaller volumes. A 5.25" 300Mhz Geode SBC factor computer - not many people are aware of the industrial PC world - will run you around $300-400US for a brand name with a great MTBF.

My whole athlon system with a geforce ti cost $400us.

picking a nit

[Anonymous Coward]

those are pretty io bound tasks if you ask me. computationally expensive is the wrong way to think about it.

Re:picking a nit

[mi]

Ever timed bzip2? It is certainly CPU bound -- even on the fastest processors...

ITX boards

[Shadow51]

Why did they use VIA chips for a cluster? I doesn't seem like a good choice considering the whole point is performance. IBT (link:www.ibt.ca) sells 533FSB P4 boards in mini-ITX form factor! They also sell 800FSB P4 boards with ISA slots for some reason? It's kind of cool if you got some old hardware lying around.

Re:ITX boards

[alcyon73]

Low power use, so you can run a cluster at home without a huge utility bill or needing special air conditioning.

Use 2 Way dish access

[bigmoosie]

Get a 2 way sat dish and run your webserver off the grid. I wonder what sort of gov't/business incentives this sort of aragnement(sp) can generate.

Other than a latency issue it would be a very cool setup.

Initially solar power is rather expensive per kwh but if the system is maintained very well it will last and last and the long run cost curve will go down per kwh.

Re:Image a Beowulf Cluster of these babies!

[alcyon73]

Yes, linux is a popular os for clusters. I found FreeBSD to work a little better for me.

At last...

[slashnull]

Finally...no more imagining beowulf clusters. This one has been realized.

Waaaiiit a Second! Credit where Credit is Due...

[Anonymous Coward]

here you will find pictures of the LoneWoulf. It has been in service for over a year.

Burt who runs the site has had specs and the same lousy pictures I'm sharing with you now since february of last year.. yet when Burt shared the idea with Glenn they somehow forgot to attribute the original designer who probably does have a copyright on the powerbus and skewers.

Please mod me up and thanks for giving these guys a good slashdotting!

Inexpensive but perhaps not the best choice

[burnin1965]

Not to take away from what this guy has done because it is quite interesting, but I wanted to play around with clustering and was considering the mini-ITX platform as I have a few of them in use now and they are great little machines.

However, when calculating the cost to performance ratio you are better off with a cheap all in one motherboard and AMD processors.

I went with $65 shuttle all in one motherboards and $80 AMD XP2200+ CPUs. The final total was more like $1400 but it is close to your $1000 mark with much greater processing power.

www.aros.net/~burnin/ironcluster.html

burnin

Re:Nothing on CNN

[Anonymous Coward]

"Oh yeah? What about the hordes of trotskyists and marxists who keep on pushing their revolution to the masses every friggin' night."

What the hell are you trying to say?

and the palestinian issue..

Palestinian issue is a lot of more complicated than you think. If you take a look at the casualty statistics, there are more palestinian civilians killed than israelis. How is that possible? There is also much more palestinian property destroyed than israeli. How is that possible?

I'm absolutely sure that you can't find those statistics at CNN. This is a basic example of why I concider CNN to be really, really biased.

Re:Waaaiiit a Second! Credit where Credit is Due..

[alcyon73]

That is really cool. There is alslo a fellow who built a 16 node cluster and was getting quite good performance from it. I do not have the url for his page, maybe he will come forward with a link.

Re:Waaaiiit a Second! Credit where Credit is Due..

[alcyon73]

Don't be such an asshat.