10 Years of Beowulf Clustering

Quirk writes "Wired News has a blurb celebrating the 10th birthday of the Beowulf cluster. Attendees recalled the initial fear and loathing the Beowulf project had to overcome. The Beowulf project takes its name from an epic poem penned circa 1000 A.D."

Comments

Strange...

[PapayaSF]

...it feels like I've been reading Beowulf cluster jokes on Slashdot for longer than that....

Re:Strange...

[Richard Dick Head]

I know what you mean...it actually seemed like a Beowulf cluster of 10th anniversaries.

It seems like a good time..,

[Omestes]

Imagine a beowolf cluster of x

Couldn't resist.

Next up the 10th aniversary of the
1. xxx
2. ?
3. PROFIT!

model of buisness.

Oh Lord..

[Anonymous Coward]

Might as well have posted "Hey, practice your cliche overuse in here!"

I'm gonna get modded down for this, but...

[Quattro Vezina]

I, for one, welcome our new beowulf cluster overlords.

Re:I'm gonna get modded down for this, but...

[grolschie]

Yeah but does it run Linux?

Re:I'm gonna get modded down for this, but...

[StandardDeviant]

In Soviet Russia, Linux runs YOU!

(actually, with the increasing sophistication of implanted prosthetics like pacemakers and such... someday it very well might. "root@heart# echo 60 > /proc/sys/heart/bpm" )

Imagine /. Without Beowulf

[SEWilco]

In pointing out Beowulf clustering, I might have started the "Imagine a Beowulf cluster..." catchphrase.

I'm somewhat sorry.
Now go and cluster something.

Finally!!

[kb9vcr]

I was just thinking, we never talk about beowolf custers anymore!!

I wonder, can we beowolf custer a beowulf cluster?! ;)

Re:Finally!!

[CoolGuySteve]

I wonder, can we beowolf custer a beowulf cluster?! ;)

You might be interested in grid computing, in which a group of academics with heads too big for their common good decide not to build one fucking huge computer in one place, but instead spend all their grant money on fiber transceivers and other equipment that can transfer at a few dozen GBit between far less powerful clusters. Whenever you see a grid built with modern equipment (rather than one that strings together a few older machines), it means the people involved at some level were playing politics so that they could 'me too' their department into owning a piece of it.

I once watched some of this process in motion, which helped to smack down a far more sensical and quite impressive machine proposal, and found the whole thing to be entirely retarded.

Re:Finally!!

[samhalliday]

You might be interested in grid computing, in which a group of academics with heads too big for their common good

you seem to have overlooked the fact that these people are indeed academics... the people who push boundaries and bring about new ways of doing things. grid computing isn't working now, but when the technology is in place for it, it will be revolutionary. these kind of ideas don't work first time round, and certainly don't fix themselves overnight.

your ignorance to the sheer amount of information processing which will be required by, for example the new generation of projects at CERN, is perhaps the reason why you do not see the need for grid computing.

I once watched some of this process in motion, which helped to smack down a far more sensical and quite impressive machine proposal, and found the whole thing to be entirely retarded.

that is no fault of grid computing... the blame must be placed upon the persons who chose the wrong solution. grid computing is nowhere near ready to be used. the only people who should be playign with it should be people who wish to aid the current research. it is equivalent to running a beta kernel with debian unstable on a production server. if they had work to be done, then a cluster would have been a more sensible option.

Here's your BIG chance!

[FunWithHeadlines]

Ah, finally, a story where all those 'imagine a Beowulf cluster of...' comments actually would be ON TOPIC! Naturally that means there won't be that many, other than comments such as this one that is commenting on such comments...a meta-comment about Beowulf clusters. Speaking of which, can you imagine a meta-cluster...oh never mind...

Re:Here's your BIG chance!

[NeoSkandranon]

Don't you mean "Can you imagine a beowulf cluster of meta-comments?"

At Last

[Spencerian]

Imagine a Slashdot topic full of Beowulf clusters...

Re:At Last

[DarkBlackFox]

Imagine a Slashdot topic full of Beowulf cluster comments!

Re:At Last

[Spellbinder]

Imagine a Beowulf cluster of moderators, modding those posts down

Re:At Last

[Anonymous Coward]

It'd probably still return 503 errors...

Yeah, but...

[pyrrhonist]

Okay, now imagine a Beowulf cluster of Beowulf cluster anniversaries. Oh, man, I just blew my mind!

Sad

[maelstrom]

The blurb doesn't explain what a beowulf cluster is, but explains what the origin of the name is. Are slashdot readers that ignorant?

Re:Sad

[Daniel Dvorkin]

[shrug] Different standards for ignorance; most people would recognize the origin of the name*, but wouldn't have any idea what a B. cluster is. And by /. standards, that's igorance. I'm a believer in the "one culture," myself -- if you don't recognize either, there's a serious gap in your knowledge.

*I may be giving "most people" too much credit, of course.

Re:Sad

[NotAJock]

Becuase of your low UID, I'm guessing that you aren't asking what a Beowulf cluster is - so you're right on the mark with your comment. So for those Slashdotters who don't actually know what a Beowulf cluster is, here's my blurb (feel free to add/subtract/etc - this is paraphrased from a longer piece I wrote some time ago):

Beowulf aims at minimizing computation time. One option for reducing the processing time of a program is to divide it into independent sub-tasks that can be processed by different CPUs. When the results of these sub-tasks are available, they can be returned to one of the processors for final processing. It is possible to use Ethernet transfers to extend this strategy across multiple computers. This is how Beowulf works: divide programs into many parts that are executed by many CPUs all of which transfer their data and instructions via Ethernet.

Re:Obligatory comment

[Omestes]

imagine a beowulf cluster of bad /. beowulf cluster jokes...

Gr... I've already done one, I feel dirty...

Seriously though (and offtopic, but more intelligent), anyone ever noticed that only geeks have this bad penchant for inside jokes? BAD inside jokes. REALLY bad... etc... Why is this, are us geeks to preocupied to come up with new jokes? And the sad thing is that everyonce in awhile I still giggle at them, even if I can see the "In soviet russia, a beowulf clusters you (with hot grits and natalie portman!)" joke coming. WHY?!

I got out of my social/group psych class with a thesis on /.ers and MUDers, and still I'm as confused as ever.

reminds me of my first cluster project...

[morcheeba]

In 1995, I put together an animation of a satellite my company was working on. I used POVray running on DOS, and wrote a little pair of programs that would hand off render-jobs to different computers. I used 16 computers (mostly P60's) lying around the office to render about 400 frames total. The whole job took about 35 hours of wall time, which was important because I had only three 1/2 days to tweak my small demo & make a final rendering.

I didn't know network programming, so all communication was through read/writing a few networked control files. One acted as a semaphore - if you had sucessfully written your computer ID to it, you could modify the main to-do-list file. One specialized computer was assigned the task of copying the finshed files onto my new 810MB laptop's hard drive; otherwise the file server didn't have enough space for all the .TGA files.

It was a fun project & I've got it included on my resume. Today it sounds kindof trivial, so I've had to explain that general-purpose clustering tools weren't available then. I guess Beouwulf beat me to it by a year (and a zillion-fold on capability), so I was wrong. Information travelled so much slower those days...

Re:reminds me of my first cluster project...

[drinkypoo]

I'm not sure when it was written, but DQS (the distributed queueing system) was around in 1996, and I don't believe it was especially new then. this document alleges that the whole clustering thing began at NASA in 1994. Apparently FSU developed DQS starting in 1992 but I don't know when the first release was.

I used to work for a company called silicon engineering in scotts valley, ca - formerly sequoia semiconductor and last I heard they were part of creative labs called creative silicon or something. We used DQS to schedule jobs for IC simulation for testing.

Of course, DQS doesn't work on DOS, it's a Unix-type program. For anything that can be batched (like rendering frames in POVray) it can be amazingly slick and it takes relatively little configuration. It has a keen little program that watches when your system is idle and signals the queue master to feed it jobs, which is an X client. Using DQS and the berkeley automounter it was possible to easily submit jobs and not care where they ran, for instance we had the paths set up such that the same commands worked on SunOS4 and SunOS5 so verilog was always in the same place, et cetera.

DQS also has a parallel make utility, which I never used, because I hardly ever compiled anything. :)

Beowulf - the name

[NotAJock]

Just because I'm an English (the language) geek, here's the lowdown on the name 'Beowulf':

The Beowulf poem is the oldest known epic in the Anglo-Saxon language (that's like, early english). It's about the life of a king of the "Geats" called Beowulf. It starts off as him as a young rash figher and follows through to his death after fighting a dragon.

Damn great story - there's probably loads of online texts (like this one?). The only surviving manuscript (possibly the only one ever written) is in the British Library. You can go there and see it.

Re:Beowulf - the name

[meringuoid]

An English geke I am, and /. I do rede forsooth. Spelynge is not thy strength, nor ys thyn gramyre of the fynest; yet it is for the tales of hackerye that I remayneth, and ye troles I do endure willynglye.

LO, praise of the prowess of moderators of troll..

[j1m 5n0w]

LO, praise of the prowess of moderators of troll-armed Slashdotters, in days long sped, we have heard, and what honor the NASA reasearchers won!

Just think...

[Beek]

Once this article gets duped a few times, we'll have a Beowulf cluster of stories.

Alternate Source for name "Beowulf"

[Nova Express]

I am given to understand that the name came not firsthand from the epic poem, but second-hand via Niven, Pournelle & Barnes' SF novel The Legacy of Hereot. In it were some big, nasty monsters they dubbed "grendels," which they then proceeded to wipe, only to find out that the adult form was what was keeping the numbers of the immature form under control, resulting in a massed attacked by thousands of "baby grendels." I remember reading that this was what inspired the Beowulf name in a FAQ several years ago. Anyone know if it's true?

"Joke" posts

[Anonymous Coward]

That's it. Every one of you boneheads is now my enemy.

And 5 1/2 years of jokes

[veg_all]

A quick and sloppy google suggests that this is the first "imagine a..." comment (bottom of the page), though it's possible that the joke predates google search capability since Beowolf clusters and slashdot are both older than google. Still, the fact that it's not used as a joke, and the fact that it got a 1 rating (while, inexplicably, all those repeat jokes get modded to the stratosphere these days) lends an air of authenticity to the claim, lacadasical though the research may be.

Re:And 5 1/2 years of jokes

[the pickle]

More to the point, this comment seems to be the one, from February of 1999.

Where were *you* on 25 February 1999?

p

Beowulf

[Mahtan]

Beowulf was a poem Tolkien really liked. A lotta the Rohirrim are based off of it.

Re:If I was going to post a LINUX STORY...

[templest]

Imagine all the beowulf clusters running linux out there, now imagine the beowulf cluster of lawyers you'd need to get all those shut down. No company could afford a small army of lawyers like that, that's suicidal, that's like... making a beowulf cluster of a million Apple RAID servers, good way to go bankrupt. :P

Imagine a beowulf cluster of all the computers of people that read slashdot everyday, damn, you could hack the planet with that thing in 0.832595 seconds.

epic versus clustering

[danny]

The top result on a Google search for "beowulf" is about clustering, but 8 of the top 10 are about the epic. And I doubt there'll ever be nearly as many books and articles written about the clustering system as about the poem.

Danny.

The Year?

[swingerman]

"...name from an epic poem penned circa 1000 A.D."

That should be A.D. 1000. A.D. means "Anno Domini", Latin for "in the year of Our Lord", and should properly *precede* the year. B.C., on the other hand means "Before Christ" and properly *follows* the year.

Imagine a....

[ErichTheWebGuy]

Imagine a beowulf cluster of people telling you to RTFA!

Seriously though, the linked article goes into a lot of detail about what a beowulf cluster is.

Wow

[Pan T. Hose]

I just imagined it...

Imagine

[Waffle Iron]

Imagine there's a cluster
Of 64 G5s
Or 128 Opterons
Between them only CAT5
Imagine all the boxen
Benching Quake FPS...

Imagine no shared memory
It isn't hard to do
Nothing to spinlock or thrash for
No cache coherence too
Imagine all the boxen
Crunching local tasks...

Imagine there's no mainframes
I wonder if you can
No need for Crays or S/390s
A cluster loosely bound
Imagine all the boxen
Sharing all the LAN...

You may say that I'm a uniprocessor
But I'm not the only one
I hope someday you'll join us
And we'll simulate nukes as one

Question

[ErikZ]

One thing that the article mentions, but doesn't get into. That scientists were hostile to the concept at first.

Can anyone who was around at the time shed some light on this?

Re:Question

[Anonymous Coward]

I happen to know Dr. Sterling, I worked with him at CACR/Caltech. To answer your question, from what I heard and so on:

1. commodity hardware had some serious problems back then compared to today (so did big iron but that was always hush-hush and part of the mystique of running on big iron), and the software was a lot rougher, did less compared to the commercial Unixes, etc.

2. Running on big iron was, well, not only accepted, but sexy, it was the thing to do. to be able to say you are running your code on the new super fast SGI, Intel or IBM is much more uber (at the time anyway) than to say yeah we ran it on a bunch of desktop PCs in a cluster. Took awhile for the non-computer tinkerer scientists to accept the whole thing.

3. Even today, some jobs run better on a shared memory big iron machine than parallelized out on a
cluster using message passing. That was true then, also.

4. Scientists don't always think of a very scalable (i.e. increasingly faster potentially) thing like a cluster as good. Gone are the days where you can start a run and disappear to go mountain climbing or sailing for two weeks. At best a long simulation (or portion thereof) buys you a long weekend. The faster the number-crunching goes the more work you have to do, the more results are expected faster, etc, etc. A vicious circle really. If this concept shocks you, pretend grants, academia and all of it has no politics, only wonderful breakneck pursuit of fact and conquering new horizons...

That's just my take. Oh yeah, and highly unlikely to get funding or donations back then from the big companies of equipment to build a cheap alternative to their flagship HPC products... They didn't exactly encourage that sort of thing.

Beowulf in L33t/aolspeak..

[actor_au]

I would like to apologise first off for creating this monster.
I did it out of boredom more than anything else a long time back..

Its only the first three verses' however, I wasn't insane enough to do the whole thing.

D00d u no Cl4n Danish?
Cpl 98, 99
Th3y b l33t
[danish]Shield < [danish]sheaf
pwnd in Q3 & C$
Romero was like "D00d??! WTF?"
Bcase he w4z working solo.
He wez top of teh ladder
and teh Chatroom was like
"D00d??! WTF?"
He pwnz

[danish]shield N3\/\/ a n00b
T0ok nOOb to clan danish
n00b was fr0n CowboyNeil
k3pt the Cl4n as 1
M4de them > l33t
[danish]grain w4a his 'nic
He was l33t.
in the CPL
4ll n00bs shuld l34rn
Spread the lewt
when u are a noob
so wh3n ur 19
teh clan wi1l still restecpt U
wh3n j00 at w4r
a man is l33t
if he is k00l
on any of teh servers(xsept teh telstra 1s)

[danish]shield was pwnd after his 19th
w3nt to uni stoned but still with l33t
teh clan danish gave him pr0n
which was sw33t
1n his gargae sat
a l33t machin3
old-school but built 4 teh road
There [danish]shield slept
next 2 teh handbrake
surrounded by teh pr0n
russ4n, german and teh japanes
teh Cl4n watched teh car leave
a shitbox it was
filled with the lewt
and teh goatse mails and gifs
4 teh long wait till uni
he was without a gfx card
he w4s liek a n00b
wh3n he left teh clan
A l4mer n00b
teh clan put a bumper sticker on his c4r
Goatse it procle.. proclaine... said.
teh clan l3t him le4ve
a|\|D 3 teh uni he went
teh most l33t in teh clan
don't no what uni is liek

I feel unclean now.

Wow

[Dachannien]

Attendees recalled the initial fear and loathing the Beowulf project had to overcome.

And now, "Beowulf" is a term synonymous with the most downmodded /. joke of all time.

You've come a long way, baby.