Slashdot Mirror
FreeBSD Cluster At Purdue
luddite writes: "Two guys at Purdue University have assmbled a FreeBSD based cluster built cheap - very cheap. With under $2500 spent on the cluster, it's one sweet set-up. Just shows that if you take the time and put some effort into something, money doesn't have to limit your resources! The site also goes into some detail about what the cluster is made of, where they found the parts, how it's been configured, and what they plan to use it for."
How did you get dual motherboards for K6-2?
WWJD? JWRTFM!!!
But it takes so long to install the operating system.
I also like to run sendmail, apache, name server, nfs, samba, etc.
When I use any of this while I playing mp3s through its sound card, my stereo sound awful.
You know, this 75mhz Pentium probably does more than my 650mhz Athlon.
Yes, I know why Bind 8 isn't included. Red Hat changed Bind to run under the non-priviledged "named" user precisely because of this problem in 6.2.
I guess I probably feel the same about BSD and SunOS 4.1.3 - classic operating systems, but these days a little behind the times.
>Nononononono. The Ethernet is used only on the fighters. Anyone can
.VBS file created on the Mac to the mothership and the rest as they say was history
>clearly see that the fighter umbilical is FDDI. (And of course the
>mother ship runs Windows -- don't you know that the reason Jeff
>Goldblum was using a Mac (apart from the fact that they own him
Yeah. That's why they were able to infect the mothership so easily with the virus. They just uploaded a
It looks like the boxes have served most of what they are good for, recognition.
With ~16 single proc 450's with 32mb of ram... what sort of problems can you solve. Sure, sure, they offer parallel tasking. But 32mb? How many large tasks that need parallel tasking can be done on a several single proc boxes with that little memory... and with PVM?
I have used PVM before on a similar set up of systems (ok... they were 300's and I only had like 4 of them) but I was able to sort numbers and compute digits of PI... but that was about it.
Now... on the other hand, buying real systems... like 42 dual 600's with 1gb ram. Those can rip through problems... well... rendering. And... Beowulf isn't always the soultion. Sometimes shared resources aren't good... can you imagine a network supporting a shared memory of up over 500mb's for a process?
Beowulf has its place, pvm has its place... but in a lot of places, it is good for research. I hope that they were able to acomplish this.
-I just work here... how am I supposed to know?
I made a 8 node cluster of Linux boxes, cost $250. I spent more money on a liquid cooling system for it, I was drunk, theres not much to do at Purdue.
Acctually they also have the 126 node Intel Paragon with the nifty little lighted display that shows the interprocess communication, and memory links.
What?
You really mean to tell me you didn't see Independence Day?
Everybody knows they have Mac workstations with their built in ethernet.. *duh*
Would it kill Nik to use the BSD Icon for these stories? Isn't that why its there for?
Just something I've noticed which I find irritating. Sorry to be so anal...
There's no reason these technologies shouldn't migrate to OS X as well at some point. Many reports of the full SMP demo'd at Apple's WWDC under OS X beta on Apple hardware. Distrbuted processing may come along as well but may or may not make the first release as it's MUCH less sexy than SMP.
Apple's plan would probably center on "farming" your Mac network's extra cycles at night more than the kind featured in the story. But plain distributed machines - like the headless one's in the story - shouldn't need full OS X install, though, unless they need Carbonlib, OGL or something. Maybe a lite Darwin-ish install would be possible for those machines - who needs display code without a display?
Of course you're probably talking about an extra 32Megs of RAM to make up the difference so why worry about it?
=tkk
Bill Gates - Creationist?!?
"I'm sure a watered down version of Aqua could be created"
:)
LOL
Reminds me of that guy who wanted to sell dehydrated water...
Sorry, couldn't keep it for myself...
-- It's always darker before it goes pitch black.
I just wrote an essay on this very subject for a friend of mine who is trying to decide which non-MS operating system is best for her. The essay I wrote simply explains why FreeBSD turned out to be the right choice for me, since I can't say what would be right for someone else in a different situation. If you're interested, my essay is here.
-Joe
Dude, prior to reading your post, I was a Democrat, I am now switiching to Republican. :)
Thanks!
------ Curiosity killed the cat. {satisfaction brought it back | it didn't die ignorant | lack of it is killing mankind
It is quite a nice little system when you have enough users to keep it busy, and I imagine is quite easily scalable. The only trick is that if there are differences between the various machines you can have really annoying and hard to track down bugs. Fortunately consult is very responsive. Once they even contacted me about a bug in my code, because they saw the error in the system logs and thought it might be their fault!
Take a look at the picture---it's a FreeBSD cluster with a sound card. Kick ass! ;-)
----
----
Am I the only one who thinks Microsoft is a misnomer? Perhaps Macrosoft would be a better fit?
The picture doesn't say all of those machines on the rack were for ACME. The picture also doesn't say that all of those boxes were up and running and not just sitting there empty.
IANAL, but I play one on
We cheat. If you look closely at the picture, ACME is sitting on a raised floor. It lives in a little-used computer room left over from holding two dual Vax and a couple of Goulds.
As a result the room has a raised floor with forced air from below, Liebert cooling, power conditioning, better security, etc etc etc.
I think not.
There is no reason at all that you can't do similar things. Your arguement is much like that of someone who says "oh you've comitted fraud because you didn't pay for that FREE operating system." FreeBSD and Linux happened because thousands of people gave their time (in huge amounts) to these projects. What would you have us do on the budget to account for the free O/S that we were given?
Unless you've lived under a rock, *lots* of things get given to places (non-profit and for-profit alike). Educational places get zillions of dollars worth of things every year (especially a person from MIT should know this!). Just this past week a group I consult for gave another group (us)$10k worth of equipment in a different division of the same company. It wasn't _fraud_ that they can now clain they didn't have to spend the money to buy that stuff. Their budget wasn't decrimented a dime.
The ACME budget stands as it is as that is what it cost in dollars to build it. If you're going to pick at us, then pick at the time that we've got invested in getting it working. But since this is /., I'll guess that everyone presumes that their time is free.
17 at the moment.
We found cases last summer that all matched, so that's why we have all the cases. Since that picture was taken, the pile of computers on the far end has gone away and another rack replaced it.
The end you see closest to you is Column 'E' so all the nodes in C, D and E are up plus the top two in column B. The bottom machines in D and E are the two connected to the outside world. The bottom one in D now really is in the new rack, but we've not got new pictures since that was just done last week.
We add roughly a motherboard a month. Hard disks are not a problem (I see four within reach right now that are all 880mb). CPUs we purchase. Memory we find here and there mostly for free. The motherboards are the sticker. PU Salvage gets probably 100 machines a month and we try to get every one of them looked at before someone else gets them purchased.
Disk drives less than 1gb often go into the dumpster even out here in the midwest. Most of ours came from junk at junk yards or from piles of stuff that was headed for the junkyard.
We have yet to have to pay for any disk space.
Watch ACME's news page... in a couple of days we should have eclipsed our 'free' disk space with even cooler stuff.
So far, with over 70,000 hits since it started 9 hours ago, ACME.ecn.purdue.edu has done very very well. Our guess is FreeBSD 3.3 as it currently is tuned with plenty of Apaches and enough memory to prevent swapping and we could have dealt with at least 10 times the load we've seen so far with no changes.
/.
/.ed is an intersting thing. We'll write more about it after the hits are below 4 a second.
On the other hand, we'll also bet that the campus connection to the Internet probably couldn't have taken 10 times the load.
So far so good. I figured we'd get maybe 10,000 hits over the entire life of ACME and we crossed that in the first hour of
We've seen some security trouble, but nothing that reasonably well configured systems couldn't handle, though I'd be running a bit more security stuff if I'd known this was coming.
Being
David Moffett
The nifty Paragon was just decommissioned a couple of weeks ago. I'll guess they needed the space for the SP/2.
:-)
Perhaps it will show up someplace where we can, ummm, 'acquire' it.
K6-2 because at the time we selected it K6-3 was still in the first revision and the motherboards that we had couldn't do the power requirements (3.x volt current in particular) that K6-3 needed. Also there was (and I'll guess still is) a pretty steep price premium on K6-3. This was one of those topics we discussed for a while before we decided. Original K6-2s were just on the current limits of what the P55T2P4 Rev 3 boards could do.
Our rule of thumb is each node is roughly a Pentium II/330. Most of the code we're running (at least at the moment) is in Fortran (80% - a 4800 line legacy) & C and is mostly integer.
I'm reminded of the expression, "when all you have is a hammer, everything looks like a nail." We have a hammer and some of the problems will certainly be screws, but at less than $3k we'll live with it. It isn't a perfect world.
It's safe to say we can bring up a new node in well under 30 minutes. That includes flashing it to a new BIOS, setting up the NV RAM in the network cards and then loading FreeBSD (via the network of course).
Sometimes it takes longer, but that is usually because we've got some piece of bad hardware (memory or disk usually) or we've screwed up along the way from raw tiredness. Most of the hardware work we do on this thing is done on Friday nights.
Of course FreeBSD allows for symbolic links. They've been around for a long time- 4.2BSD or so. Fully documented in the D&I of 4.4BSD..
If you are _sure_ about symlinks not working on SunOS, then that was probabaly because they were disabled for a reason.
-bugg
Speaking as a Purdue Alumni, I thought it was to hire a few more vice presidents. ;)
I would use my real name, but the Alumni Association would be after me for money.
I dont know a whole lot on this topic but wonder
what the performance and usability issues are
between PVM and Mosix, other than issues dealing
with kernel mods.
Hmmm..
$2,390 for a 4/5 diskless cluster of Celery 500/533 done right.
Go with scrap cases, a crossover cable, used Boomerang cards in a hypercube, cheapo DFI mobos and a smaller SCSI drive and you can get out the door for $1,800. Go for offbrand memory and O/C Celery and you can push it down another $150.
.sig: Now legally binding!
At no cost? It's not like they're buying new ram.. there's a cost.. beyond high failure potential it was hardware which was just recycled.. nothing more.
"And how can this be? For he is the
Yet another reason to eliminate anonymous coward postings.
Purdue Salvage is where all stuff that University no longer wants goes. You can find all kinds of things there, from beds and desks, to computer parts. It was a great place to buy old hardware!
I've got a similar cluster with a somewhat different focus - I need to be able to generate enough HTTP client traffic to saturate (well, nearly saturate) a single gigabit ethernet server. The clients in my system are eight eTower 266s that I got on clearance at buy.com for $229 each. These have 200MHz Cyrix M-II CPUs, and running FreeBSD that's enough horsepower to saturate a 100baseT ethernet, so eight can pretty much saturate the gig ether.
However, if the client machines were running Linux, they would not be able to saturate, since Linux still has about half the networking performance of FreeBSD. That's the main reason I run FreeBSD and not Linux. Other reasons include Linux's tremendous supply of bugs that were fixed in BSD years ago, and the general obnoxiousness of the Linux and Gnu user community.
If not for those things, sure, I'd run Linux.
The current problem the cluster is solving is a very complex transportation engineering equation. My best undersatanding is that the computer knows the result of the problem and general form of the equation but does not know the equation coefficients and exponets. It is just iterating the problem untill it finds an awnser. Solving problem requires very little in the way of memory (and hence disk/swap space); just a lot of horse power. I would guess the equation will fit entierly into the 64kb of L1 cache of a K6-2. The problem has been cranking for over a year on various personal computers. Once the equation is solved, it will be used for a structural engineering problem; most likely matrix algerbra.
Did the site say anywhere the ultimate purpose of the cluster? Is it just to test speed and cost-effectiveness? Also, it sounds like the machines are lacking in some hardware departments. Do they have high-end graphics cards, what kind of drives are they currently sporting? My research group is also based at Purdue, and it sounds like the cluster we have set up is superior. We currently have ten nodes (one dual) each running Redhat. The processors are AMDs, 6 K6/2's and 3 Athlons. 550MHz and 6-700Mhz respectively. All have 256Mb of RAM and 44x CD drives. We also have 100MBit intranet. The current setup ran 10 grand, with monitor and monitor switches. I doubt the ACME cluster could run the simulations we run. Our simulations have a run-time of 12 hours. I wonder what it would be on theirs...
Okay, now I'm done with the "Mine is better than yours" rant.
Ciao
nahtanoj
>Guess you didn't notice that a majority of stuff was donated. Only the stuff they paid for was listed. But that's my point! Saying you can build X for under some amount of money when a lot of X is donated is fraudulent.
First, there's a BUNCH of things missing, like motherboards, memory, etc.
Second, the numbers don't make any sense, 40 network cards for a cluster of 16 machines? But wait, only 15 CPUs??
Third, many of the prices they got was sheer luck (and perhaps a little bit of work, which I would applaud them on)...I mean comeon, they managed to get from their salvage dept. (what's that, BTW?) network cards for ONE DOLLAR a piece and tape drives for $2.50 a piece (their keyboard adapter cost $2.32, only 18cents less than a tape drive). Those prices are so not good examples of real prices others would be able to consistently find -- or most likely at all!
All in all, almost none of their budget makes any logical sense, and it all just strikes me as stretched truth + luck = a cheap cluster here. I mean comeon, I'm going to have my friend by a $5000 computer, and sell it to me for 1cent, and then post to the world on my webpage that I bought the cheapest PentiumIII-whatever around?
Ya, I noticed that too. I also didn't notice any hard drives on the list. What do all these babies boot off CD-ROMS? I wouldn't think so because I don't remember any of those on the list either.
Damm dis shit be scandalous.
-Jon
this is my sig.
The interest in this is that it was done for so cheap.
The cluephone is ringing...
/* Bill Fumerola (billf@FreeBSD.org) */
Why in the hell did they buy 107 network cards for a 16 node cluster?
Look at the budget! They bought 107 network cards! Now that's a lot of bandwidth.
--
--
Actually the FreeBSD installation does support CDRom drives. Ever since 2.05 or so (that is the earliest version I have used). The CDRom support in the low 2.x kernels was sometimes tempermental but nonetheless functional.
BSD originated from a set of modifications to various versions of the 'original' AT&T Unix, which mostly (and especially at Berkeley) ran on DEC PDP11s and Vaxen.
f.
Every enlightened person knows that Linux is optimized for doing one thing at a time. Therefore, Linux will easily saturate a 100mbs connection. Unless, of course, it does something else at the same time.
-T
You went about installing FreeBSD the long and hard way! All it take is only 2 floppies with images put onto them (kern.flp and mfsroot.flp). The floppies will boot the kernel then you have the option of installing off the CD's, FTP, or network.
RTFM:
FreeBSD Handbook Installation Guide and the FreeBSD Newbie install for screenshots with play-by-play instructions (screenshots are for 2.2.5, but they look the same for 4.0).
Could this mean, at least theoretically, we could see an open source base for clustered OS-X?
The idea could be awesome if it happened. Designers using OS9 are screaming for multiprocessing.
If some OSS team came up and gave them "BeoMac" or whatever, Apple had better watch out or pull its hardware - roll - out - socks up.
Is it just me?--or does anyone else see 22 computers in that rack.
They probably just don't quite have enough for a 5x5 (25), and so are dropping down to a 4x4 until they get a couple more machines going.
-k
What about a beowulf cluster of these. Oh, wait ....
But seriously. Anybody got any info on multilayer clustering technology?
Eh...
Maybe the rest are going to Area 51? Sorta like those $50 government hammers. BTW. The added costs are usually the product of the complex buearacracy, not some government plot. If they were plotting, they could do a much better job... A lot has to do with pork barrel politics and paying everybody down the line who touches that hammer... and some people who don't in order to make sure that we don't discriminate against them (IE the wealthy ).
Eh...
IT's not a Beowulf cluster, it uses PVM. PVM != Beowulf
You know, I'm tempted to go out and spend a few thousand on a BSD beowulf cluster. But I'll be damned if I can think of a single productive thing to do with it.
Ok, it's cheap, but AMD K6-2 are really *bad* for floating point (except when using 3DNow!, but gcc/pgcc does not generate these instructions). My K6-2 300 MHz is equivalent to a Pentium MMX 166 when doing floating point computations without 3dnow.
Notice that Luddite's links are to the very system he's describing. This system seems to simply shrug off the /. Effect. If I were going to start up a Web Server Farm, I wouldn't definitely talk to these guys!
Someday, we'll be installing OSes with stacks of CD-ROMS... that'll be when I need to install FreeBSD on this machine:
500 GHZ Quad Intel Octium (80x886) Processors
30 Terrabyte HD
256 Gb RAM
1.44 floppy
--Cr@ckwhore
Skiers and Riders -- http://www.snowjournal.com
Calling BSD a Conservative Republican incarnation is like saying that RFK was a centerist. Its ludicrous.
BSD came out of the most liberal atmosphere one could think of. Anybody remember those yellow t-shirts with the daemon dressed up as a flower child, and the slogan "Peace, Love and Rdist" across the bottom.
The truth of the matter is that the BSD development psychology may seem outwardly conservative when compared to Linux, but is really democratic, and quite liberal. There is simply more control over the final product than there is in Linux.
Linux is just weird. I cannot understand it. It seems to be a mix of part anarchy and part autocracy.
I'm not arguing with the PC architecture concept, the choice of operating system, or anything like that - but in the real world companies will be forced (by cost) to go for 3 * 4 processor Enterprise servers rather than 16 freebsd boxes (the comparison used on their web page)
Why?
Ever seen how much machine room space costs?
It'd be interesting to see a price comparison where individual nodes were as powerful as individual Sun units (though add on the cost of freebsd OS support, since you do get that from Sun)
james
--
ALL YOUR BASE ARE BELONG TO US!
Erm. Could you be more specific? "FreeBSD is superior to Linux in every way" just sounds like plain ol' FUD to me.
And stop pretending our camps are at odds; both camps share. FreeBSD & Linux are both kernels; both kick ass.
BTW, does FreeBSD allow for symlinks? I remember my days of using SunOS (pre-Solaris days) and symlinking was verboten then.
so the idea of posting this story was to see whether the cluster could handle the /. effect ?
There's a more recent, less in-depth review comparision of FreeBSD vs. Linux vs. Windows NT here. Why Yahoo! Uses FreeBSD is also interesting.
Our Athlon based KLAT2 Beowulf cluster at the University of Kentucky achieved over 64 GFLOPS on LINPACK for only $41K using 3DNow! instructions. The FreeBSD Cluster at Purdue doesn't even mention ANY benchmarks for performance. I'm a Purdue Alum, so I think this is great that they are getting slashdot coverage for an inexpensive cluster. However, when we submitted KLAT2's (Kentucky Linux Athlon Testbed 2) story to slashdot last month, which in many respects is much more "news for nerds", it got passed over. Ah well, thats the way of slashdot.
--
--
Tim Mattox
As for using the 3DNow! stuff, their K6-2's can have some real punch if they are willing to code for it... Check out our SWAR - SIMD Within A Register compiler technology for doing just that. Actually, the Ph.D. student doing most of the work on SWAR is AT Purdue.
--
Tim Mattox
Every single mother board is an old Asus P55T2P4 modified to accept a K6-2 400. Most boards came out of machines being upgraded and therefore had zero cost. Some were also purchased at swap meets for little cost. The memory also what was left over from upgrades and available at no cost.
Or why not Linux? A couple of reasons each way:
The BSD people are great, and Linux owes a lot to them. BSD continues to make great contributions to the world of Linux. It would be the best of all possible worlds if each had the same capabilities. But, because of the hype factor and the real development it brings, BSD has no hope of being as flexible as Linux in the near future.
I guess it is a question of what you grew into, the level of risk you are willing to tolerate, and the hardware that you need to support. The decision of BSD or Linux starts there.
These guys are buying AMD K6-2 3D 450 processors, which they say work in a variety of motherboards. Do these work in non-MMX (single voltage) motherboards?
I'm using an old Gateway P75 as a masquerade box for my cable modem. It would sure be nice to upgrade it to 450 on the cheap.
I am looking for the best way to squeeze a little more life out of this box.
Actually beowulf clusters commonly use eithe MPI or PVM. Neither is required to classify a cluster as a Beowulf. The first demonstrated Beowulfs were by NASA and ran on the PVM libraries. Refer to the Beowulf into http://www.beowulf.org/intro.html
--- Linux... a college project gone horribly right
let me see. free/netbsd runs on 68k machines.
:) and all for about the price (including cabling & hubs) of a cheap celeron machine.
macbsd (netbsd/mac68k) runs on my LCII.
LCIIs can be had at the local surplus auction for $5 apiece. Most of these are formerly lab machines, and have ethernet already.
i think a cluster of 25 of these low profile 16mhz monsters could fit on a desk, _maybe_ put out the MIPS of a PII (and only about twice the heat
hmm.. for $10 i can get powermac 6100s by the truckload, and freebsd/ppc...
- Entertaining Bits from the Ancient Kernel Tree
So THAT'S where all the money we pay for Ethernet in the dorms goes...
"Save the whales, feed the hungry, free the mallocs" -- author unknown
There is a date in there from 1997. Sounds like a lot of that stuff is old and outdated. Linux has come a long way since 1997.
Shine on, you crazy diamond.
Read more of the article first...
From the news page: I think that the tide still says that the power is in the almighty dollar$$$$
--
Interesting start, but they've still got memory, motherboards, and some other stuff to go. That's going to crank the price tag at least a couple thousand dollars...
Look through the budget, and there's no entry for motherboards. That is unless you can get a AMD K62 450 + mobo for 64 dollars.
From what I can tell the price (~$2500) doesn't include the motherboards or memory which they obtained from various sources. This is probably one of the most significant outlays they had to make, next to the processors.
It is still really a great price, and I can't believe what they paid for the racks.
-k
Ok, look at the budget they laid out. Yes, a university with a significant excess of computers can do this cheaply. That's the argument that many people are using trying to get MY university to build one. LOOK AT WHAT THEY ARE PAYING FOR EQUIPMENT. If I got CASES at $1 a piece, and NIC's for the same price, and RACKS for $10. Then I could build just about whatever you want for under $3Grand myself...
Eh...
Nononononono. The Ethernet is used only on the fighters. Anyone can clearly see that the fighter umbilical is FDDI. (And of course the mother ship runs Windows -- don't you know that the reason Jeff Goldblum was using a Mac (apart from the fact that they own him anyway) was that the aliens couldn't backcrack him? Go ahead, moderate me down; my karma is a bit swollen anyway, and this bit wasn't half as funny as I'd hoped... /Brian
Is it just me?--or does anyone else see 22 computers in that rack.
--
Have fun: Join D.N.A. (National Dyslexics Association)
Boy.. I bet that thing gets hot..
ACME currently consists of sixteen Pentium class computers, each with 450 MHz AMD K6/2 processors and at least 32 MB of memory.
I have two 450 Mhz Amd boxes in a small room, and they sure pump the heat up there..
-
air and light and time and space
It's a kind of high-performance massively parallel computer built primarily out of commodity hardware components, running a free-software operating system like Linux or FreeBSD, interconnected by a private high-speed network.
http://www.dnaco.net/~kragen/beowulf-faq.txt
--- Linux... a college project gone horribly right