Slashdot Mirror
Practical Beowulf
elsewhere sent us
Linux gushes savings for oil giant
where you can read about the 32 node beowulf cluster being
used by an oil company to replace IBM super computers.
← Back to Stories (view on slashdot.org)
← Back to Stories (view on slashdot.org)
Anyway, the beauty of this is that when big companies like this start using Linux, it's easier for me to convince my boss that it's a good choice for our small company.
World domination: the sooner the better.
Werd.
The Extreme Linux CD and most of the RPMs it contains are woefully out of date. The EL CD has a hacked RH 5.0 install with some kernel modifications, none of which are really necessary. Most clusters are being moved to 2.2.x kernels (for network and SMP reasons) and at least the channel bonding modifications have been moved to 2.2. My 24-node cluster is running 2.2.3 which I hand patched with the TCP_NoDelay and Channel Bonding modifications.
If you are serious about getting a cluster running, take a look at:
http://www.xtreme-machines.com/x-cluster-qs.html
This is very up-to-date and basically begins by advising you to use your EL CD as a coaster.
Mike Prinkey
Actually our exploration IT budget is probably closer to 4-5% of our exploration budget. You are only counting one project. We also have 150 Sun Workstations, several hundred Windoze PCs, 6 Sun Servers,SP2, Auspex, employees, software etc. Also, this sea-floor information is extremely expensive and adding just a little improvement could well cost close to a million dollars so it does really help. When you have all the information, it doesn't seem so bizarre and one would have to think that Jed's career here might be in jeopardy.
....Jeff
1. Costs - Linux is clearly cheaper
2. Reliability - Linux was clearly more reliable which has been born out. Our clusters have now been up over 3 months straigt.
3. Scalability - How does one manage 96 NT boxes in a cluster. I had no idea. Unix has things like rsh and robust scripting making this a snap
4. Portability - it was relatively easy porting our sp2 apps to linux. by the way we use Linda from SCA (Scientific Computing).
....Jeff
here is the code for my little dsh script. very simple and rudimentary but it works. all you do is create a file with node names in it say /collection1. no security per se so maybe someone can show me how to add ssh or something else.
Let /collection contain
node1
node2
node6
node8
Usage Examples:
export BCOLL=/collection1
dsh date
dsh /nfs/tech/update_kernel.22
dsh reboot
dsh "echo node1:/nfs/data /nfs/data nfs rw,bg,hard,intr >> /etc/fstab"
The code
#/bin/ksh
if [ "$BCOLL" = "" ]
then
echo "Environment Variable BCOLL not set"
echo "Please set BCOLL to a file with list of nodes it"
echo "export BCOLL=/root/allnodes"
echo
exit
fi
if [ "$1" = "" ]
then
echo "Please provide command to run"
echo "Usage: dsh command"
echo
exit
fi
echo "Collection: $BCOLL - Command: $@ - Date:`date`" >>
for i in `cat $BCOLL`
do
echo >
echo "Running $@ on $i" >>
echo >>
rsh $i "$@" >>
done
wait
for i in `cat $BCOLL`
do
echo
"----------------------------------------------
cat
rm -f
done
Jeff Davis
....Jeff
We actually evaluated IBM Netfinity PIII 500s. They were good performers. However, they were about equal to the Dells and since we have already been using Dells, we saw no reason to switch at this time. But we will continue to evaluate systems based on price/performance as we grow our system larger than 96 nodes.
....Jeff
We work for Amerada Hess on the beowulf project in the article. The author of the article was genuinely interested in how we used Linux and the difficulties in getting our upper managment to sign off on the project. The Beowulf project at Amerada Hess has really opened some eyes about the power of linux and hopefully this article will spread the word. Slashdot, Freshmeat, Redhat, Dejanews, Dell, and Paralogic have really helped this project run smoothly. I would highly recommend any company with a need for serious computation and a few programmers to give beowulf a shot. It really works and it can really save you money Jeff Davis Harry Duffey Amerada Hess Corporation Houston, TX
....Jeff
1) The main tools we had to create was a similar program to IBM's PSSPs dsh (distributed shell). However, we didn't take the time to put in all the features that IBM has in their dsh. As of right now, their is no similar corollary to PSSP on Linux that I'm aware of. There is a product called SMILE (SCMS) which we were implementing beowulf wasn't quite ready. There is also a product call masshosts by John Mechalas which we are looking at. One nice thing about Linux/Unix is that there is really that much need to use these tools that often as the systems run and run without much need for interference. We also used the kickstart process to install the nodes. The SP2 definitely has a one up on installation and you basically setup the install process and say go. several hours later the system is ready. however, linux installs about 6 times faster than AIX. 2) I guess i could put my little tool out somewhere but i really think others could probably do a much better job of writing a more complete tool. I will try to compile our config and post it. Jeff Davis and Harry Duffey
....Jeff
I was thinking the same thing while reading the article when I remembered that IBM not only knows of Linux's capability in Beowulf clusters but has demonstrated it. Anyone remember the 17 Netfinity cluster IBM demonstrated at LinuxWorldExpo that matched a Cray on the PovRay benchmark.
I feel IBM will be fine for quite a while as far as their AIX systems are concerned. The beowulf project does great work on high computation problems but fall relatively short on applications require large and fast disk space. Mostly due to a lack of a fast distributed filesystem. At least this is my understanding. Down the road maybe IBM will go back to being a hardware manufacturer primarily.
--- Linux... a college project gone horribly right
Seeing as how IBM has a team porting Linux to their RS/6000 workstations, I think IBM is planning on using Linux to help profit margins
Christopher A. Bohn
cb
Oooh! What does this button do!?
I think it's a shame that this has been moderated to -1. That's not a flame after all. I know that "the community" won't agree on any single application of a free OS to be morally good or bad (and subsequently our licenses usually and rightfully do not include clauses to prevent code use by certain entities). However, I can understand why a person chooses to not jump up and down because of the use of Linux by an oil corp, and Slashdot should be a place where it's possible to to express such concerns. I too would be more happy if the news read "Linux used in scientific effort to significantly reduce oil usage worldwide". OTOH, the chance that penguins are washed up on Norwegian shores is quite small, considered that Antarctica is quite far away from Norway :) Dead auks are not better, though
"When I first heard Daydream Nation it quite frankly scared the living shit out of me." -- Matthew Stearns
I think this is definately going to happen. I taked with the reporter who wrote this Computerworld article (I'm part of the group at the University of Texas that is mentioned at the end of the article). He said that Computerworld is mainly read by business-types who want to use technology to cut costs. I think Linux just starting to be accepted in this community, and will soon take off there like it has in other areas.
-jason
You don't need the rpms from beowulf.org unless you want to use extensions to the standard kernel such as channel bonding or distributed pids. Otherwise, just install your favorite distribution on several machines, grab MPICH or PVM, and start experimenting.
Feel free to send questions my way if you need help with this. Or check out any of the documentation at http://www.beowulf-underground.org/.
-jason
Since they are saving so much money, do you suppose gas prices will go down now?
Oh wait, what was I thinking?
I was quite confused about this a second. How could they possibly have saved anything by buying NT instead of using Linux? They would have to port 2M LoC rather than write a bit of custom management for the cluster.
But then I realised that they would have saved (only) a few $100.000 if they used NT instead of the IBM SP2. Now they seem to save $1.870.000 by opting for Linux! (Minus the cost of writing custom code)
As an IBM SP specialist, and a Linux advocate I am impressed and intrigued by the Beowulf configuration built by Hess.
The SP does have a fairly robust management toolset (PSSP) and uses kerberos with parallel commands for secure multi-node management.
To be honest, I haven't read up on the capabilities of Beowulf but the impression I get from the article is that Hess had to write some apps to approximate the management capabilities of the SP.
Two questions:
1) Did Hess have to write these apps because Beowulf has a weak management system, or did they have to write them because Beowulf has a functional system that is different from what they were used to?
2) Any chance that these management apps would be available under the GPL? Or even that more details about how they configured their systems will be posted?
I'd love to be able to show my customers what Linux can do. Maybe Jeff Davis from the above post can shed some light on this.
Ted Richardson
Yes, in order to save the wildlife, I've been fighting agains strategic oil drilling my entire life, I never drive anywhere, buy any petroleum based products, or buy from anyone who does support petroleum... Wait, you yutz, unless you can find that magical elixer for all the world's (around 75%) energy needs, don't freak about Linux being used to drill a hold in the ground. Besides, I'd rather see a beowulf cluster being used to draw up detailed maps of the ocean floor than being used to play quake at 1600x1200 at 32bit color. Don't get me wrong, I think that we need to figure out alternate means, but right now we're slaves to our addiction to things like electricity, mobility, and civilization, if it weren't for oil drilling, you wouldn't even be running that fancy pants computer on your desk. Just a little rant. Zealots piss me off.
-Zen I'm gonna make the _world_ my bitch.
It's just you.
Seriously thought, while the article states that they would have saved several hundred thousand dollars by going to NT, they were able to save even MORE money by going with Linux. Also, it appears (and I may be reading too much between the lines here) that they also felt that the programmign aspect of the move would be easier if porting to a *nix-like operating system, rather than to Windows.
Just some thoughts from a guy ripping out all the NT server machines at his work and replacing them and the RS6000's with linux boxen.
I wonder how long IBM will be so supportive of Linux when it starts to cut into their high profit market. Although, I suppose with the right leadership, they'll roll with the punches and realize that markets change, and they won't get stuck like last time. Has anyone else noticed the "small company" attitude that IBM has been trying to create? Almost like, "We're big and reliable", but at the same time they're saying, "Hey, we're small and flexible, and new!" Just a few thoughts. Really this is just a ploy to get a score greater than 1.
Vidar Leathershod
The brains of a chicken, coupled with the claws of two eagles, may well hatch the eggs of our destruction.
One group at Los Alamos did just that with previous generation (21164) Alpha boxes and were quite happy with the results. I understand that they're now looking at building a cluster from 21264s, which have significantly better FP performance. Pretty zippy little buggers.
http://cnls.lanl.gov/Internal/Computing/Avalon/
You can't fight in here - this is the war room!