Most enterprises want to put on fixes that solve a problem. Sure it is nice to be a patch set on, but ultimately, the idea with patches is to fix/improve/add something. A good sysadmin will always keep current on security patches as a priority even if they have to be applied one at a time. The is most especially true on systems that are attached directly to internet.
1. Need better multi-system performance monitoring tools. IBM has a really nice performance tool called Performance toolbox which lets you create widgets to monitor a wide variety of system characteristics on a variety of system all within the same window. We would also liked to have had native drivers for our IBM 3590 tape drives. On AIX we use the features to put the label of the tape on the tape drive led. It would be nice to have better distributed management tools for installation and management. However, none of these things would disuade us from using linux. Linux has been a big success here at Amerada Hess.
2. I would like to see journaled filesystems, more drivers for various hardware devices, better monitoring utilities, better installation tools for mass installs and upgrades. We would like to see gcc integrate the P3 mmx instruction set for floating point purposes.
3. We chose RedHat really because that was what Harry Duffey and myself were most familiar with. For our programmers and users, the version of linux was irrelevant. It doesn't hurt with upper management the RedHat is probably the most visible and known Linux vendor at this time. In fact, we are in the process of taking a look at FreeBSD, Debian, and Suse just to level check RedHat. I have not relied upon RedHat for any support. The few times we called RedHat, we had a hard time getting support. I typically use Dejanews and find almost all the support i need.
4. IBM is aware of what we are doing as they had an oppurtunity to sell us Netfinity PCS for our project. IBM has real nice systems (PC and Rs6ks) and they did a good job in our evaluation. However, all the PIII 500 performed almost exactly the same in our seismic processing benchmarks. Hence, we had no good reason to switch from Dell. I think IBM is very interested in the market. We have worked for IBM for years and they are good partners and I suspect if they are smart they will have an excellent Beowulf offering. I am not sure what this will mean for the SP2 though although I suspect for the time being, many large companies will buy the SP2 for name and reputation regardless of the price.
The Key Point is that our cluster is now up to 96 Nodes. Hence we can now run 3 jobs at the same time. Maybe my math is wrong but my guess is that our total through put is 2.5 times more than the sp2 and we still save 1.5Million.
We actually evaluated IBM Netfinity PIII 500s. They were good performers. However, they were about equal to the Dells and since we have already been using Dells, we saw no reason to switch at this time. But we will continue to evaluate systems based on price/performance as we grow our system larger than 96 nodes.
Re:Passing the savings on to the consumer
on
Practical Beowulf
·
· Score: 1
I agree. I wish my company would do just that. Makes for a longer career and probably a better environment which i think everyone is in favor of. Good points!!!! I believe lightsabers use some those generic batteries you can get at Wal-Mart
here it is. it is very simple and i know you guys/most anybody can do better but since you asked.
here is the code for my little dsh script. very simple and rudimentary but it works. all you do is create a file with node names in it say/collection1. no security per se so maybe someone can show me how to add ssh or something else.
#/bin/ksh if [ "$BCOLL" = "" ] then echo "Environment Variable BCOLL not set" echo "Please set BCOLL to a file with list of nodes it" echo "export BCOLL=/root/allnodes" echo exit fi
if [ "$1" = "" ] then echo "Please provide command to run" echo "Usage: dsh command" echo exit fi
for i in `cat $BCOLL` do echo >/tmp/dsh$$.$i 2>/tmp/dsh$$err.$i echo "Running $@ on $i" >>/tmp/dsh$$.$i 2>>/tmp/dsh$$err.$i echo >>/tmp/dsh$$.$i 2>>/tmp/dsh$$err.$i rsh $i "$@" >>/tmp/dsh$$.$i 2>>/tmp/dsh$$err.$i & done wait
for i in `cat $BCOLL` do echo "----------------------------------------------- ------------------" cat/tmp/dsh$$.$i/tmp/dsh$$err.$i rm -f/tmp/dsh$$.$i/tmp/dsh$$err.$i done
Jeff Davis
Re:Passing the savings on to the consumer
on
Practical Beowulf
·
· Score: 1
check out gas prices over the past 20 years. i bet you'll see that gas is one of the best bargains around. how much did a car cost about 10 years ago. I remember not paying much more now that i did then for gas. people always complain about the price of gas. but in fact, factoring in inflation it has gone up very little. in fact alot of the price of gas goes to taxes. maybe you should contact your local politicians and complain to them.
Speaking only for myself, not my company, we don't want to stain or destroy anything. Maybe when people stop using oil derived products, all this might stop. We love to buy fast cars, products made from plasics (sneakers, computer parts, etc), products made from chemicals (hair gel, medicines, etc), power to power a computer, home, car. Supply is a result of demand. If you want to decrease the use oil, maybe you should try to work on the demand. Lower the demand for oil, i bet oil companies will stop poking holes. No one complains when linux is used for nuclear weapons research. what do people think that linux is being used for in all these universities and labs. if linux is to grow, it needs to used by as many types of apps as possible. besides, linux is free for use by all. who should determine how it should be use. i think the spirit on linux means that everybody can use linux for whatever results the want. anyway, that is just my oppinion (not my companies as i do work at a corporation and have to deal with suits)
This also doesn't take into account software costs in which linux definitely saves a bundle cause its free. So far as I remember, the SP2 software is not free and has fairly hefty support/maintenance costs.
I think the statments means that by using NT we could have saved money in comparison to the SP2. However, we really never even considered NT as on option for several reasons
1. Costs - Linux is clearly cheaper
2. Reliability - Linux was clearly more reliable which has been born out. Our clusters have now been up over 3 months straigt.
3. Scalability - How does one manage 96 NT boxes in a cluster. I had no idea. Unix has things like rsh and robust scripting making this a snap
4. Portability - it was relatively easy porting our sp2 apps to linux. by the way we use Linda from SCA (Scientific Computing).
1) The main tools we had to create was a similar program to IBM's PSSPs dsh (distributed shell). However, we didn't take the time to put in all the features that IBM has in their dsh. As of right now, their is no similar corollary to PSSP on Linux that I'm aware of. There is a product called SMILE (SCMS) which we were implementing beowulf wasn't quite ready. There is also a product call masshosts by John Mechalas which we are looking at. One nice thing about Linux/Unix is that there is really that much need to use these tools that often as the systems run and run without much need for interference. We also used the kickstart process to install the nodes. The SP2 definitely has a one up on installation and you basically setup the install process and say go. several hours later the system is ready. however, linux installs about 6 times faster than AIX. 2) I guess i could put my little tool out somewhere but i really think others could probably do a much better job of writing a more complete tool. I will try to compile our config and post it. Jeff Davis and Harry Duffey
Actually our exploration IT budget is probably closer to 4-5% of our exploration budget. You are only counting one project. We also have 150 Sun Workstations, several hundred Windoze PCs, 6 Sun Servers,SP2, Auspex, employees, software etc. Also, this sea-floor information is extremely expensive and adding just a little improvement could well cost close to a million dollars so it does really help. When you have all the information, it doesn't seem so bizarre and one would have to think that Jed's career here might be in jeopardy.
We work for Amerada Hess on the beowulf project in the article. The author of the article was genuinely interested in how we used Linux and the difficulties in getting our upper managment to sign off on the project. The Beowulf project at Amerada Hess has really opened some eyes about the power of linux and hopefully this article will spread the word. Slashdot, Freshmeat, Redhat, Dejanews, Dell, and Paralogic have really helped this project run smoothly. I would highly recommend any company with a need for serious computation and a few programmers to give beowulf a shot. It really works and it can really save you money Jeff Davis Harry Duffey Amerada Hess Corporation Houston, TX
Let's see well since a good Unix admin should be able to do more than setup systems. Well maybe the admin can do some programming. And, how about a little system tuning. Maybe setup a web server or mail server. How about a smooth upgrade. A good sysadmin sets up good systems but also chips in other areas, plans for the future, tries to make things run better. I would think paying someone the fix a OS that crashes is a waste of money. How about putting in a system that allows an employmee to add to the compnay in a wide variety of areas.
I thought this article was a very good read. Many people have lives like this and many don't realize how common it is. I really think that Katz has written some good articles and I look forward to his future work.
We have a 32 node beowulf running on Dell 410 Workstations. They installed flawlessly and are keeping up with our 32 node IBM SP2. They are good machines and the dell people are real high on Linux.
Slashdot Mirror
Most enterprises want to put on fixes that solve a problem. Sure it is nice to be a patch set on, but ultimately, the idea with patches is to fix/improve/add something. A good sysadmin will always keep current on security patches as a priority even if they have to be applied one at a time. The is most especially true on systems that are attached directly to internet.
The beauty of linux is that we just installed standard redhat (no tweaking) on the systems, hook them up on a switch and we have a supercomputer.
2. I would like to see journaled filesystems, more drivers for various hardware devices, better monitoring utilities, better installation tools for mass installs and upgrades. We would like to see gcc integrate the P3 mmx instruction set for floating point purposes.
3. We chose RedHat really because that was what Harry Duffey and myself were most familiar with. For our programmers and users, the version of linux was irrelevant. It doesn't hurt with upper management the RedHat is probably the most visible and known Linux vendor at this time. In fact, we are in the process of taking a look at FreeBSD, Debian, and Suse just to level check RedHat. I have not relied upon RedHat for any support. The few times we called RedHat, we had a hard time getting support. I typically use Dejanews and find almost all the support i need.
4. IBM is aware of what we are doing as they had an oppurtunity to sell us Netfinity PCS for our project. IBM has real nice systems (PC and Rs6ks) and they did a good job in our evaluation. However, all the PIII 500 performed almost exactly the same in our seismic processing benchmarks. Hence, we had no good reason to switch from Dell. I think IBM is very interested in the market. We have worked for IBM for years and they are good partners and I suspect if they are smart they will have an excellent Beowulf offering. I am not sure what this will mean for the SP2 though although I suspect for the time being, many large companies will buy the SP2 for name and reputation regardless of the price.
The Key Point is that our cluster is now up to 96 Nodes. Hence we can now run 3 jobs at the same time. Maybe my math is wrong but my guess is that our total through put is 2.5 times more than the sp2 and we still save 1.5Million.
We actually evaluated IBM Netfinity PIII 500s. They were good performers. However, they were about equal to the Dells and since we have already been using Dells, we saw no reason to switch at this time. But we will continue to evaluate systems based on price/performance as we grow our system larger than 96 nodes.
I agree. I wish my company would do just that. Makes for a longer career and probably a better environment which i think everyone is in favor of. Good points!!!! I believe lightsabers use some those generic batteries you can get at Wal-Mart
here is the code for my little dsh script. very simple and rudimentary but it works. all you do is create a file with node names in it say /collection1. no security per se so maybe someone can show me how to add ssh or something else.
Let /collection contain
node1
node2
node6
node8
Usage Examples:
export BCOLL=/collection1
dsh date
dsh /nfs/tech/update_kernel.22
dsh reboot
dsh "echo node1:/nfs/data /nfs/data nfs rw,bg,hard,intr >> /etc/fstab"
The code
#/bin/ksh
if [ "$BCOLL" = "" ]
then
echo "Environment Variable BCOLL not set"
echo "Please set BCOLL to a file with list of nodes it"
echo "export BCOLL=/root/allnodes"
echo
exit
fi
if [ "$1" = "" ]
then
echo "Please provide command to run"
echo "Usage: dsh command"
echo
exit
fi
echo "Collection: $BCOLL - Command: $@ - Date:`date`" >>
for i in `cat $BCOLL`
do
echo >
echo "Running $@ on $i" >>
echo >>
rsh $i "$@" >>
done
wait
for i in `cat $BCOLL`
do
echo
"----------------------------------------------
cat
rm -f
done
Jeff Davis
check out gas prices over the past 20 years. i bet you'll see that gas is one of the best bargains around. how much did a car cost about 10 years ago. I remember not paying much more now that i did then for gas. people always complain about the price of gas. but in fact, factoring in inflation it has gone up very little. in fact alot of the price of gas goes to taxes. maybe you should contact your local politicians and complain to them.
Speaking only for myself, not my company, we don't want to stain or destroy anything. Maybe when people stop using oil derived products, all this might stop. We love to buy fast cars, products made from plasics (sneakers, computer parts, etc), products made from chemicals (hair gel, medicines, etc), power to power a computer, home, car. Supply is a result of demand. If you want to decrease the use oil, maybe you should try to work on the demand. Lower the demand for oil, i bet oil companies will stop poking holes. No one complains when linux is used for nuclear weapons research. what do people think that linux is being used for in all these universities and labs. if linux is to grow, it needs to used by as many types of apps as possible. besides, linux is free for use by all. who should determine how it should be use. i think the spirit on linux means that everybody can use linux for whatever results the want. anyway, that is just my oppinion (not my companies as i do work at a corporation and have to deal with suits)
This also doesn't take into account software costs in which linux definitely saves a bundle cause its free. So far as I remember, the SP2 software is not free and has fairly hefty support/maintenance costs.
Have you priced out an SP2 lately. This was a tremendous price/performance savings.
1. Costs - Linux is clearly cheaper
2. Reliability - Linux was clearly more reliable which has been born out. Our clusters have now been up over 3 months straigt.
3. Scalability - How does one manage 96 NT boxes in a cluster. I had no idea. Unix has things like rsh and robust scripting making this a snap
4. Portability - it was relatively easy porting our sp2 apps to linux. by the way we use Linda from SCA (Scientific Computing).
1) The main tools we had to create was a similar program to IBM's PSSPs dsh (distributed shell). However, we didn't take the time to put in all the features that IBM has in their dsh. As of right now, their is no similar corollary to PSSP on Linux that I'm aware of. There is a product called SMILE (SCMS) which we were implementing beowulf wasn't quite ready. There is also a product call masshosts by John Mechalas which we are looking at. One nice thing about Linux/Unix is that there is really that much need to use these tools that often as the systems run and run without much need for interference. We also used the kickstart process to install the nodes. The SP2 definitely has a one up on installation and you basically setup the install process and say go. several hours later the system is ready. however, linux installs about 6 times faster than AIX. 2) I guess i could put my little tool out somewhere but i really think others could probably do a much better job of writing a more complete tool. I will try to compile our config and post it. Jeff Davis and Harry Duffey
Actually our exploration IT budget is probably closer to 4-5% of our exploration budget. You are only counting one project. We also have 150 Sun Workstations, several hundred Windoze PCs, 6 Sun Servers,SP2, Auspex, employees, software etc. Also, this sea-floor information is extremely expensive and adding just a little improvement could well cost close to a million dollars so it does really help. When you have all the information, it doesn't seem so bizarre and one would have to think that Jed's career here might be in jeopardy.
We work for Amerada Hess on the beowulf project in the article. The author of the article was genuinely interested in how we used Linux and the difficulties in getting our upper managment to sign off on the project. The Beowulf project at Amerada Hess has really opened some eyes about the power of linux and hopefully this article will spread the word. Slashdot, Freshmeat, Redhat, Dejanews, Dell, and Paralogic have really helped this project run smoothly. I would highly recommend any company with a need for serious computation and a few programmers to give beowulf a shot. It really works and it can really save you money Jeff Davis Harry Duffey Amerada Hess Corporation Houston, TX
Let's see well since a good Unix admin should be able to do more than setup systems. Well maybe the admin can do some programming. And, how about a little system tuning. Maybe setup a web server or mail server. How about a smooth upgrade. A good sysadmin sets up good systems but also chips in other areas, plans for the future, tries to make things run better. I would think paying someone the fix a OS that crashes is a waste of money. How about putting in a system that allows an employmee to add to the compnay in a wide variety of areas.
I thought this article was a very good read. Many people have lives like this and many don't realize how common it is. I really think that Katz has written some good articles and I look forward to his future work.
We have a 32 node beowulf running on Dell 410 Workstations. They installed flawlessly and are keeping up with our 32 node IBM SP2. They are good machines and the dell people are real high on Linux.