Slashdot Mirror
Flattening Out The Linux Cluster Learning Curve
editingwhiz writes "IT Manager's Journal has a good look at a forthcoming book, The Linux Enterprise Cluster, that explains in clear, concise language how to build a Linux enterprise cluster using open source tools. Writer Elizabeth Ferranini interviews author Karl Kopper in a Q&A. Is this more complicated than it appears to be? (IT Manager's Journal is part of OSTG.)"
Steepening the curve?
Now it's not just geeks, but also IT Managers who can imagine a beowulf cluster!
I must have missed this, and for anyone else who didn't know, OSTG is the new name for the Open Source Development Network (OSDN) Slashdot is a part of. They're now called the Open Source Technology Group.
The guy puts a single 10 node cluster together and this qualifies him him to write the 'definitive guidebook called "The Linux Enterprise Cluster"'.
Dont think so.
Do not try to read the dupe, thats impossible. Instead, only try to realize the truth
What truth?
There is no dupe
Installing and administering the various open source tools can be tedious work, especially without documentation of how to put things together.
A quick Google search though reveals a lot of free papers and manuals on this very topic.
...is that there are a gazillion ways to do it, and every cluster vendor comes up with their own way, and there is no agreed-upon standard yet to easily deploy these things (AFAIK). Now the fact that there is no single vendor controlling how clustering works is a good thing, without a lack of a good standard as to what a clustered environment will offer to the application developer, the task of setting up clusters for different types of applications remains a tedious task.
Lars Marowsky-Brée had a paper in the proceedings of OLS 2002 describing the problem and a suggeted solution in his paper entitled "The Open Clustering Framework". I'm not sure how far standardized clustering has come since then. Anyone has any insight on the matter?
"Backups are for wimps. Real men upload their data to an FTP site and have everyone else mirror it." -- Linus Torvalds
End users would often complain about the system's slow response time.
He says, "Because we couldn't print the forms for the warehouse people to select the products to be put on the truck, we'd have dozens of truck drivers sitting in the break room each day for more than 10 minutes.
I actually don't get it, most logistics got wireless for about a decade now...
and the truck driver has no right for a break...
Publications like this play an important role in establishing best practices and community, two key enablers of standardization.
These in turn will lead to greater adoption, and more publications. A virtuous cycle.
And please, don't be put off by VMS because DCL = your first exposure to a VMS system - feels more awkward than bash (in many ways, it certainly is!). It's in the underlying architecture of the OS where the fruits of tight engineering are really demonstrated.
And he built a ten node cluster OF ten node clusters, then this is lame. and he is under-qualified to do the book(most likely) as most ACTUAL enterprise clusters are at least 20 nodes, possibly more if its clusters of blade servers.
Who edited that article before it went live? It is a mess!
I will start by admitting that I am just a dumb university student talking out my ass. I have never set up an enterprise scale cluster.
However, last january we set up a small (six node) cluster with the help of CLIC. Once we realized the link between a Mandrake and consective dead CD drives, we installed the cluster in little time.
CLIC might focus a little too much on userfriendlyness and a little too little on flexibility, but for our purposes it was great. It sports ganglia, gexec, distcc and MPI (and probably more), and administration and deployment of nodes is a breeze.
I heartily recommend CLIC for student/test/proof-of-concept projects.
I thought at first it said "author Karl Popper"...now that would've been a trick.
Or is the editing in that Q&A just horrible? Spelling and grammatical errors abound... Someone should invest some time in Hooked on Phonics before writing an article about something they know nothing about...
I read about setting up a cluster about six months' back, and they said that you can only really run programs that are specifically designed to run on a beowulf cluster. It seems like if you could set up a cluster and be able to run any old app on it without special coding, then you'd have your massive adoption of linux. Plug-n-play supercomputer, using the crappy old boxes gathering dust under the cubicles.
Is there any plans to take beowulf in this direction? Is it already possible, but I was just reading the wrong FAQ?
Do what you can, with what you have, where you are.
Yet still no PT Cruiser? =)
Of course it's much more difficult than it appears to be... Ehm, look... Don't tell anyone, OK.
Deleted
A flat learning curve is a bad thing.
The term "learning curve" was invented by the aerospace industry in the 1930s as a way to quantify improved efficiency from mass production (basically, the more you do a task, the easier it becomes). The term was later adopted by psychology and the social sciences, where most people first encounter it.
In both cases, the horizontal axis of a learning curve represents time or effort, and the vertical axis represents amount learned or productivity. Therefore something that is intuitively obvious in fact has a steep learning curve.
"Learning curve" was a technical term with a specific definition for decades before it was ever a (misused) marketing buzzword.
Thank you for your time :)
The term "learning curve" was invented by the aerospace industry in the 1930s as a way to quantify improved efficiency from mass production
Aren't you confusing "learning curve" with "economy of scales" here?
"You mortals are so obtuse." -Q
They flatten it vertically. Wohoo! Zero investment, complete knowledge!
Im hearing everyone say how they would like a cluster that doesnt need cluster aware applications... Well check out openmosix. Its a simple kernel patch and userspace utilities that make a cluster that load balances any application.
Sometimes the majority just means all the morons are on the same side.
then it would be a time curve, not a learning curve. duh.
What might I use to cluster together linux servers such that they actually act as a single SMP machine?
I'd want to do more than just loadbalance webservices with it.
It want shell accounts on this cluster that act as one maineframe. People would shell into their home directories (which I suppose would all be from one big NFS), and run processes and whatnot on the entire cluster.
Any ideas?
Love zaq
A better term might be
"steep prerequisite" curve
i.e. each advancement to the learning process requires a much higher prerequisite.
If step 1. requires a high prerequisite then you would get the "running into a wall" effect.
Unless you're running a single-system-image cluster (i.e. mosix), and perhaps even then, cfengine is a godsend. It may seem a chore the first time you use it, but it's worth it. Just learn it. Use e.g. a pxe kickstart install that installs cfengine in postinstall, and sets it to run on boot. Make changes to your cfengine configuration, not on the nodes. That way when you inevitably replace something, it's brought right up to speed.
http://www.cfengine.org/
So now I have to buy another book to replace the other 34 books that I bought to put together a simple 4 machine cluster? I hate books.
I can't wait to beat someone with my new knowledge tomorrow! thanks!
My Linux Command of the Day site : LCOD
1) It's been ported to Itanium, and will ship on that platform very soon.
2) ISV support for VMS on Itanium is strong and its customers are very loyal.
3) You said you wish it would die and allow "better systems" to take over. What better systems? In the context of clustering (cos that's what this discussion is about) VMS Clustering for High Availability systems is still at the top of the food chain. Nothing exists on Linux to match it. Not yet anyway.
Tru64 Unix clustering is about the next best thing, but there won't be any further development there now until HP port TruCluster onto the next-generation of HP-UX. Unfortunately that's taking way too long, and by the time they get there it just might be irrelevant. OpenSSI.org may just eat its lunch.
OpenSSI (Single System Image) Clusters for Linux
The main features are: single root and single init, cluster filesystems and DLM, single process space and process migration, load leveling, single and shared IPC space, device space and networking space, and single management space
I have been looking at network filesystem level clustering and failover and NFS, SMB/CIFS and OpenAFS look like good choices for that. With NFS and CIFS you can have an active/inactive fail-over cluster.
I don't know about NFS, but in the case of CIFS, the protocol spec has provisions for renegotiating locks if a connection is broken, but I don't know if there are bugs in win2k/XP clients with samba 3 servers. OpenAFS can have a sort of active/active setup, but the archatecture is such that there is only one server that handles the writes and the rest are read-only. In all of these you can have a semi active/active failover cluster if you move half of the active volumes to the backup server, but this adds a lot to the complexity of your fail-over system.
Those services have a low to moderate amount of state information kept on the server. In the case of a graphical (VNC) terminal server, I don't know of any open source projects that will allow gnome session to be on one server, have that server go down, another server take over its ethernet MAC and IP address and continue processing where it left off on the backup server. The best I can think of is OpenMosix or maybe OpenSSI which are two single system image type clustering systems. If anyone knows anything, please reply and let me know thanks.
There: Something at a specific location.
Their: Owned by someone.
Please make sure your english compiles.
I think this is a very good idea because people where I live are always talking about "If I had a cluster" when they know almost nothing about Linux
First, about the book being a "definitive" guide. I cannot possibly claim to be an expert on every topic in the book--in fact, no one person can. The book is definitive, however, in that project leaders from each of the open source projects participated in editing and reviewing the material for the book.
It is an over broad statement to say it is the definitive guide for building any and all types of Linux Clusters. The book describes how to build a cluster that can be used to run mission critical applications to support an enterprise (it has little or nothing to do with working on the "Big Problem" as Pfister would call it).
(The book took four years to write by the way.)
I do hope it helps with the learning curve, but this is one of the advantages of building what I'm calling a Linux Enterprise Cluster--the system administrator can leverage his/her knowledge of Linux and add concepts that will allow them to build a cluster capable of supporting the enterprise.
I did not invent anything new for this book, and you CAN already find just about everything on-line that is in this book. I started work on the book in 2000 because, at the time, I wanted to have a guide book like this one that would hold my hand through the process of building a cluster that could support mission critical applications running GNU/Linux.
Finally, let me just agree with the comments about the number of nodes ("You don't need 20 nodes if 6 can do the job"). This book is not about building clusters for scientific applications where thousands of nodes and sophisticated batch job scheduling systems are required. How many nodes does it take to build the ideal cluster for your environment? I think that will depend on a lot of things including your budget, the impact of the failure of a single node in the cluster, how many instances of your application can run concurrently on a single node, performance bottlenecks from your node hardware, and so on. In my opinion, the ideal number of cluster nodes for an enterprise cluster--from the system administrator's standpoint--is about 10 (in a pinch you can log on to every node fairly quickly).
The cluster this book was based on has been in production long enough (over 18 months) to have undergone a complete hardware refresh by the way; so the text is based on actual experience (not just theory) and, as I mentioned earlier, it has been reviewed by subject matter experts to insure its technical accuracy.