Linux Implementation For 2500 Workstations?

Jeff Kwiatkowski asks: "We are looking to roll out Linux to over 2500 desktops and could use any advice that we can get. We need security info, implementation suggestions and any other advice that you would care to offer. We are currently evaluating Debian, Caldera and Red Hat. I also want a minimalist desktop, so I have been leaning toward WindowMaker as the window manager. In addition, we currently have machines with 32 meg of RAM (fast processors, though) and would like to keep the upgrade to 64 meg, only, if possible. Lastly, do any of you have any thoughts on Word Perfect vs. Applixware?" For those of you who think that the claim 'Linux is not ready for the desktop' is a falsehood, then this story is for you. As you can see, people are looking at deploying Linux on the desktop, and suggestions from you guys could make this process a lot easier.

Comments

Define requirements better

[hatless]

For an installation even one-tenth of this size, X terminals and big servers are the way to go. And no, the terminals don't need to be fast. We use P133s with 32MB just fine. Administering two, or even ten servers is a whole lot easier than administering 2500 independent workstations, even with the slickest use of NIS and NFS to pull off centralized configuration and management.

How much document sharing will there be with organizations that use other (read: Microsoft or Corel) office software? For interoperability and the shallowest learning curve for MS Office users, Corel WordPerfect Office and StarOffice are the best choices. Both will read existing MS Office files well enough, and will export new files in that format well enough, too. Neither is up to the task of heavy back-and-forth collaboration with MS Office users, but that's true of any office suite on any platform. StarOffice can't deal with most WordPerfect Office files. This would be a non-issue under 98% of circumstances, as so little of the world really uses WordPerfect anymore. However, as a state court system, you're part of the 2%: WordPerfect still has a strong presence at law offices, so interview a representative sample of users and managers to find out whether they currently do receive a notable number of WordPerfect files via e-mail or on disk. Frankly, they probably don't, instead getting them via fax or in hardcopy. So it may well not be an issue.

This answered and all other things being equal, I'd opt for StarOffice; Sun is in better financial shape these days than Corel, and bulky though StarOffice is, it's also natively written for Unix/Linux. It's also got more features that suit it for network and large-environment use. Both Corel and StarOffice are available in identical versions for Windows, so the laptop brigade can work seamlessly withe the terminal crowd.

For another thing, many job functions don't require an office suite at all. Don't assume everyone needs one, and don't just reflexively give one out, even if it's free of license fees like StarOffice. If someone just sends simple faxes and email, that's all they should be able to do. If someone simply accesses an AS/400 or mainframe and works with e-mail, they need access to nothing more than a web browser for e-mail (or perhaps Netscape Communicator with its IMAP mail support), and a tn5250 emulator. For sending faxes, use email-to-fax and fax-to-email gateways. Hylafax is your friend. And incidentally, StarOffice has nice hooks for printing and "emailing" through networked Hylafax servers. It can also sync with Palm gizmos.

For more complex environments, StarOffice has some further advantages. Enterprise-caliber support contracts are available, as are user and administration courses. Macros and scripts for it can be written in Javascript or in VBA. MS Office VBA scripts themselves won't work, but the skills some users and managers may have can be leveraged very easily. For another thing, it can be scripted with and interact with Java. So what? Ah, here's the nifty enterprise-caliber part: not only can StarOffice 5.2 access ODBC databases. It can also access any JDBC data source--which means pretty much any database on the planet, regardless of OS. In addition, you can take advantage of Java toolkits and SDKs for all sorts of things. For example, IBM has a toolkit for Java access for AS/400 client APIs. Want to add a menu item to StarOffice for retrieving data directly from a mainframe into a spreadsheet? Or linking calendar items (have I mentioned StarOffice's Outlook-like group calendaring?) dirtectly to AS/400 screens? You can. In ways that can be reused on other platforms and environments.

What Linux distribution you use is the least important piece of this. Choose something that offers easy creation of kickstart disks, to ease installation on new machines, and that offers good, reliable security upgrades. Me, I'd go with something that offers decent commercial support contracts. The terminals won't need any such nonsense, but it sure is nice to know you can get an engineer on the phone if a $50,000 server has a memory leak you can't squash. There are several ways to do this: you can go with a Linux vendor that offers contracts, like RedHat, or with a hardware vendor that sells Linux OS support on its hardware, like an IBM or VA Linux. For this reason, Mandrake comes out weaker than RedHat. It's not about the quality of the default installer or the number of "extras" on the CD. It's about maintainability going forward and the quality of support you can buy for those times when Usenet and online documentation cost too much downtime.

Depending on what you need in the way of file storage or backend applications, you may well want to look into a commercial Unix (say, Solaris, which runs StarOffice splendidly) on the backend. Linux is great, but if your needs really call for a SAN (and it doesn't sound like they do), or you want to go with one gigantic 24-CPU server instead of several 4-CPU ones, Linux may not cut it. Don't compromise your implementation for politics. Keep in mind that at this level, Linux is Unix is Unix, and that things like data and email and so forth can move fluidly between flavors without a moment's thought. Linux can certainly support 2500 users well; it can support tens of thousands of users well, as most large universities can tell you. There are also things it can do that something like Solaris can't, such as attach seamlessly to Windows file shares. But plan your applications and your network before you make the final OS decision, so the OS doesn't force compromises.

And another nice thing about these Linux X terminals is their flexibility. Just need green-screen VT102/3270/5250 access in the mailroom? Can do. Need to mix in some Windows-only applications after all? Set up a Metaframe server and give the X terminals access to an ICA client. Want users to be able to save to floppy, or attach a barcode reader? You can. With no changes on the terminals themselves.

A matter of taste. Centralization.

[Hanno]

On the question of distributions...

I use Suse on the desktop and found it to be *very* practical. But mostly, the question "what distribution shall I use?" comes down to "that's a matter of taste".

There are differences in their target audience, of course. Corel Linux is targeted mainly at beginners, while RedHat, Suse, Mandrake and Caldera try to be useful "for everyone" (they all can be installed and used by dummies but they all offer features for the pros, too). Debian is a little different because unlike the previous distributions, it doesn't offer automated configuration scripts. Again, it is solely a matter of taste if you like Suse's "yast" to mangle your configuration files for you (I do) or if you prefer to edit them by hand. Those people using Debian tell me that they are very happy with it (especially with its easy upgrade process and its security model).

Installing on auto-pilot...

BTW, if your machines are 2500 identical hardware setups, you could easily create one reference linux setup and copy the entire harddisk across the network, using a simple custom bootdisk and the "dd" command. Also, all distributions offer automated install features (ask their support about it) so that you just put in the CD and they auto-install your custom setup.

Centralization...

Reading from your requirement list, you may want to hire a Unix (semi-)professional for the setup. I mean, 2500 machines!

There are a number of Unix features that can make life easier in such a situation, so it's good to have someone who knows how to setup things like that...

Here are a few ways of centralizing things in the Unix world. Each step means a bit more centralization and means that the server must be more powerful and vice versa that the client doesn't have to be a powerful, fast machine anymore.

- You can use a centralized NIS/YP server for the user and password administration, which will make life a lot easier for your admin. I have never done this myself, but I worked with such a setup at University and it was incredibly practical.

- How about setting up a central file server for the user's home directories? If all user-related information is mounted via NFS, your workstations can easily be replaced and employees can easily move offices. Just login on someone else's machine and your personal files are right there.

- Next, you could setup a central file server that contains all the application binaries. This makes updates easy and avoids the need to upgrade your workstations' harddisks.

- And the final step would be to make all those machines pure X-Terminals that only run the X-Server and a local window manager, while the applications run on a central server. I don't know if this is for you, since this requires buying new powerful servers. On the other hand, since 32 MB is more than enough for an X-Terminal, you can avoid buying RAM for 2500 machines.

Some more thoughts on memory...

If you want all the applications to run locally, you should choose a minimalistic window manager (not KDE and not Gnome, both work with 32 MB, but ask for more) and Applixware. I have tried Applixware and it runs fine on a small machine, but I mostly use StarOffice now. StarOffice is a memory hog, though, so it isn't an alternative for you. (I have not tried Word Perfect, so I cannot judge about it.)

Finally...

Good luck for your project, but please expect a few weeks, possibly even months of time before everything works smoothly. The sheer size of your network is a true challenge.

------------------

Re:Tailored installation, user/system separation

[austad]

Mandrake 7.1 has an option at the end that asks "Make a boot floppy for Linux replication?" which allows you to just pop the disk in to clone the install on another machine, getting rid of all of the interactive stuff. I haven't tried it yet, but I have used redhat's version, mkkickstart, and it required alot of messing around with the options in the config file before it would work. This was in Redhat 6.1 that I had problems.

Mandrake 7.1 seems to be a bit more user friendly than redhat, and comes with more tools. It will download and install crypto packages like ssh, pgp, gpg, openssl, and mod_ssl during the install if you have it plugged into a network with internet access too. On login, the user can select what windowmanager he wants to use, they have a choice of about 8 or 10 of them, KDE and Windowmaker included. I use windowmaker and the only thing I don't like about it is it's lack of a decent pager/virtual desktop system. I want to be able to scroll between my desktops with the mouse instead of having to click on something. :)

Probably the easiest way to do this is buy the Mandrake box set so you get all of the goodies CD's, copy them to an NFS or FTP server on you network, do a network install with all of the programs options that you want, make sure you set it up to get an address via dhcp, answer yes to the "make boot floppy for linux replication" question at the end, and start replicating. With the Redhat version, all of your machine have to be EXACTLY alike, including the exact same hard disk because the floppy stores how big each partition is and the start and ending block, so if you install on a machine with a different size disk, you're out of luck.

User Community?

[garver]

The type of user community that you are supporting is very important. I am guessing at two different types of groups: operations and business users.

1. Operations: These people run one or two applications and don't care about the computer underneath. The workstation might as well be an appliance. Linux is ready for this since it provides a stable, light platform and you don't run into compatibility issues.
2. Business Users: These people run anything and everything and need to import files from other groups. In other words, compatibility is very important for this group. I don't think Linux is ready for this, and god bless you if you are trying it.

As for Operations, here are my tips:

1. Automate the install, completely. I think you will find plenty of support for this out there now. Either scripted network installs or blowing an image onto a hard disk. Worse case, you end up with a boot disk that runs a script to make the filesystems, mount a NFS image, copy the image locally, run a script to patch up local info (like IP address and hostname), run lilo, and reboot. Since everything is tangible in Linux (e.g. no stinking registry!), this shouldn't be too bad.
2. Make all of the machines rubber stamps of each other. If one dies, you should be able to plop another down, power it on, log in the user, and they are home. To do this, make sure all of their non-transient data goes into their home directory. Anything that goes into /tmp better not be needed. This is one area that Linux sings in. It should be a no-brainer to make this happen.
3. Come up with some sort of diagnostics test that you can run to help determine what part of a workstation is failing (e.g. memory, video, etc.). Otherwise, just lease the computers and let someone else fix the hardware.

For Business Users, good luck. I would first worry about whether you can solve the compatibility issue. From there, try to do as much of the above as possible. Mobile users will make it more interesting. Keep a replicated copy of their home directory local and a boot option/script to pick Mobile/Office.

Have fun.

Use the terminals as terminals

[Oestergaard]

My suggestion: Leave the terminals with 32 meg, as that is plenty for the X server. Your standard terminal should have one single harddrive with a minimal installation of some distribution. You will want to have a standard way of setting up such a machine when a disk dies and gets replaced, but backups aren't needed (as there is no user data on the terminal) and you won't have to care much about updates either (as there should be no daemons running on the terminals).

One big benefit: As the terminals do not hold data, it doesn't matter if they are stolen. Terminals are not trusted.

The X protocol is made for networks, and a 10 MBit/s hose to each terminal would be just great. However, it's not encrypted, so you should at least consider how physically secure your network is, and what the requirements would be.

Then set up one server for each N users. If they are doing web access and text editing, your average ``high end but not that high'' server should be able to run 15-40 users. Maybe more, but I haven't tried this type of workload myself so I can't say. Anyone ?

You will end up with a server farm. Each server should hold a home filesystem locally, and preferrably the users with the homes on that local fs should log in on that server. You can choose to let the server export their home fs'es to the other servers as well and share user accounts with NIS, which would let any user log in anywhere. If a terminal is tied to a user and vice versa, there should be no need for a terminal to be able to choose other servers, but if they're not, then the need will be there.

I've done a few such setups, but at a *much* smaller scale. I can tell you that it is a relief to _only_ have to update software on the server(s).

Large rollouts

[Phill Hugo]

Back at University, we had a very hacked up Slackware distribution which did nothing special, expect on bootup where it would download and install any packages that sat on the upgrade server.

The same principle is absolutely essential for anything more than 100 or so machines (even if upgrades aren't a priority, bug fixes and security fixes will be).

In truth, I can't imagine any distribution would be better suited than any another here, especially if you are willing to write a boot up script which can download any new RPMs or DEBs and install them. The only problem is making sure they are not "interactively installed". Lots of Debian packages are but this is easily remedied. In fact, if you used Debian, adding apt-get update && apt-get dist-upgrade to your boot script and setting up your own packages repository (a simple FTP folder) would do that for you. You may need to tweak the odd package to force some settings but that's what your network of 5 machines reserved for testing are for right...

I'd also go with Sawfish/Sawmill instead of Window Maker. While I'm a huge fan of WM, I think sawfish has a much more desktop friendly future ahead. It can also look pretty identical to WM, and some of the other themes are very practical for desktop use. Its memory footprint on my machine is just under 4MB with half of that as shared libs which lots of other programs are using. Perhaps a choice at login would be useful, especially if offered with something pretty like GDM.

The major issue will probably be support, although that's more likely to be for specific applications than the whole system. I take it that to be entrusted to install 2500 desktops, you know your greps from your seds and are pretty capable of writing some scripts to manage upgrades. If not, find someone who is and pick their brains.

Re:I'll let others slug it out over desktop ideas.

[Ramses0]

Rather than bogging down the network with remote X apps, *please* investigate Debian's apt-get tool. In some ways, the Debian distribution is a 10,000+ distributed cluster of homogenous systems.

For my home Debian box, all I have to do is run apt-get update; apt-get upgrade once a day, and then my system is homogenous with the official Debian distribution.

If you put those two commands into your user's init scripts (probably with the --force option), then lock down the /etc/apt/sources.list, then...

1. Your system is just as secure as any other unix system.
2. You have a central point of software management (your local "debian mirror"), which you control completely.
3. You can guarantee that anytime a user turns on a computer, it will be sync'ed with the master server. (or put it in a cron job if your expect computers to be left on. apt will happily run in the background, and update most things without intervention)
4. Debian has been doing this for years under much weirder conditions, and dammit, it just works.

The biggest disadvantage of using apt-get is that your network will probably get bogged down after you change a large package. The next morning, at 8am, when 1000 people turn on their computers, they'll all be trying to download the same package at the same time, which could be a mini nightmare. If you're a good sysadmin, you'll figure out a good way around it.

The other big advantages:

1. You can have different ~groups~ of systems-set your sources to ftp://workstations.packages.myplace.org/, then ftp://development.packages.myplace.org/
2. You already have a great policy to work from, specifically about maintaining the central repository. Why reinvent the wheel?
3. Debian software is guaranteed not to get you into licensing trouble. Pretty much every package in Debian meets the Debian Free Software Guidelines, which is what the Open Source Definition is based on.
4. Debian has an overwhelming support infrastructure. Nothing much commercial, but every package has a maintainer (who you can contact), and debian has a central bugtracking system, used to keep track of what bugs are found in which packages, and eventually send those patches back to the original package authors.

If you don't need to roll out this installation tomorrow, I'd recommend that you install a copy of Debian (Debian 2.1 is stable, but out of date, Debian 2.2 is not quite released yet).

Once you install Debian 2.1, hang out for a while talking to people on the irc channels (irc.debian.org), and get all your stuff configured, then run the command apt-get update; apt-get dist-upgrade, and your distribution will automatically be upgraded from 2.1 to 2.2 (hopefully with almost no user intervention).

This message turned out to be a lot longer than I expected, but there's a lot to consider in your situation. Good luck!

--Robert

debian + replicator!

[raphinou]

Debian is easy to install and update through the network. And there is a package replicator available to replicate an machine installation through the network. It's really great. The replicator maintainer is also very responsive. I had lots of mail exchange with him and he helped me if I needed. got to http://www.ens-lyon.fr/~schaumat/replicator/

Tailored installation, user/system separation

[Morgaine]

For a rollout of that size, I'd say that you need two key things: first, either a network or CDR-based install from a cut-down release tailored to your business environment, with all options pre-selected, and secondly, the seemingly trivial but massively important separation of system and user areas, each in their own filestore.

The first is important because one of your major costs is going to be support --- this will skyrocket if you use standard distro CDs because they're all based on interactive user choice in varying degrees, and corporate handholding costs money.

The second is important because without the separation, upgrading will become a nightmare over time --- again, this will increase your support costs. In fact, consider seriously the possibility of not holding any user data on the workstations at all, but on a central filestore instead. That simplifies data backup as well as workstation upgrading, because then you can regard workstation state as throwaway.

I'll let others slug it out over desktop ideas...

[trims]

... I'm a Sys/Net Architect, so guess where my biases are? :-)

Anyway, what you have on the desktop matters (esp the mechanism you use for clone workstations (you are planning to clone workstations, right?)), but I'll concentrate on something else equally important, and which will affect how you set up the desktops: Network and Backend System Design

First off, you don't want any data locally. That's right. I don't care who has the workstation, the only thing sitting on the local disk should be the OS. All user files, and major applications should be sitting on a remote filesystem. Otherwise, you end up with a completely intractible backup and upgrade problem. Trust me on this.

As a correllary to the last statement, you don't want to use NFS as your file sharing method. Hell, even SMB would be better. You want to look at either AFS or Coda. I would recommend the latter, as it's nowhere near as nasty to set up.

As part of Coda/AFS, you are going to have to think about how you design your file server setup. A central bank of servers is tempting, but this tends to be really harsh on the campus backbone, as it puts the workstation relatively "far" from the server, and all traffic has to traverse the backbone. Consider local file servers which may cache user data for replication back to the master server(s) later.

Printing is also a bit of a problem. I heartily recommend the CUPS system talked about here a couple of days ago. Have all your workstations spool to dedicated print servers. They don't have to be powerful, but make them dedicated. You won't regret it.

As far as security and other mishmash goes, do the usual /etc/inetd.conf edit, and comment EVERYTHING out. Don't run ANY daemons on the clients (other than what is absolutely necessary for Coda). Have all mail blindly forwarded to a central mail server. As a correllary, use IMAP (preferably IMAP-over-SSL) as your mail server. Stay away from local UNIX mail, and POP. And look at running postfix or exim instead of sendmail.

You can think about using application servers (i.e. run X apps remotely) if you want, but realize that this will up the bandwidth requirement, and honestly, you probably can't run more than two dozen major X apps over a LAN before it bogs down completely. That is, you need a local app server with 100Mbit connections to about 25 machines so each can run 1 or 2 X apps remotely.

If you can afford it, and have the time, use LDAP as your user info directory - avoid NIS and NIS+ (the first is horribly insecure, and the second is nasty).

This is a first approximation of what you might do. If you want a serious proposal, I'm available nights and weekends (for a modest fee, of course... heehee)

Good luck!

-Erik

No Beowulf comment?!?!

[blogan]

There's been about 80 posts already and not one refers to a Beowulf cluster. Come on! He has 2500 machines here. Keep on your toes!