10 Dos and Don'ts To Make Sysadmins' Lives Easier
CowboyRobot writes "Tom Limoncelli has a piece in 'Queue' summarizing the Computer-Human Interaction for Management of Information Technology's list of how to make software that is easy to install, maintain, and upgrade. FTA: '#2. DON'T make the administrative interface a GUI. System administrators need a command-line tool for constructing repeatable processes. Procedures are best documented by providing commands that we can copy and paste from the procedure document to the command line.'"
Command line was used by Nazis to kill Jews, and is unconstitutional.
I, for one, welcome our new command line overlords.
In Soviet Russia, line commands you!
Obligatory xkcd
.
I hate every term in that series.
1. DO switch every don't to a do and do to a don't on that list. You are now a user.
10 is an even number. There's no duplicates. None of them are filler.
I don't understand how this happened.
Did someone plan this before they wrote it? What gives?
slashdot: where everyone yells sarcastic metaphors to themselves to understand the issue
nt
Kwisatz Haderach
Sell the spice to CHOAM
This Mahdi took Shaddam's Throne
I agree with most of these; however, I'd revise #3 to refer to "a substantial API" and not the minimal, in terms of usefulness execrable, deposits of code provided by some software.
DO NOT be a sysadmin
In essence, all 10 items on the list say "Use Linux!"
Yeah, ok, thank you Captain Obvious, I mean CHIMIT :P
-Billco, Fnarg.com
It's a top-10 list that actually has insightful information on how to do software right, instead of being a random collection of ten things to make a fluff article. Bonus points for being things that I actually agree with.
The article author is also behind The Practice of System and Network Administration, truly an excellent text into the practicalities of work in IT.
(blush)
If you want to make a sysadmin's life easier (as if any programmer ever wants to do that), you can start by making your error and status messages 1.) plentiful and 2.) easy to understand. Also, provide several logging levels so we can drill down as needed, and make sure the logging levels are meaningful. Too many programmers put just two log levels: one which shows nothing useful, and another that spews out indecipherable hex dumps of every call it makes.
Face up to the fact that no matter how awesome your software is, it's going to fail. Not only that, but it's going to fail in ways you never thought possible at the worst possible times. Make sure we have enough information to figure out what happened. Otherwise, stuff like this happens:
Program: *crash for no apparent reason*
Sysadmin: Why did you crash?
Program: Because something went wrong.
Sysadmin: What went wrong?
Program: Something.
Sysadmin: I need more detail. Increasing log level.
Program: Something bad went wrong.
Sysadmin: I need more than that. Increasing log level again.
Program: Fuck you. Here's a 16GB hex dump of system memory. Figure it out yourself jackass.
Sysadmin: *picks up a crowbar and goes off to find the programmer*
Don't make me use a real browser to click all the way through your site, make me agree to a stupid set of conditions for using the software, and then provide my browser with a cookie that it can subsequently use to download your software; when my browser is on one continent and the machine that wants the software is on another continent; you ass-fucks...
10 is an even number. There's no duplicates. None of them are filler.
I don't understand how this happened.
I know how they came up with a high-quality top ten: They had 13 or so, and they cut the weakest ones.
Some of work in the field and we should be able to read the documentation via telnet or lynx. It's not always convienient to grab a laptop, sometimes there's no wireless availiable and either all the switch ports are used or getting internet access is a kafkaesque nightmare. Yes I have a smartphone -- it's not an efficient way to browse documentation!
> DO have a configuration file that is an ASCII file, not a binary blob.
And by ASCII we mean something that can be edited by any editor.
XML is the equivalent of a binary blob when you are up to your ass in alligators trying to get things working again with minimal tools available.
2. DON'T make the administrative interface a GUI.
Amen, the number of times I have dumped on products because of the lack of a CLI is almost rude and funnily enough it saves a lot in licensing costs so "almost" everyone is happy. Pretty pictures and buttons will get you past the management and sales but if you come near my systems with your "button pushing monkey" toys expect your time in the building to be very short indeed.
I demand a meta top 10. The top ten list of top 10 lists.
...if the GUI is well done and complements command line.. Some tasks actually ARE much better performed with Point&Click.
One example of a "good" GUI that I use a lot is the ASDM for Cisco ASA firewalls. Most of the simpler admin tasks are in fact *faster* via ASDM. If you have your network objects all properly set up and you need to add a firewall rule, it's far simpler to select it from a list (actually, in this case it's a combobox - just type first few letters to filter your choices and then click) than typing that stuff in manually. Packet tracer to check the rules is much nicer to use via the GUI. Setting up VPN profiles is simpler via ASDM. Handling network object groupings is simpler via ASDM.
Editing access-lists, doing routing configuration and most of the more "rudimentary" tasks are still something I do via command line, though.
All very good suggestions. But as a programmer it's my job to ignore these. Thank you come again. I joke, I try my best to make my tools admin friendly. Because that admin might be me.
Tiger Blooded Bi-Winning Machine
You sir have no idea what it is sysadmins do. Try doing the job some time and you will see that these items make that 1 sysadmin managing 10k computers possible and saves the company millions.
APK likes to ask for responses to the same things over and over. Maybe he just likes the responses?
if it fails in a way that you never thought possible, how would you write an error message that describes the failure?
This list is a good start. Don't consider purchasing software that doesn't meet these criteria.
No sig for you. YOU GET NO SIG!
8. [...] Similarly, use the operating system's built-in authentication system and standard I/O systems.
This can be a bad thing if your application runs on a platform whose built-in authentication is a nickel-and-dime revenue stream for the platform's publisher. Microsoft Windows Server is like this: each user account on the built-in authentication system requires a Client Access License.
Feel free to make a GUI for the administrative interface, but not at the expense of an underlying CLI.
There are two ways to do this: have your GUI call the CLI when necessary, or use a common API behind both. Other methods will lead to bitrot in one of the interfaces, most likely the CLI.
GUIs are fine and even enjoyable to a certain extent, but the author is right that the CLI takes priority.
My Amiga was a computer that was equally comfortable with CLI and GUI interfaces, and may programs with GUIs had plenty of shell commands with the same executable. I'm not sure why the rest of the computer industry turned it into a religious war that continues to this day.
Some planning and thought are all that's required to make a balanced interface that handles both methods well.
I manage almost exclusively Linux servers and i must say the command line saves me ooodles of time. Some quirks can be alleviated by just restarting some services before they run out of memory, some needs a bit more magic but nothing takes time like having to login to many computers every day and click on the same friggin GUI stuff on multiple servers.
Bash saves me time by totally taking repetitive tasks away. Ive tried the same with some Windows machines but while powershell has potential, it does not work in reality unless you are a 100% Microsoft shop, and you happen to run the limited set of applications that has full support for powershell.
Maybe in time Windows will climb up to the level of Linux when it comes to manageability but right now i spend most of my time doing repetitive stuff on my Windows boxes while i write scripts that handles anything on the Linux boxes.
HTTP/1.1 400
Unless you want me to drop the product and choose something else less irritating. Hello VMWare? Xen?
- If you don't provide documentation, requiring the sysadmin to dig through the source code for configuration information, please add some useful comments and meaningful function names.
- Make the option parsing code clear enough so we can at lease see what words are used for the options. Cute parsing code may save a millionth of a second each time the program is run, but is useless if we can't figure out how to configure it.
- If you have a configuration file, tell us where it is and what it is named, at a very minimum. Scanning a binary file for strings to locate the configuration file is a major pain.
- Don't develop yet another build system. Especially if it only works on one specific version of a specific operating system on specific hardware.
- Don't make every error message exactly the same. Trying to figure out which one of the fifty "oops" messages was triggered is painful.
Who would win this election: Andrew Weiner vs Andrew Weiner's weiner.
We have a hard job because we have morons like you writing our software. If we have competent people, it goes fine and we don't need to whine and bitch and moan because there's nothing to whine and bitch and moan about.
Now go write some more trivial business apps for me, please.
Copy Pasta Security®
10 is an even number. There's no duplicates. None of them are filler. I don't understand how this happened. Did someone plan this before they wrote it? What gives?
Its an acm.org article. Not only did the author probably plan, re-read and revise the article before submitting it but a technically knowledgable editor probably read it and may have offered useful and insightful suggestions. Now there may not have been a formal peer review process but the editor may have also had one or more experts in the field read it and offer comments and suggestions.
;-)
Yes the above seems an archaic process but consider that the acm is full of old people who had experience publishing back when things were done with dead trees.
You sound like one of those people that piss all over the public bathroom floor and say, "ha! it's someone's job to clean it up! why should I care!" (since you made the janitorial analogy, I thought I'd complete it)
The biggest change in business in the last 10 years is the realization that IT is not a "cost to be reduced" but the driver of innovation that should be invested in, respected, and optimized.
This reads like a specification for building a unix system.
Those who don't understand Unix are doomed to reinvent it... or something like that.
Alex, I'll take keybindings not used by Emacs for $400....
Once on a live solaris box, "swapoff" was executing too slowly for me and I wanted to delete the swapfile (yes it was) to free up some discspace.
Because I could:
> /swapfile
and a squicro-second later the box just slammed to a halt
had to drive to the data centre to reboot it.
Posted anon cost I want to re-tell this at a christmas party without someone spoiling the ending
In reference to point 8, this is something I wrote I while ago after dealing with several Windows apps that either horribly abused the Eventlog or refused to use it entirely:
Do not assume that your software is running with elevated access... (root/administrator)
Make sure its clear whether you meant '10' in base 2, 8, or 16.
Have gnu, will travel.
"None of them is filler." But you get full credit for the question marks. So many people seem to think they are optional.
Damping absorbs vibrations. Dampening is caused by moisture.
A GUI is NOT fine for administering a broken system over a slow link to the other side of the world.
I used to remotely administer a set of servers in the middle east. The bandwidth was tiny, and the latency was insane. I would type a command out, then take a sip of coffee while waiting to see it displayed before hitting "enter." I had to use a GUI for one application, and it took over 40 minutes to fire up and display on my machine.
Mandatory (and well-designed) GUIs should be for using an application, not administering or installing it.
"People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
#10 should probably be #1. Support and documentation is everything. Because when it hits the fan, finding the original install CDs or manual is almost always a requirement. It's also why I stopped buying Nvidia cards. They got rid of almost all of their patches and drivers as well as installation CDs on their site and now force you to use their "all in one" tool. And lo and behold, you're screwed 90% of the time with an older machine if you don't have the original install CD because it simply doesn't work without the CD.
Case in point - I tried to recover an old machine's crashed system(video drivers and dirext X had eaten themselves when "upgrading" as is typical) - but the online driver was useless. The original CD was the only option, but it wasn't to be found. (as is typical, few customers keep driver CDs where they can find them). The manufacturer didn't have the original CD to download, either.(honestly, a 50mb ISO file isn't going to kill their server space) I had to buy a new card to solve what should have been a ten minute problem. Nobody was happy about it, either, as you would imagine, since the card wasn't even two years old at the time.
(note - a "roll back" option also needs to be available when "upgrading") I'd wager that 95% of the time it is simply not there.
Dealing with some really awful "enterprise" logging and file transport replacement software packages right now (replaced good old Solaris boxes). Every blinkin' update they "enhance" the web gui's to make them flashier and more useless.
Instead, why not try using, oh, I dunno, "tar" and "make" and friends -- you know, the standard 'nix tools that every system administrator has been working with quite happily for decades and which suffice nicely to install tens of thousands of software packages ranging from the dirt-simple to the incredibly complex.
I'm looking at you, SAS.
The list is constructed for quality, intelligent system administrators who strives to understand and maintain the underlying system. Instead we have an idiot man-child who thinks system administration is clicking on the right button and rebooting. He's completely befuddled if he has to type in a command, or edit a file as if either of these tasks is like constructing your own working nuclear reactor. I actually believe most applications are targeted at this level of stupidity.
But hey, I'm not bitter about it at all.
The IT equivalent of a Janitor? I know at the ISP I work for our server admin isn't considered that way at all. I would love to know half as much as he does about how our servers run and how everything is put together.
Then again being at an ISP we tend to put a little more value on keeping the network alive, and I think this is not the case at standard corporations.
Regardless, the tips given in TFA are not just for sysadmins. I don't consider myself sysadmin-I'm a standard software engineer, and I would also like to see programs with the characteristics given. It's not just helpful to one group, it's solid programming advice.
All the world's a CPU, and all the men and women merely AI agents
How about layering a nice GUI on top of command line tools? This allows repeatability, scriptability, etc., but also can provide access to things a GUI does more easily, such as browse for files, perhaps, or pick your favorite feature. It's not like all GUIs suck at all things.
Currently hooked on AMP
As long as we're talking sysadmin duties, which imply that you have ONE admin to many boxes. If it's any kind of setting that a user should have to touch, please make it part of the GUI. A good GUI is infinitely more structured and helpful than a long, long flat man page of cli switches. The reason you want a CLI is scripting, not user friendliness.
Live today, because you never know what tomorrow brings
1. DO have a "silent install" option.
Silent install is nice, but so is an intelligent install, or a well thought-out, correctable upgrade process.
These systems do it well:
Debian and RedHat derived; Windows, post-2003. OS install is still a bit of a bitch with Windows. The upgrade process for MediaWiki is also stupid easy and effective (basically: untar new tree and run db alter scripts).
Poorly:
FreeBSD, and, really, most BSDs, are horrible for upgrading. I suspect OS X is similarly stupid when it comes to "promptless installs". Cacti, likewise, is awful.
2. DON'T make the administrative interface a GUI.
A useful amendment to this is: don't make the administrative interface shitty. GUI is fine, as long as I can leverage it progmatically. CLI tool is great, as long as it's fucking documented and not obtuse.
Case in point (in opposition): MegaCLI, for MegaRAID cards. Absolute. Shit.
3. DO create an API so that the system can be remotely administered.
An API is great, and allows for programmers to dig in and extend the product. I'm thinking of VMWare, XenServer, and Virtualbox right now. The latest Windows versions with PowerShell and the management consoles are not a bad combination of usability/power/utility.
Most sysadmins don't have the time to dig into the API, though, so a good initial tool that isn't terribly dense or limited in functionality is a must (XenServer, please improve your shitty-useless UI on xsconsole and XenCenter; I'd like a little more access to my VM disks without digging into lv/pv commands, too).
4. DO have a configuration file that is an ASCII file, not a binary blob.
No argument here. Likewise, configuration should be human-readable and not have vague incantations.
Good: samba, and all tools which use similar configuration syntax.
Bad: sendmail is the worst offender I can think of at the moment. I'm sure all the djb* stuff, too.
5. DO include a clearly defined method to restore all user data, a single user's data, and individual items (for example, one e-mail message). The method to make backups is a prerequisite, obviously, but we care primarily about the restore procedures.
Good: any UNIX system and it's $HOME; modern Unix MTAs like Courier.
Bad: Cyrus IMAP. Pretty much any tape archive system comes close to frustrating as hell. Windows still has a long way to improve until it's capable of Unix-style $HOME utility.
6. DO instrument the system so that we can monitor more than just, "Is it up or down?"
WMI is great. SNMP on Unix/Linux hosts, not so much, due to the configuration and divergence involved. Most OEM Linux/Unix based machines or systems (XenServer) are relatively shitty in this regard, too.
7. DO tell us about security issues.
Telling us about them is great, but upgrading these things are the most important, time-sensitive upgrades we need to make, so they should also be the easiest. We should not have to break two-three different things just to get the upgrade done.
BSDs are bad about this; horrible, even. The time consumed by a simple upgrade is enormous.
Linux is mediocre, but better than most.
Windows, in this case, "just works". Except when it doesn't (though I'd argue the degree is no greater than, say, the Linux upgrade process). Your biggest cost will be when it installs something you've explicitly told it not to (*cough* new IE versions) or in bandwidth and/or uptime requirements.
8. DO use the built-in system logging mechanism (Unix syslog or Windows Event Logs).
Something which doesn't do this isn't even worth looking at. It's yet one more thing to manage and uses exponential
Addition: make your logging sensible, please. I don't want to see a full trace of everything in the logs and not be able to configura
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
With a pure GUI interface you completely rule out any sort of automated configuration tool (puppet etc) and can turn a single person job into one where you have large numbers of temps running around kicking people off their PCs just so they can click their mouse on a few things on the screen. While the programmer may think their program deserves the individual attention of somebody sitting down at the console and pointing at pretty pictures there is always the possibility that the people configuring it have other things to do and may want to run it on more than one machine. Even if you think it is a one per company thing some large place may get a hundred of them and put one person in charge of deploying them. That one person shall curse you and you GUI and wish they were given an interface where things could be scripted instead of having to behave like a three year old pointing at pretty pictures.
There are many that cannot grasp this idea, being stuck with the non-networked single user idea we saw with the Apple ][ and got stuck with when Microsoft hit us with a cut down version of CP/M without the concept of multiple users. Time to grow out of the concept of a three year old pointing at pretty pictures and understand that we can use words to tell the machines what to do. Present it as both if you like but NEVER make us sit through a dozen mouse clicks on one hundred different machines if we need to do one hundred installs.
A GUI is NOT fine for administering a broken system over a slow link to the other side of the world.
Agreed, or when the broadband connection is down (or DDOSed)
And especially when the *tards demand *cough* Adobe *cough* be deployed across 12K machines running different OS versions, in different States, in different time zones.... a packaging nightmare at the best of times. Add in a few hundred headless boxes and cli becomes the only feasible way to implement the changes.
Maybe, just maybe, when the economic fuckup we're in becomes obvious to Accounting - the same cost/benefit tests that are applied to the purchase of a chair will be applied to IT decisions
In the meantime IT decisions are not being made by IT managers. And the people who choose the chairs are not getting free trips to the Gold Coast, and winning Golf Buggies (sigh)
I on the other hand am suprised that nobody has equated the poster's name to Lemoncello and it's mascot.
http://www.youtube.com/watch?v=46wakJ8oggM
don't do anything microsoft does.
The grunt-work of installing and re-installing software has turned into a great job creation machine. True, it's boring and mundane most of the time, but jobs is jobs. And it can't (yet) be off-shored to Asia.
Also, vendors generally don't want to simplify and standardize the installation process because it also simplifies pirating. A mess is harder to clone and copy than a logical and consistent arrangement.
Table-ized A.I.
That would be close to the top of my list. You need at least two levels: minor fixes, and major upgrades. Fixes should be minimal, make the system strictly better, and not get in the way of things.
Java is the example of how not to do it: 1.6.0u23 - and upping the last digit has often introduced new features and broken other things (plus the installer is extremely unpredictable). Adobe gets at least the version numbers right, but their upgrade path is often a miracle. Firefox gets it right, most of the time.
2. DON'T make the administrative interface a GUI. System administrators need a command-line tool for constructing repeatable processes. Procedures are best documented by providing commands that we can copy and paste from the procedure document to the command line. We cannot achieve the same repeatability when the instructions are: "Checkmark the 3rd and 5th options, but not the 2nd option, then click OK." Sysadmins do not want a GUI that requires 25 clicks for each new user. We want to craft the commands to be executed in a text editor or generate them via Perl, Python, or PowerShell.
Since I've had to work with Windows servers in my new job, I thought I'd better read up on them, so I've been reading Windows Server 2008: The Definitive Guide. The sections on the underlying principles and theory of the OS are fine. But that's one third of the text, at most. Most of the text is useless blow-by-blow accounts of sequences of clicks in GUI applets. It's completely unreadable -- the descriptions are meaningless unless you're working through the instructions with an instance of Windows Server 2008 in front of you. And who's going to set up several instances, just to make sense of the description of the applet for configuring load balancing?
I can't blame the book, particularly, as it's a problem of GUIs.My workplace has lots of documents with step-by-step instructions for configuring services, which have one sentence of text, followed by a screenshot, followed by another sentence of text, and another screenshot, and so on.
On the flip side, one of the great things about text configuration files is that while they're full of obscure configuration options, they're also full of the documentation explaining the obscure configuration options. Config files are rich with documentation. GUI configuration applets frequently aren't. I'll take a documented option in a config file over an undocumented option in a GUI config applet any day.
Not even that. All applications should have their functionality usable from the command line.
I am trolling
mod parent up. I had to administer a windows server that was in Malaysia. The client had the fastest ADSL line that could be installed at their location, and I had a massive 15kb/s bandwidth up from them. Remote desktop was a pain in the rear at these speeds. Not only that - but this line was serving several hundred remote users through special client software.
If there was ever a problem we couldn't solve it during work hours, because remote desktop used all of that bandwidth, even at 1024x768 B&W
shudder
The CLI is not important, the things that you can do with a CLI are important. You can easily use it with a crappy network connection (SSH is usable over GPRS or an analogue modem). You can script it. You can document exactly what the process is, easily.
If you can provide all of this functionality without a CLI, then by all means do so. A web interface might be better than a CLI for remote access. A proper set of scripting APIs might be a lot better than a CLI for scripting. A record and replay facility for all configuration changes at the layer below both the scripting API and the GUI might be better for reproducibility.
I am TheRaven on Soylent News
Actually I know exactly what you all do. You whine and pule and cry like you have a hard job, and you bitch and moan and get basic concepts incorrect and regurgitate things we programmers invented that you ill understand.
Go unclog the toilet.
On behalf of all admins - please accept my apologies.
Your .net programming skills are awesome - you what? wrote a program that performs mathematical equations? only how many megabytes? you are awesome!
Not only do I love the idea of a digital calculator - I also admire your cubicle art. Have you ever considered painting in a colour other than brown?