How To Adopt 10 'Good' Unix Habits

An anonymous reader writes to mention an article at the IBM site from earlier this week, which purports to offer good Unix 'habits' to learn. The ten simple suggestions may be common sense to the seasoned admin, but users with less experience may find some helpful hints here. From the article: "Quote variables with caution - Always be careful with shell expansion and variable names. It is generally a good idea to enclose variable calls in double quotation marks, unless you have a good reason not to. Similarly, if you are directly following a variable name with alphanumeric text, be sure also to enclose the variable name in square brackets ([]) to distinguish it from the surrounding text. Otherwise, the shell interprets the trailing text as part of your variable name -- and most likely returns a null value."

Comments

FP?

[Shadow-isoHunt]

export POST="first"

welll..

[macadamia_harold]

An anonymous reader writes to mention an article at the IBM site from earlier this week, which purports to offer good Unix 'habits' to learn.

I seriously doubt reading this article is going to get anyone to start showering on a regular basis.

Re:welll..

[AchiIIe]

Some of the points he is making are BS. They are not good `Unix habits` they are simply hacks that marginally reduce the workload but (arguably) increase complexity.

Ie there is NOTHING bad about piping cats. While you might indeed get a ~30% performance increase if you skip the cat, the complexity increases. We often sacrifice performance in order to increase abstraction and understanding.

What makes unix so powerful is its modularity, the fact that you can pipe any output from any application to any applications stdin. This makes it possible to use common tools app1 | app2, app1longoutput | grep thingsIwant. The possibility to mix and match common elements that (arguably) makes unix powerful.

Advice that says "stop piping cats" is akin to "stop using helper functions, they overload the stack, instead do everything in one function"

--
A better articulated article on the programmers intellectual ability vs proper abstraction techniques:
http://www.acm.org/classics/oct95/ - Dijkstra, Edsger - "Go To Statement Considered Harmful"

Re:welll..

[SharpFang]

Ie there is NOTHING bad about piping cats

PETA would disagree.

Re:welll..

[t_ban]

Some of the points he is making are BS. They are not good `Unix habits` they are simply hacks that marginally reduce the workload but (arguably) increase complexity. Ie there is NOTHING bad about piping cats. While you might indeed get a ~30% performance increase if you skip the cat, the complexity increases. We often sacrifice performance in order to increase abstraction and understanding. What makes unix so powerful is its modularity, the fact that you can pipe any output from any application to any applications stdin. This makes it possible to use common tools app1 | app2, app1longoutput | grep thingsIwant. The possibility to mix and match common elements that (arguably) makes unix powerful. Advice that says "stop piping cats" is akin to "stop using helper functions, they overload the stack, instead do everything in one function"

but he never said you should stop using pipes. he is talking only about a specific situation -- cat-ing a file and then piping it to grep. surely that is a good point he is making, because grep already takes filenames as an argument?

Re:welll..

[johnw]

I argue that using> grep file1 file2 file3 regex can be more confusing Not least to the grep utility, which expects the regex first, not last.

Re:welll..

[Znork]

I'd tend to agree with the GP. Consider for example if you have excessively badly named files like '-whatever' in a particular directory; cat has very few destructive ways it can go wrong, other commands may be less forgiving, and cause much more surprise.

Further, the assembly line abstraction of cat as 'input the contents of these files into the beginning of my pipeline' is predictable, simple and very clear and readable. Using the filenames in the commands means you have to be certain each command will take filenames, and if you replace the first step (from a grep to an awk, for example), you have to rethink your input method semantics again.

Any typing speed gains and performance improvements you may get will probably get shot the first time some command does something unexpected, or by the extra steps of thought.

And if performance really was a serious concern you probably shouldnt be writing it as a shell script...

Re:welll..

[hackstraw]

cat-ing a file and then piping it to grep. surely that is a good point he is making, because grep already takes filenames as an argument?

That list was fairly arbitrary, but the piping cat thing is something that basically only annoys the most anal of anal, and they probably do it sometimes too.

Its common for me to do cat foo and then hit the up arrow and append a pipe to another command instead of editing the whole command line. Computers are pretty fast, and real anal people would use fgrep instead of grep, but again I always use egrep, because I never know when a regular expression will be edited into a more complex one, and to me all of the speeds are the same.

My #1 habit to tell people, although it is not a habit, but just where to start it to learn your shell. No science guys, csh is not a worthy shell in 2006. If you have to suffer with the wacky behavior of a csh variant, at least use tcsh.

My #2 thing to learn is a text editor.

As far as habits go. First and foremost, unalias cp, mv, rm to have the -i flag. In my opinion, that is a BAD habit to start. You WILL lose files sooner or later, and the more painful the better so that you will think so you will stop doing it. the -i flag will NOT stop you from redirecting into a file, and the most dangerous is the -rf flag with rm will override that -i. Remote copies via rcp or scp will not honor the -i flag. Unarchiving an archive will not honor the -i flag. There are tons of ways to lose files, and you will lose them. Its a much better habit to universally save yourself from yourself to not lose them by testing with -i, working off of a copy, and thinking before you hit return, creating new directories to eliminate clobbering a file, NEVER, EVER, do tar cf foo.tar . or tar cf foo.tar *. You will piss yourself and others by doing that.

Actually, this top 10 list is pretty lame, and should be ignored.

Re:welll..

[bugg]

I don't think it ever makes sense to use cat with one file - something I have seen far too many people do. To do so, logically, is to tell the commands to run through the file twice.

First you are telling cat to output the entire file, and then you are telling grep to go through the entire output of cat. If you're working with gigabytes of data here, that can quickly be a frustrating exercise! Folks who are in the mentality of using cut | grep and even a visual editor like vi instead of sed are up the creek when they find themselves needing to manipulate and get portions of very large data sets.

Re:welll..

[jc42]

Ie there is NOTHING bad about piping cats

PETA would disagree.

Oh? Just imagine the cacophony of a bunch of cats playing bagpipes while "singing" along.

Square or Curly brackets?

[Beolach]

enclose the variable name in square brackets ([])

~ $ ls tmp/
a b
~ $ VAR="tmp/*"
~ $ echo $VARa

~ $ echo "$VARa"

~ $ echo "${VAR}a"
tmp/*a
~ $ echo ${VAR}a
tmp/a

Their example correctly uses Curly brackets, {}, but their text says square brackets []. That seems like a typo to me.

Re:Square or Curly brackets?

[lmfr]

The correct form is {}, not []. There are other things you can use with ${VAR}:

• ${VAR:-text if $VAR is empty}
• ${VAR:=text if $VAR is empty and set VAR to this}
• ${VAR:+text if $VAR is set}
• ${#VAR} -> length of $VAR
• ${VAR#pattern} or ${VAR##pattern} -> remove match of pattern from beginning of $VAR (## -> longest match)
• ${VAR%pattern} or ${VAR%%pattern} -> remove match of pattern from end of $VAR (%% -> longest match)

There are other formats (see the man page), but these are the ones I use the most. Eg:

for i in *.png; do convert "$i" "${i%.*}.jpg"; done

Re:Don't use shell

[Anonymous Coward]

or even better -- use perl.

This article...

[nevali]

...is so littered with basic errors that it really shouldn't be recommended to anybody. How is 'tar xvf -C tmp/a/b/c newarc.tar.gz' expected to work, for example? Quote variables with square brackets? Running subshell commands using ; instead of && ? No mention of 'xargs -0' ? Don't pipe from cat to grep? Does anybody actually care that people do this (primarily so that the syntax is consistent between a munged- and unmunged-grep, and also such that the order of the command-line is logical from a human point of view)? Plus, of course, it's possible that cat | grep could yield better performance than grep alone: if cat uses mmap() to efficiently read the input files, and the kernel's pipe implementation is good, then it could do better than a grep implementation alone that simply read()s the files.

Re:This article...

[treat]

You're the only one who hasn't mentioned xargs -0. I think it's important to elaborate on this. You should never do "find | xargs" or "find | cpio", you should always do "find -print0 | xargs -0" and find -print0 | cpio -0". The former will break if filenames have spaces or newlines in them. You break xargs if filenames have quotes, backslashes, or spaces in them. I never come across a large data set where you can do find | xargs without the -0 options.

If you are encountering data created by untrusted users, don't forget the strange consequences of filenames that contain newlines.

Failing to use -0 is dangerous malpractice.

Re:Don't use shell

[Anonymous Coward]

OK, I agree. Please provide a concise Python script that unpacks a tarball (a .tar.gz or .tar.bz2 file), copies new files in to a said tarball, patches based on the contents of said new files, runs make from various directories in said extracted tarball, and then changes the name of the top-level directory created by the tarball to a new name and repacks the tarball.

Or a concise Python script that opens up a text file of URLs, and extracts the files listed in the URLs:

#!/bin/sh
for a in $( cat file | awk '{print "'\''" $0 "'\''"}' ) ; do
              wget $a
done


Python has it place, and is far better for medium to large projects, and projects where the code needs to be maintainable. Shell, however, works a lot better for automating UNIX tasks than Python does. Not to mention embedded systems: I can compile Busybox to have both a good shell and all of the commands that one would run from shell scripts (including grep, cut, sed, and, yes, awk) in only about 300k. A Python binary is about a megabyte big, and you need about ten megabytes to fit all of the libraries Python 2.4 comes with.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:Don't use shell

[Anonymous Coward]

Someone mod up parent and grandparent, PLEASE!

No, don't mod up anybody in this thread. Perl and Python are abominations. Pure, unadulterated Bourne shell is for the true, seasoned *nix user. Just like Java is an answer to a question nobody asked in the GUI world, so too is Perl and Python in the command line world.

Very helpful

[EvanED]

I really like this example:

~ $ time grep and tmp/a/longfile.txt | wc -l
2811

real 0m0.097s
user 0m0.006s
sys 0m0.032s
~ $ time grep -c and tmp/a/longfile.txt
2811

real 0m0.013s
user 0m0.006s
sys 0m0.005s

I am so glad that he showed what a difference can make, because I was *really* getting annoyed at having to wait that extra .084 seconds.

Re:Very helpful

[trip11]

There is no such thing as .084 seconds. Surely you mean .084 hours.

Re:Don't use shell

[Bazman]

If I've got a simple task to do (eg the text-file-of-URLS example) then I knock it up in shell script. By the time that simple task has feature-creeped up to more than 20 lines I start to wish I'd written it in Perl. So I rewrite. By the time that Perl script has crept up to more than 200 lines I start to wish it was written in Python. So I rewrite. By the time that Python script has crept up to 2000 lines I start to wish I'd farmed the job out to a team of programmers, and I give up caring what language its written in and make them do it as a web service. Then I write a small shell script to call their web service. When that shell script has feature-creeped up to more than 20 lines...

Things I had to learn the hard way

[gd23ka]

1. Don't rm with an absolute path because you could easily

#rm -r -f / tmp/dir

when "all" you wanted was

#rm -r -f /tmp/dir

instead do this:

#(cd /tmp ; rm -r -f dir)

or even better use sudo if you have it:

$(cd /tmp ; sudo rm -r -f dir)

2. When logged on as root or when using sudo on a production system think things over
at least twice before hitting enter.

3. Make sure at all times you're on the right machine, logged on as the right user in the right directory.
Set up your shell prompt to look like this user@host /path$

absolute drivel

[Anonymous Coward]

This is, without a doubt, the most worthless article I have ever seen, both on Slashdot and on ibm.com, of which I thought better. It is not that the article is boring, but that it is factually incorrect in some places.

"the only excuse to define directories individually was that your mkdir implementation did not support this option, but this is no longer true on most systems. IBM, AIX®, mkdir, GNU mkdir, and others that conform to the Single UNIX Specification now have this option."

This is nonsense. The expansion of the path components in the {braces} is not a function of mkdir(1), but of the shell, and how its argument expansion is configured. I cannot believe that anyone "with 20 years of experience" is brazenly quoting names of standards in an effort to give his ramblings an air of credibility. Actually, wait a minute...

Another bad usage pattern is moving a .tar archive file to a certain directory because it happens to be the directory you want to extract it ...

Better is to check what's in the archive before extracting it in case some inconsiderate fool has failed to put a top-level directory in it.

His research interests include digital publishing and the future of the book.

Let me give you a couple of hints.

Re:mkdir

[SharpFang]

Especially the habit of using || and && on command line seems ridiculous to me. These have room in two situations:
- scripts
- commands that take long enough that you go have a coffee.

This makes sense:

make install && lilo && reboot

This doesn't:

cd tmp/a/b/c || mkdir -p tmp/a/b/c

If you fail the first part, well, you typed " || " instead of pressing enter.
If you succeed the first part, you typed " || mkdir -p tmp/a/b/c" without a bloody reason.

Type first part. Press enter. Observe result.
If necessary, type the second part, otherwise correct the first without baggage of the second one hanging around.

Re:Don't use shell

[Anonymous Coward]

This code is not pure shellscript : it uses awk and wget to get the job done...

A Python equivalent might be :

#!/usr/bin/python
import os
for a in file('filename').readlines():
  os.system('wget ' + a)

It's not that much longer, it's much easier to read and less error-prone (especially the awk part), and it uses fewer external utilities.

To me, the *only* advantage of shellscript is that it's the only language that you are sure to find on any Unix system.

Re:Don't use shell

[Haeleth]

This code is not pure shellscript : it uses awk and wget to get the job done...

That's what a pure shellscript is! The whole point of shell scripting is that you use the shell script as glue to tie together simple single-purpose utilities that come as standard with every flavour of Unix ever.*

To me, the *only* advantage of shellscript is that it's the only language that you are sure to find on any Unix system.

No shit, Sherlock! You have clearly never worked in a large organisation, where - believe it or not - you, as a standard user, do not actually get to insist that the already-overworked IT department jump through bureaucratic hoops to install your favourite bloated scripting language, unless you have a damn good business case for it. And probably not even then.

Hint: if the task you want that scripting language to accomplish is trivial to achieve with a simple shell script, you don't have a good business case.

* This doesn't apply to wget, obviously, but if your platform really has no standard alternative, you are more likely to persuade IT to install something small and simple like wget, fetch, curl, etc. than a complete programming environment like Python.

Eh? There's more to Unix than shell scripting

[Taagehornet]

"10 good habits that improve your UNIX command line efficiency" would probably have been a better title.

The title did however bring back fond memories of Eric Raymond's The Art of Unix Programming. The book is available online, and if you were hoping for something a bit more substantial as well, then the section Basics of the Unix Philosophy might be worth a read.

Re:Don't use shell

[duguk]

Um, whats wrong with

wget -i filename

Or have I missed something?

Monkeyboi

Actually useful hints

[Artraze]

As has been pointed out, this article is riddled with errors. It's also not very interesting. So in the interest of perhaps actually providing some interesting tips:

In scripts, prefix dangersous commands with an 'echo' for a test run (So you can catch all those rm -rf /).

Single quotes are the best quotes for plain strings. The only reasion to use double quotes is if you need to quote a variable or a single quote.

Completion is fun, but using wildcards is more flexible (though you'll only want to use benign commans like cd, less, etc):
    nano /etc/modules.autoload.d/kernel-2.6
    nano /etc/m*a*d/*6

Note that the use of subpaths reduces the amount of flexibility.
    cd /etc/m* -> /etc/mail
    cd /etc/m*d -> /etc/modules.d
    nano /*/m*/*6 -> /etc/modules.autoload.d/kernel-2.6, and /etc/modules.d/i386 (not quite!)

Finally, as a comment for the article, using:
    test -e $DIR || mkdir -p $DIR
is much better than their suggestion and probaly faster anyway. Though I'd just do "mkdir -p $DIR" and maybe "&>/dev/null" under most circumstances anyway.

That's all I can think of at this point. Anyone else have tips?

No actual habits in the article

[xyloplax]

Here are some more important general IT rules (Unix rules can easily be OS and version dependent and frequently come from usage in YOUR environment)

• Copy before edit
• Tape backup before delete/decommission
• READ YOUR COMMAND before hitting return
• Check where things are symlinked to
• Echo in your scripts instead of destructive commands as a test run
• Test your changes on a lesser-importance box
• Use proper Change Control procedures
• Cover your ass and capture your terminal output
• When taking something out of service, turn it off for a few days/weeks before deleting/purging it

Re:Don't use shell

[JohnFluxx]

Wow, do you think you could be just a little bit more polite next time?

Re:lowercase uppercase

[multipartmixed]

Oh, for the love of God, stop bitching about how hard this is to do under UNIX without package x.y.z installed and imagine doing it under Mac OS9 or Windows.

Break it down into its constituent parts: Iterate, rename. Whoo! Simple now, eh?

# cd cameraDir
# find . -type f -prune | while read file
> do
> mv "$file" "`echo \"$filename\" | tr '[A-Z]' '[a-z]'`"
> done
#

... should do the trick. If you don't have tr, you can use sed with the y command. But I can't think of the last time I saw a box w/o tr.

bad ibm no cookie

[illuminatedwax]

Great, IBM, way to ignore the dreaded "xargs" security bug! Seriously, IBM notices some kind of obscure danger about underscores, but completely ignores the fact that xargs separates arguments by newlines??

Let's say I'm a sysadmin and I'm running as root, trying to remove all the files in the /tmp directory by a certain user for some reason:
find /tmp -user 1001 | xargs rm

User 1001 has a directory in /tmp called "haxor\n". Inside there he puts another directory "etc" and inside there he puts a file called "passwd."

Can you guess what happens?
find prints: /tmp/tmp43cc91 /tmp/haxor /tmp/haxor /etc/passwd

xargs sees: ["/tmp/tmp43cc91","/tmp/haxor","","/tmp/haxor","/e tc/passwd"]
Oops!! You just hosed your system!

The correct way to use xargs is to use the -0 switch, which will separate the input by null characters, which cannot appear in filenames. find has a handy -print0 option which will output the correct output:

find /tmp -user 1001 -print0 | xargs -0 rm

And your system is safe.

Re:If I wanted to upload binaries...

[lahi]

Right. So 10 *really* good Unix "habits" would be:
1. Never use csh or any derivative thereof.
2. Know the portable behaviour of your Unix tools.
3. Learn to use ed, one day you'll be glad you did. You can also use ed and ex from scripts or from a command.
4. A shell command is a small program. If you are unsure about a command, test it first, like you would any program.
5. Learn to use the standard shell on your system.
6. Learn useful nonstandard extensions of utilities, but use them with care.
7. Never rely on an extension to the point that you forget how to do it portably. The definition of "portably" is up to you.
8. Learn to use csh enough that you can make do in an emergency, and learn *why* you shouldn't use it.
9. If your standard shell is Bash, learn Korn too. And vice versa. Learn both, how they differ, and how they differ form your standard shell.
10. Sometimes a real C program or a script in a different language is better than using shell.

-Lasse