How To Adopt 10 'Good' Unix Habits

An anonymous reader writes to mention an article at the IBM site from earlier this week, which purports to offer good Unix 'habits' to learn. The ten simple suggestions may be common sense to the seasoned admin, but users with less experience may find some helpful hints here. From the article: "Quote variables with caution - Always be careful with shell expansion and variable names. It is generally a good idea to enclose variable calls in double quotation marks, unless you have a good reason not to. Similarly, if you are directly following a variable name with alphanumeric text, be sure also to enclose the variable name in square brackets ([]) to distinguish it from the surrounding text. Otherwise, the shell interprets the trailing text as part of your variable name -- and most likely returns a null value."

Comments

FP?

[Shadow-isoHunt]

export POST="first"

Re:FP?

[Anonymous Coward]

Imbecile! POST wasn't defined yet! This won't work on /bin/sh, only with the proprietary bash shell.

Re:FP?

[lahi]

Actually it will work with Korn shell as well, and probably zsh too. Not to mention that many systems have a /bin/sh which is Bourne-compatible but enhanced. Many systems have a /bin/sh implementation that supports this, not just bash-based Linux systems.

-Lasse

Re:FP?

[Mr Z]

Proprietary? From the man page:

Bash is intended to be a conformant implementation of the IEEE POSIX Shell and Tools specification (IEEE Working Group 1003.2). Bash can be configured to be POSIX-conformant by default.

welll..

[macadamia_harold]

An anonymous reader writes to mention an article at the IBM site from earlier this week, which purports to offer good Unix 'habits' to learn.

I seriously doubt reading this article is going to get anyone to start showering on a regular basis.

Re:welll..

[AchiIIe]

Some of the points he is making are BS. They are not good `Unix habits` they are simply hacks that marginally reduce the workload but (arguably) increase complexity.

Ie there is NOTHING bad about piping cats. While you might indeed get a ~30% performance increase if you skip the cat, the complexity increases. We often sacrifice performance in order to increase abstraction and understanding.

What makes unix so powerful is its modularity, the fact that you can pipe any output from any application to any applications stdin. This makes it possible to use common tools app1 | app2, app1longoutput | grep thingsIwant. The possibility to mix and match common elements that (arguably) makes unix powerful.

Advice that says "stop piping cats" is akin to "stop using helper functions, they overload the stack, instead do everything in one function"

--
A better articulated article on the programmers intellectual ability vs proper abstraction techniques:
http://www.acm.org/classics/oct95/ - Dijkstra, Edsger - "Go To Statement Considered Harmful"

Re:welll..

[SharpFang]

Ie there is NOTHING bad about piping cats

PETA would disagree.

Re:welll..

[t_ban]

Some of the points he is making are BS. They are not good `Unix habits` they are simply hacks that marginally reduce the workload but (arguably) increase complexity. Ie there is NOTHING bad about piping cats. While you might indeed get a ~30% performance increase if you skip the cat, the complexity increases. We often sacrifice performance in order to increase abstraction and understanding. What makes unix so powerful is its modularity, the fact that you can pipe any output from any application to any applications stdin. This makes it possible to use common tools app1 | app2, app1longoutput | grep thingsIwant. The possibility to mix and match common elements that (arguably) makes unix powerful. Advice that says "stop piping cats" is akin to "stop using helper functions, they overload the stack, instead do everything in one function"

but he never said you should stop using pipes. he is talking only about a specific situation -- cat-ing a file and then piping it to grep. surely that is a good point he is making, because grep already takes filenames as an argument?

Re:welll..

[Anonymous Coward]

Ie there is NOTHING bad about piping cats.

Why are you increasing complexity by advocating avoidance of stdin command line arguments? It appears that you are standing in the way of progress in stdin innovation. Unnecessary pipes are for the weak-minded and dim-witted. Perhaps you'd be more comfortable with Windoze's crippled cmd.exe, eh?

Re:welll..

[AchiIIe]

I argue that using > grep file1 file2 file3 regex can be more confusing than > cat file1 file2 file3 | grep regex since grep is almost always used with a single argument. When you go back 1 month later and you try to figure out what's going on, the first example is more likely to confuse you. ~ $ time cat tmp/a/longfile.txt | grep and 2811 real 0m0.015s user 0m0.003s sys 0m0.013s ~ $ time grep and tmp/a/longfile.txt 2811 real 0m0.010s user 0m0.006s sys 0m0.004s ~ $ I argue that the extra 0.005 (!) seconds you saved by avoiding the cat is worthless when compared against the loss of abstraction.

Re:welll..

[johnw]

I argue that using> grep file1 file2 file3 regex can be more confusing Not least to the grep utility, which expects the regex first, not last.

Re:welll..

[Znork]

I'd tend to agree with the GP. Consider for example if you have excessively badly named files like '-whatever' in a particular directory; cat has very few destructive ways it can go wrong, other commands may be less forgiving, and cause much more surprise.

Further, the assembly line abstraction of cat as 'input the contents of these files into the beginning of my pipeline' is predictable, simple and very clear and readable. Using the filenames in the commands means you have to be certain each command will take filenames, and if you replace the first step (from a grep to an awk, for example), you have to rethink your input method semantics again.

Any typing speed gains and performance improvements you may get will probably get shot the first time some command does something unexpected, or by the extra steps of thought.

And if performance really was a serious concern you probably shouldnt be writing it as a shell script...

Re:welll..

[Anonymous Coward]

Yeah piping cats, it's not a big deal, I totally agree with you. For everyday stuff while I'm actually _on_ the command line I don't worry about it.

You obviously know and I totally don't propose that I am teaching _you_ anything about usage though when I say that in a script, when perhaps you can potentially avoid thousands of instances of cat, you shouldn't do it.

This is an IBM article. The only AIX boxes I Admin have hundreds of users on them. I have to keep that in mind, even when running something like cat, in a script.

Re:welll..

[hackstraw]

cat-ing a file and then piping it to grep. surely that is a good point he is making, because grep already takes filenames as an argument?

That list was fairly arbitrary, but the piping cat thing is something that basically only annoys the most anal of anal, and they probably do it sometimes too.

Its common for me to do cat foo and then hit the up arrow and append a pipe to another command instead of editing the whole command line. Computers are pretty fast, and real anal people would use fgrep instead of grep, but again I always use egrep, because I never know when a regular expression will be edited into a more complex one, and to me all of the speeds are the same.

My #1 habit to tell people, although it is not a habit, but just where to start it to learn your shell. No science guys, csh is not a worthy shell in 2006. If you have to suffer with the wacky behavior of a csh variant, at least use tcsh.

My #2 thing to learn is a text editor.

As far as habits go. First and foremost, unalias cp, mv, rm to have the -i flag. In my opinion, that is a BAD habit to start. You WILL lose files sooner or later, and the more painful the better so that you will think so you will stop doing it. the -i flag will NOT stop you from redirecting into a file, and the most dangerous is the -rf flag with rm will override that -i. Remote copies via rcp or scp will not honor the -i flag. Unarchiving an archive will not honor the -i flag. There are tons of ways to lose files, and you will lose them. Its a much better habit to universally save yourself from yourself to not lose them by testing with -i, working off of a copy, and thinking before you hit return, creating new directories to eliminate clobbering a file, NEVER, EVER, do tar cf foo.tar . or tar cf foo.tar *. You will piss yourself and others by doing that.

Actually, this top 10 list is pretty lame, and should be ignored.

Re:welll..

[bugg]

I don't think it ever makes sense to use cat with one file - something I have seen far too many people do. To do so, logically, is to tell the commands to run through the file twice.

First you are telling cat to output the entire file, and then you are telling grep to go through the entire output of cat. If you're working with gigabytes of data here, that can quickly be a frustrating exercise! Folks who are in the mentality of using cut | grep and even a visual editor like vi instead of sed are up the creek when they find themselves needing to manipulate and get portions of very large data sets.

Re:welll..

[theonetruekeebler]

The best reason to pipe to grep is to keep filenames out of the output. grep foo ?.txt will produce
a.txt: foo
b.txt: foo
c.txt: foo
whereas cat ?.txt|grep foo produces
foo
foo
foo

I've also seen Unixes where their shells are linked against spectacularly broken libc's. Under Tru64's Bourne and Korn shells, for example, a multithreaded program foo fork bombs when run as foo < z.txt, but works fine as cat z.txt|foo (foo < z.txt under Bash works, though, because Bash is linked against the GNU libc).

Re:welll..

[nightfire-unique]

Damn you... this has had me giggling for the last 10 minutes. :)

Re:welll..

[cuantar]

No science guys, csh is not a worthy shell in 2006. If you have to suffer with the wacky behavior of a csh variant, at least use tcsh.

I admin a linux cluster for the high energy physics group at my university, and although we're moving away from fortran and PAW in favor of C++ and ROOT, most of my users insist on tcsh. I suspect the reason may have something to do with the fact that less than half of them really know there's a difference between tcsh and, for example, bash. Every time I have to debug some strange behavior related to someone's environment (usually just something like an nfs share being down) it occurs to me that ~/.tcshrc is almost a carbon copy of everyone else's. :) A few people wrote some useful things in tcsh, and since no one else knew or cared enough to rewrite them in bash, they ended up just being copied around. It can be quite frustrating sometimes to have to maintain two copies of scripts I've written in bash that people wanted in tcsh as well.

Re:welll..

[Anonymous Coward]

"IE: there is NOTHING bad about piping cats"

I disagree completely, there are some very serious side effects.

http://http//uncyclopedia.org/wiki/Kitten_Huffing# Undesirable_Effects_of_Kitten_Huffing

Re:welll..

[ultranova]

I argue that the extra 0.005 (!) seconds you saved by avoiding the cat is worthless when compared against the loss of abstraction.

How is giving a filename to cat and then piping the output to grep any clearer (or more abstract) than just giving grep the filename directly ? If anything, it's harder to figure out, since you need to know two programs (cat and grep) instead of just one (grep) to determine the outcome.

Abstraction only makes sense if it hides some underlaying details or allows you to apply the same code to a multitude of problems; simply adding complexity for complexity's sake is stupid.

Re:welll..

[pthisis]

You can still build pipelines without cat.

For instance, instead of:

cat file |grep regex

you can just use a redirect at the beginning of the line:

file grep regex

There's not a really good reason to use cat unless you're concatenating multiple files or using one of cat's filtering flags. OTOH it's fairly harmless and not really worth being on the list, either.

Re:welll..

[pthisis]

It makes sense if cat is doing something more than just reading the file (e.g. "cat -n" to add line numbers, "cat -s" to eliminate multiple blank lines in a row, etc).

But other than that, there's practically never a reason to cat a single file. Even if you want the filename before the command, a simple redirect like:

< file grep foo

works fine.

Re:welll..

[SanityInAnarchy]

I agree, partly -- I don't like to try to figure out what things like gzip will do with actual filename arguments, and instead use gzip and bzip2 either as part of tar, or with stdin and stdout.

But still, why would you need cat for this? Just do:

gzip < file > file.gz

Re:welll..

[Anonymous Coward]

Looks like a new grep version in on the loose!

Re:welll..

[AndroidCat]

I thought people should stop piping cats because the fur was clogging the tubes of the Interwiggy thing.

Re:welll..

[dgatwood]

I believe that would be < file grep regex....

&lt; is your friend.

Re:welll..

[TheLink]

The article talks about _efficiency_ as if it's the most important thing: "Adopt 10 good habits that improve your UNIX® command line efficiency "

Once you have a human manually sending commandlines to a shell, efficiency is far from the most important goal. Correctness and robustness are more important.

If you are doing the same thing very often, write a _correct_ and _robust_ program to do it, then you can get the program to be more efficient.

What next- an article on top 10 "good" programming habits, which just lists the fastest ways to type different stuff in vi or emacs? Or end up being just a bunch of one-liners?

Sheesh.

Re:welll..

[jc42]

Ie there is NOTHING bad about piping cats

PETA would disagree.

Oh? Just imagine the cacophony of a bunch of cats playing bagpipes while "singing" along.

Re:welll..

[lahi]

I have several colleagues, who have the habit of logging on to an AIX system (which has ksh as /bin/sh) and then type tcsh to get a shell with "sensible" editing. One of them used to tease me about my fierce evangelism for using the default Korn shell instead (and just type set -o emacs to get editing) rather than tcsh. He stopped when one day he did a cd to a log directory, then grep somepattern * >filename. The partition on which this directory was located was just about half full already, and thus soon filled up completely. We lost a few minutes of log data before he discovered the problem.

tcsh is just as borken as csh. Dangerously braindead if you ask me. The Perl hacker Tom Christiansen has written the Csh considered harmful FAQ, which everybody using csh should read. To my surprise I just discovered he doesn't mention this particular bug.

-Lasse

Re:welll..

[camperdave]

Speaking as a novice, cat|grep is easier to understand. Until I skimmed the fancy article, I was unaware that grep could take a filename like that. I thought that to do filenames you needed fgrep. To an experienced unix guru, it may not be any clearer, but to a novice there's a world of difference.

Re:welll..

[Onan]

If what you'd like to change is the behaviour of grep as a tool, then I'd suggest that the most correct solution is altering grep's behaviour, rather than working around it.

ie, grep -h

Re:welll..

[theonetruekeebler]

grep -h In practical life I only ever use grep against (a) one file, (b) all of them, or (c) a stream. The [ivlnc] switches are all I can ever remember from day to day. Brainspace I could use for the rest is already occupied by things like my wife's birthday.

Re:welll..

[Mr Z]

I'm lost: How is "grep somepattern * >filename" any different in tcsh versus bash, except in the case where the directory's empty? It seems like both shells should set up the redirection before forking off the grep, which means it should be equally disasterous in either shell. I guess tcsh does the glob after setting up the redirection, whereas bash does it before?

Interestingly, this steps near the one thing I think tcsh gets right over bash: If a wildcard matches nothing it should not return itself (as it does in bash), it should generate a trappable error. Is there a way to configure bash to generate an error on a wildcard that matches nothing?

--Joe

Re:welll..

[sholden]

They give different output.

In many cases the "cat ... | grep ..." version is what you want. And I think it's pretty safe to assume that more people know the cat command than know the -h option to grep.

The "cat | grep" versions also works better when you change it to zcat or to something silly like:
for i in *;do zcat "$i" 2>/dev/null || cat "$i";done | grep ...

Re:welll..

[db32]

I frequently cat bleh | grep whatever just because it was the first way I learned and reflexively do it that way. grep whatever * actually tells me what file it found the match in, rather than just showing me the line. So while piping cats isn't inherently bad, there is frequently better ways to do it that also give more correct output, but it depends on what the desired output is. However "I grepped all the files for 'bleh'" sounds infinitely less disturbing than "I grepped what came out of the cat I piped"

Re:welll..

[norton_I]

I always use grep's command line filenames in preference to cat|. However, I substantially prefer cat | to < when necessary because it eliminates the distinction between a single / multiple file inputs.

Re:welll..

[tallredeye]

I prefer using grep with multiple file names because the output will show you which file the match was found in. You wouldn't get that by cat'ing the content of multiple files to grep via pipe.

Re:welll..

[toadlife]

The cmd environment in Windows is not *that* bad. It's a bit hokey, but it's sufficient for most any task. I recently converted a bunch of shell scripts from our HPUX box over to Windows cmd. It wasn't as bad as I thought it would be.

Re:welll..

[Eivind]

Sure. But sometimes people do it out of habit rather than because it makes sense. It aint a large sin, but it is somewhat silly to write cat file.txt | grep foo.

Re:welll..

[lahi]

Exactly. Try this:

able:~ $ mkdir foo
able:~ $ cd foo
able:~/foo $ csh
% touch a
% echo * >foobar
% cat foobar
a foobar

As you see, the * is expanded *after* the destination file is created. Now, depending on buffers file sizes, and possibly the phase of the Moon, this may not bite you, but if grep output from other files is flushed into foobar before foobar has been processed, the grep will reach foobar, start grepping it, and appending the result to foobar. Endless loop until disk full. I've seen it happen, it's for real, and NOT a pretty sight.

Sneaky, awful, and _very_ bad. As I said, I wonder why this hasn't made it into Tom Christiansen's list of bad things about csh.

All Bourne-like shells I know expand the wildcard first BEFORE creating the output file.

-Lasse

Re:welll..

[mrmeval]

Ie there is NOTHING bad about piping cats PETA would disagree. Oh? Just imagine the cacophony of a bunch of cats playing bagpipes while "singing" along. And all this while marching under fire, yes those Scottish cats are insane.

Re:welll..

[soccerisgod]

If you're working with gigabytes of data here, that can quickly be a frustrating exercise! Folks who are in the mentality of using cut | grep and even a visual editor like vi instead of sed are up the creek when they find themselves needing to manipulate and get portions of very large data sets.

I've just tried that on a 9 GB file with random binary data using a search string that would never match, on a PC with only 512 mb of RAM:

# time cat mydatafile |grep -b "kgfklgjfsdklgjlsdkfjgklsdfjgkldjgkdjfgkldjfkgljdk flgjdkfljgkldfjglkdfjkldjfkgljrituiroemvnmb"

real 2m24.265s
user 0m4.076s
sys 0m34.906s

# time grep -b "kgfklgjfsdklgjlsdkfjgklsdfjgkldjgkdjfgkldjfkgljdk flgjdkfljgkldfjglkdfjkldjfkgljrituiroemvnmb" mydatafile

real 2m17.366s
user 0m8.313s
sys 0m21.533s

The difference is astonishing! It saved a whole 7 seconds! Never again shall I make this mistake, remembering all the time I had wasted by wrong usage of unix commands! Thank you!

Re:welll..

[Principal Skinner]

ksh doesn't have spelling correction. Am I allowed to call that a bug? That's something I benefit from several times a day, whereas bugs like what you mention I rarely encounter (I would instinctively redirect the output to a different directory, I mean to me what he did is just asking for trouble).

tcsh does blow goats in lots of ways: for example, it doesn't have functions - the equivalent is aliases, and you can't do certain things like if/then/else statements in those, AFAICT. That's the one that really hurts sometimes. But as a command shell, if I can't get zsh, I find tcsh pretty nice. Also see what someone else said about how the shell interprets * when there are no matches; I find sh's default behavior for that pretty brain-dead, and have more than once found files named * in a directory as a result of this unexpected behavior.

I agree with another previous poster who said that old csh should be consigned to the trash heap of history, but in light of the wide availability and continued enhancement of tcsh, the "Csh considered harmful" thing really needs to be updated.

Re:welll..

[tehcyder]

Oh? Just imagine the cacophony of a bunch of cats playing bagpipes while "singing" along.

No need to imagine, just buy the greatest hits CD of Girls Aloud.

Re:welll..

[tehcyder]

but the piping cat thing is something that basically only annoys the most anal of anal

i.e. anyone interested enough to read this thread in the first place. Oh...

Re:welll..

[Anonymous Coward]

What makes unix so powerful is its modularity, the fact that you can pipe any output from any application to any applications stdin. This makes it possible to use common tools app1 | app2, app1longoutput | grep thingsIwant. The possibility to mix and match common elements that (arguably) makes unix powerful. Indeed. But why do I need to program in AWK if I want to extract some information from more complex output? I shouldn't have to restore structure to a flattened byte stream.

Windows Powershell seems to solve this by using a stream of .NET objects instead of a byte stream. Want to see the PID's of all processes matching certain criteria? Produce a stream of Process objects with the powershell equivalent of the ps command, and filter on the right properties.

I would like to experiment more with this kind of shell, but unfortunately there doesn't seem to be any free-as-in-speech equivalent. Maybe we can build something equivalent with mono or java.

Re:welll..

[virtual_mps]

Speaking as a novice, cat|grep is easier to understand. Until I skimmed the fancy article, I was unaware that grep could take a filename like that. I thought that to do filenames you needed fgrep. To an experienced unix guru, it may not be any clearer, but to a novice there's a world of difference.

So by "easier to understand" you meant "confuses you as to the proper way to use the command"? fgrep has an 'f' because it works with fixed strings rather than regexes, not because it has something to do with files. It's also deprecated in favor of "grep -F" in current POSIX. Also "cat file |" isn't inherently easier to understand, it's simply more familiar because all too many examples on the 'net do something godawful ugly like that. I'd argue that "<file command", which has the same visual cue (file first), is less to type, and more efficient, is superior to "cat file |" in every case. (cat still wins for "cat file file |", which is what it was intended for in the first place--concatenation.)

Don't use shell

[ee96090]

When possible, avoid shell scripting: the language is flawed by design. Use a modern programming language, such as Python.

Re:Don't use shell

[Anonymous Coward]

or even better -- use perl.

Re:Don't use shell

[tgv]

Someone mod up parent and grandparent, PLEASE!

Anyone?

Hello...???

Re:Don't use shell

[Anonymous Coward]

OK, I agree. Please provide a concise Python script that unpacks a tarball (a .tar.gz or .tar.bz2 file), copies new files in to a said tarball, patches based on the contents of said new files, runs make from various directories in said extracted tarball, and then changes the name of the top-level directory created by the tarball to a new name and repacks the tarball.

Or a concise Python script that opens up a text file of URLs, and extracts the files listed in the URLs:

#!/bin/sh
for a in $( cat file | awk '{print "'\''" $0 "'\''"}' ) ; do
              wget $a
done


Python has it place, and is far better for medium to large projects, and projects where the code needs to be maintainable. Shell, however, works a lot better for automating UNIX tasks than Python does. Not to mention embedded systems: I can compile Busybox to have both a good shell and all of the commands that one would run from shell scripts (including grep, cut, sed, and, yes, awk) in only about 300k. A Python binary is about a megabyte big, and you need about ten megabytes to fit all of the libraries Python 2.4 comes with.

Re:Don't use shell

[Anonymous Coward]

Someone mod up parent and grandparent, PLEASE!

No, don't mod up anybody in this thread. Perl and Python are abominations. Pure, unadulterated Bourne shell is for the true, seasoned *nix user. Just like Java is an answer to a question nobody asked in the GUI world, so too is Perl and Python in the command line world.

Re:Don't use shell

[Bazman]

If I've got a simple task to do (eg the text-file-of-URLS example) then I knock it up in shell script. By the time that simple task has feature-creeped up to more than 20 lines I start to wish I'd written it in Perl. So I rewrite. By the time that Perl script has crept up to more than 200 lines I start to wish it was written in Python. So I rewrite. By the time that Python script has crept up to 2000 lines I start to wish I'd farmed the job out to a team of programmers, and I give up caring what language its written in and make them do it as a web service. Then I write a small shell script to call their web service. When that shell script has feature-creeped up to more than 20 lines...

Re:Don't use shell

[Anonymous Coward]

This code is not pure shellscript : it uses awk and wget to get the job done...

A Python equivalent might be :

#!/usr/bin/python
import os
for a in file('filename').readlines():
  os.system('wget ' + a)

It's not that much longer, it's much easier to read and less error-prone (especially the awk part), and it uses fewer external utilities.

To me, the *only* advantage of shellscript is that it's the only language that you are sure to find on any Unix system.

Re:Don't use shell

[lazysonofab]

The answer to your second point is easy, and arguably as simple as your example:

#! /usr/bin/env python
import urllib
for a in file('listofurls.txt').readlines(): urllib.urlretrieve(a,a[a.rfind('/')+1:])

Saves the list of urls to the current directory with the existing remote filename. The tar stuff, I don't know about - I'm no programmer, I'm a lazy package manager user.

Re:Don't use shell

[Anonymous Coward]

Once I see Perl or Python in SuS or POSIX, you might have a point. Until then, Bourne Shell, awk and sed are the only languages you can use portably.

Re:Don't use shell

[dallaylaen]

That's how perl was invented by Larry Wall.

However, a "modern scripting language" is an overkill if you just need 2 lines. I would consider a "real" language only if I can't fit into 7-10 lines of sh.

Re:Don't use shell

[Anonymous Coward]

Both the URL-fetching example and the make/tar example are things that I'd do with Python. You are not wrong to use shell to do these things, indeed as Perl people always say, there's more than one way to do it, but there may be better tools for the job. Before I learned Python I'd have done these things in shell or Makefiles, but now I'd use Python.

I'd use Python because I can count on it to handle things reliably and sensibly. It's easy to write a shell script badly and get odd behaviour when you hit filenames with spaces in them, or ones that start with -. Easy, also, to ignore a return code and continue processing after something has failed, or to mistype a variable name and get weird behaviour because of that. You'd get exceptions in Python if you made this type of mistake.

Python is also faster than shell because fork/exec is not required for each operation. You can also scale your Python script into a full Python library providing a service to other Python programs - that's really easy to do, and very handy. Finally I think Python scripts are much easier to read and maintain than shell or Perl, but that is subjective!

Re:Don't use shell

[Haeleth]

This code is not pure shellscript : it uses awk and wget to get the job done...

That's what a pure shellscript is! The whole point of shell scripting is that you use the shell script as glue to tie together simple single-purpose utilities that come as standard with every flavour of Unix ever.*

To me, the *only* advantage of shellscript is that it's the only language that you are sure to find on any Unix system.

No shit, Sherlock! You have clearly never worked in a large organisation, where - believe it or not - you, as a standard user, do not actually get to insist that the already-overworked IT department jump through bureaucratic hoops to install your favourite bloated scripting language, unless you have a damn good business case for it. And probably not even then.

Hint: if the task you want that scripting language to accomplish is trivial to achieve with a simple shell script, you don't have a good business case.

* This doesn't apply to wget, obviously, but if your platform really has no standard alternative, you are more likely to persuade IT to install something small and simple like wget, fetch, curl, etc. than a complete programming environment like Python.

Re:Don't use shell

[duguk]

Um, whats wrong with

wget -i filename

Or have I missed something?

Monkeyboi

Re:Don't use shell

[Anonymous Coward]

It's easy to write a shell script badly and get odd behaviour when you hit filenames with spaces in them, or ones that start with -

Agreed; that is why I had that Awk script. I work around this by having a small perl script
which I call quotes:

#!/usr/bin/perl -p

chop;
s/\'/\'\\\'\'/g;
$_ = "\'" . $_ . "\'\n";


This can also be implemented in about 10 lines of C code.

Easy, also, to ignore a return code and continue processing after something has failed

This can be worked around using the undocumented -e flag. What "-e" does is make it so the script stops running if a program called in the script (except in if and a couple of other places) fails (returns a non-zero exit code).

I think it's really bad that this flag hasn't been documented; it's one of those "inside secrets" that UNIX is nortorious for having. And, yes, Python has really great documentation.

I also appreciate the Python examples for getting those URLs. It looks like Python has a handy way of running UNIX commands. I am sure I could run the unpack-patch-update-repack script in Python, but it would have a lot of os.system() calls. Or I could do something like x=os.system and make it x() calls. In addition, Python has Expect-like functionality built in.

As for another comment pointing out that wget is not a standard Unix command: Wget is pretty universal across the Linux flavors, and is also easily added to the *BSDs. A wget is also part of Busybox for people building embedded systems. And, oh, I checked, and that Busybox is closer to 400k in size than 300k in size. The Python binary is 700k in size, but the libraries take up some 16 megs.

Re:Don't use shell

[martin-boundary]

Nonsense. The bourne shell family are languages with particular strengths, some of which simply don't have a counterpart in perl or python.

Try writing a moderately complex pipeline in perl using native popen() calls. Not only do you have to deal with lots of file descriptors explicitly, but you'll have an unmaintainable mess if you have frequently changing requirements, such as adding or removing extra filters at various places.

Always use the right tool for the job. If you want to do complex logic, by all means use python, but if you want to combine lots of existing functionality from separate programs, then don't use a scripting languague like perl or python unless you have a damn good reason.

If you don't like the cruft in the bourne shell family, take a look at rc, which is a pretty nice language for its domain.

Re:Don't use shell

[dheera]

Yuck, I never use bash scripts. I always use Perl scripts. I just do things like

#!/usr/bin/perl
system("blah");
system("blah");
if(perl code perl code) {
    system("blah");
}
etc.

why?
1. because i can't remember the awful syntax of the bash if statement. isn't it something like
    if[[""$X$$"" == ""$Y""]];; ... fi ?
2. how about accepting command line arguments in bash? in perl it's just $ARGV[0]. nice and simple and like C++ (except for the offset by one) so i don't want to have to bother learning another one.
3. because i can't bother learning how to do a regular expression in bash. in perl it's simple with =~/.../ and =~s/.../.../ and it was bad enough that PHP isn't like that.
4. because bash seems to think that sometimes you use x and sometimes you use $x
    x="hi"
    echo $x

i really don't want to learn this language. so i just use Perl everytime i need a script. it works.

Re:Don't use shell

[Anonymous Coward]

How the fuck did you get (5, informative)? Who mods this shit?

Re:Don't use shell

[wertarbyte]

4. because bash seems to think that sometimes you use x and sometimes you use $x

What's so special about that? If you want the variable, use "x", if you want the value of it, use $x. $ tells the shell to evaluate the next token as an variable name and fetch its value.

Re:Don't use shell

[Fred_A]

Um, whats wrong with

wget -i filename

It's not a shellscript, that's what wrong with it. Please pay attention next time.

Re:Don't use shell

[gmack]

2. how about accepting command line arguments in bash? in perl it's just $ARGV[0]. nice and simple and like C++ (except for the offset by one) so i don't want to have to bother learning another one.

Command line args? $1 $2 etc or $* for all of them.

Re:Don't use shell

[swillden]

This can be worked around using the undocumented -e flag. What "-e" does is make it so the script stops running if a program called in the script (except in if and a couple of other places) fails (returns a non-zero exit code).

I think it's really bad that this flag hasn't been documented; it's one of those "inside secrets" that UNIX is nortorious for having. And, yes, Python has really great documentation.

From 'man bash':

-e Exit immediately if a simple command (see SHELL GRAMMAR above) exits with a non-zero status. The shell does not exit if the command that fails is part of the command list immediately following a while or until keyword, part of the test in an if statement, part of a &amp;&amp; or || list, or if the command's return value is being inverted via !. A trap on ERR, if set, is executed before the shell exits.

Seems reasonably well documented to me.

Re:Don't use shell

[sid77]

please, try telling this to P.J. Volkerding and then come back here to tell us how it went.

Re:Don't use shell

[dheera]

yeah, but... if there would possibly be more than 10 arguments, say
    $ ./myscript a b c d e f g h i j k l m
and I wanted to print "a0", i would think of doing
    print "$10"
oops. how does it know the difference between {$1}0 and {$10} ?
when I design my variable names, I design them in a way so that such confusion is nonexistent.
i also really prefer having my arguments in an array for programming purposes... it also conflicts with the use of $1 in the second clause of a s/(.*) blah/$1/ type clause.

i know you can keep giving me solutions, but basically i find bash scripting to have too non-standard of a syntax. perl for example at least uses, for the most part, C syntax, and anything it invents is over and above that (=~, etc) and not conflicting with it. to me it seems like syntax concept X in every other language does Y in bash and Y in other languages does X in bash.

really, now, for(blah blah blah); mycmd; shouldn't do anything. for(...) mycmd; should. for(...); mycmd; rof; should. why is it for[...]; do mycmd;done; ? it's just way too strange of a syntax for my tastes. i'm not complaining that it has faults - it's just that the nitpicky differences in syntax make it unattractive and steep to learn for someone knowing C, C++, Perl, PHP, basic, etc.

Re:Don't use shell

[gmack]

It's all about using the right tool for the job. I tend not to use bash for anything complicated enough to require that many command line args. Come to think of it none of my C code ever requires 10 args either.

Re:Don't use shell

[JohnFluxx]

Wow, do you think you could be just a little bit more polite next time?

Re:Don't use shell

[archen]

one loophole, which I've seen done and done on occasion myself; is doing something like

$x=`blah`

Now I'm sure we'd all say together, we'd catch that after testing the script. I'd agree, and I'm not sure what the article was trying to get at when talking about quoting variables - you'd see it's not working correctly if you bothered testing it and saw instances where $x was always blank.

The shell is good for nice quick, or at least very simple things, but it is often abused and extensions can leave you with unexpected results at the very least. Not to mention the linux-ism that expects that sh is ALWAYS bash.

Re:Don't use shell

[canuck57]

When possible, avoid shell scripting: the language is flawed by design. Use a modern programming language, such as Python.

How did you get mod'ed up to 5 I don't know.

Python may not be on the system your managing.

Python, while not as heavy as Java, is an elephant in a china shop compared to shell script

Shell scripting is an essential basic admin skill, good admins don't need Python at all.

Java, Python, Perl -- sheshh... they have their places but not for everything. If any one tool could effectively do it all, there would be no others.

Re:Don't use shell

[multipartmixed]

Right on, brother.

And, for what it's worth, I once had to write an HTTP fetch utility using nothing but SYSVR4 bourne shell, and stuff that shipped stock with a bare-naked install of Solaris 2.5.1.

Why? I needed stuff. NOW. I couldn't wait for next week for an 'admin' to install a web browser, I would have missed my go-live date.

Oh, yes, the box also had an old version of pine on it. I remember, because after HTTP fetching whatever it was, I had to de-base64-it.

I also once wrote an LR(1) parser for arithmetic expressions using nothing but native constructs in ksh (and not ((...))!). But that was just because I was bored. IIRC it was stack based, using a stack "class" implemented on top of ksh arrays and a shitload of evals.

Re:Don't use shell

[25albert]

Use a modern programming language, such as Python

I prefer postmodern languages such as Perl

Re:Don't use shell

[CableModemSniper]

shift. Where do you think perl borrowed it from?

Re:Don't use shell

[Anonymous Coward]

Thanks for the note about -e, I wasn't aware of that. Although I am a shameless Python fanboy (until something better comes along), I do still use shell scripts for small tasks and now I know a new feature. I still think the spaces thing is a pain, but you normally only have to deal with that when Windows users have been creating files, and as you say there are some ways to manage it.

Re:Don't use shell

[osee]

wouldn't "wget -i textfile_of_urls.txt" have done this much simpler?

Re:Don't use shell

[pthisis]

readlines isn't needed(and even in old versions, xreadlines is probably better); also his version quoted the url so you can use urls with shell-unfriendly characters:

import os
 
for url in open("file"):
 
  os.system("wget '%s'"%url.rstrip('\r\n'))

If you don't have wget or don't want to fork a bunch of processes:

import urllib
 
for i, url in enumerate(open("file")):
 
    open("index.html.%d"%(i,), "w").write(urllib.urlopen(url).read())

which is more complex than the shell version because you have to specify the incoming filenames.

For something this simple I'd use shell, though.

Re:Don't use shell

[dgatwood]

This code is not pure shellscript : it uses awk and wget to get the job done...

That's what a pure shellscript is! The whole point of shell scripting is that you use the shell script as glue to tie together simple single-purpose utilities that come as standard with every flavour of Unix ever.*

Well, since awk is a programming language (and a fairly complex one at that), one could legitimately argue that using awk in a shell script is as impure as using perl in a shell script. I've done both, but it does sort of taint things. Shell scripts that use wget are still pure, IMHO.

Re:Don't use shell

[wertarbyte]

Not to mention the linux-ism that expects that sh is ALWAYS bash.

/bin/sh should always be /bin/bash. Everytime I have to use a commercial unix system, logging in feels like entering a time portal: No history, no path in prompt, nothing comfortable.

Re:Don't use shell

[Millenniumman]

Now it is:

#!/usr/bin/env sh
wget -i filename

Re:Don't use shell

[Brandybuck]

When possible, avoid shell scripting

Unless you're on the shell. Then use a shell scripting language.

Or if you don't know what version of your favorite "modern programming language" the user will have installed on their system. Or if the user is a Bad Person(tm) who doesn't even have your favorite "modern programming language" installed. Or even worse, using a system or distro that doesn't install your favorite "modern programming language" by default.

Or if you just want to do something simple. Sheesh.

Re:Don't use shell

[Anonymous Coward]

I tend to agree. What is it with reversing tokens names to end a statement block, like "fi" or "esac"? Someone must have thought they were being funny or clever by inventing that, but it's just childish and serves no benefit over "endif" or even just a simple closing brace. And it's not even consistent, they didn't use "rof" or "elihw".

Re:Don't use shell

[myowntrueself]

such as Python

Don't make me laugh, I am eating my breakfast.

Among other things, a language which can have *syntax* errors on a file with mixed tabs and spaces would be worthy of an apt-get --purge remove if it wasn't required by xen...

Re:Don't use shell

[BarkLouder]

or even better -- use perl.

or far, far better use Ruby

Re:Don't use shell

[JoshJ]

Oddly enough, Ubuntu Edgy Eft (and I'd guess Debian Etch as well) use dash instead of bash.
The Solaris servers I can ssh into at my school (Georgia Tech) use ksh.

Re:Don't use shell

[lahi]

The guy who I think invented that notation for Algol68, Aad van Wijngaarden, should be counted among the cleverest in the history of Computer Science. The reason there is no rof is because in Algol68, the loop construct combines for and while and as both can be omitted, all that remains is do...od which is why using od is the consistent choice. Bourne probably replaced od by done because od had already been taken for the octal dump command.

And don't get me started on two-level vW-grammars!

-Lasse

Re:Don't use shell

[Anonymous Coward]

guess what program is invoked to run "os.system", hint, it's /bin/sh. It also looks like it'll pass any shell characters to the shell unquoted, which is easy to stop in a shell script by putting "" around the parameters.

Re:Don't use shell

[totally bogus dude]

It's easy to write a shell script badly and get odd behaviour when you hit filenames with spaces in them, or ones that start with -

Agreed; that is why I had that Awk script.

Your awk method doesn't actually work, because it embeds the single-quote in the argument to wget.

The correct solution would've been simply:

#!/bin/sh
for a in $( cat urls ); do
wget "$a";
done

Re:Don't use shell

[grumbel]

### Or a concise Python script that opens up a text file of URLs, and extracts the files listed in the URLs:

Beside the obvious 'wget -i' (most tools have a build in 'read stuff from file' option), I would use Ruby for such things, at least when they get a little bit more complex and sanity is at risk when the whole shell-escaping drives you mad:

File.new(file).readlines.each{|url| system("wget", url.chop) }

or if one wants to go with a single wget process:

system("wget", *File.new(file).readlines.map{|url| url.chop})

Re:Don't use shell

[Anonymous Coward]

You have clearly never worked in a large organisation, where - believe it or not - you, as a standard user, do not actually get to insist that the already-overworked IT department jump through bureaucratic hoops to install your favourite bloated scripting language, unless you have a damn good business case for it. And probably not even then.

Or a large organization where some vendor has a perverse fondness for some crappy ancient version of perl that breaks anything that counts on perl other than their POS application. Or where you have dozens of administrators, some of whom are incompetent, so you can't count on their box being set up right or them fixing things so you have to find a perl compiler and distribute things as some nasty huge executable instead of as efficient scripts.

Anyway, there are lots of advantages to shell scripts. Guess what? I can do quick-n-easy stuff in shell scripts without knowing perl or python or C or anything else. All y'all programmer types see a nail and want to get out your 500 horsepower Porter Cable pro series computing language. Meanwhile I'm already on to the next task (or the one after that). And I don't WANT to learn to program, I'm not a frustrated programmer 'slumming' as a sysadmin, I'm a sysadmin because I'd rather read poetry (ugh) than source code.

Oh, you want on topic?

Here's a Unix tip: Don't let ANYONE have root who doesn't need it every day. No, application developers do not need root. Ever.

Here's another Unix tip: Check your damn box(es) all the time so you have a baseline. Better to formalize it, but at least have a good, clear idea of what is normal.

Here's another tip, not just for Unix: If a nice (wo)man from CA tries to sell you something, tell them firmly that they need to leave the building by the nearest closed window.
--
What spoon?

Re:Don't use shell

[netgecko78]

Well, if you are to fetch a list of urls with more than 255 characters in the filename, you can't use wget -i long-urls.txt.
You have to create a script and use a -O option in it.

import os
b = 1
for a in file('long-urls.txt').readlines():
    os.system('wget -O' + b + ' ' + a)
    b += 1

Re:Don't use shell

[wertarbyte]

Oddly enough, Ubuntu Edgy Eft (and I'd guess Debian Etch as well) use dash instead of bash.

For the bootfloppy, yes:

stefan@nano:~$ apt-cache show dash
Package: dash
Priority: optional
Section: shells
[...]
Description: The Debian Almquist Shell
"dash" is a POSIX compliant shell that is much smaller than "bash".
We take advantage of that by making it the shell on the installation
root floppy, where space is at a premium.
.
It can be usefully installed as /bin/sh (because it executes scripts
somewhat faster than "bash"), or as the default shell either of root
or of a second user with a userid of 0 (because it depends on fewer
libraries, and is therefore less likely to be affected by an upgrade
problem or a disk failure). It is also useful for checking that a
script uses only POSIX syntax.
.
"bash" is a better shell for most users, since it has some nice
features absent from "dash", and is a required part of the system.

Re:Don't use shell

[VENONA]

Article was about using the command line, not scripting. Calling an interpreter, and supply more complex statements, etc., may be something you want to do, but not me. For instance, sorting your environment in Perl:

foreach my $key (sort keys %ENV) {
      print "$key => $ENV{$key}\n";
}

You'll have to call Perl alone, then issue the statements line by line, or convert this into a one-liner. Which I leave as an exercise for the reader, as I wouldn't do that because in a shell it's just:

env | sort

Plus, portability can suck, if you need to call libraries which may or not be present on a given system. And by then, your command may be getting obnoxiously long, anyway. Granted that there are some powerful Perl and Python one-liners, but I just don't see that as a reason to write 'Don't use shell'. Use what's appropriate. Often, that's shell. That's why it's *there*.

Of course, you have to put in the effort to *learn* what's appropriate, which is a gradual process. Some reasons for putting out the effort:

a) You'll want to learn the basics anyway, just because of the power and convenience you gain in customizing your environment via tweaking resource files.
b) It teaches you fundamental things about Unix-y operating systems.
c) Trying doing any sort of serious system administration without being able to understand (and modify, if necessary) init scripts.
d) Easy script creation via 'script' command.
e) It's fun.

Re:Don't use shell

[tgv]

I made the most horrible typo in my message: I wrote "up" instead of "funny". And I thought that all I would be speaking to the deaf, since this site seems to be awash with Perl and Python lovers.

Square or Curly brackets?

[Beolach]

enclose the variable name in square brackets ([])

~ $ ls tmp/
a b
~ $ VAR="tmp/*"
~ $ echo $VARa

~ $ echo "$VARa"

~ $ echo "${VAR}a"
tmp/*a
~ $ echo ${VAR}a
tmp/a

Their example correctly uses Curly brackets, {}, but their text says square brackets []. That seems like a typo to me.

Re:Square or Curly brackets?

[tigga]

Their example correctly uses Curly brackets, {}, but their text says square brackets []. That seems like a typo to me.

I think somebody needs glasses ;)

Re:Square or Curly brackets?

[lmfr]

The correct form is {}, not []. There are other things you can use with ${VAR}:

• ${VAR:-text if $VAR is empty}
• ${VAR:=text if $VAR is empty and set VAR to this}
• ${VAR:+text if $VAR is set}
• ${#VAR} -> length of $VAR
• ${VAR#pattern} or ${VAR##pattern} -> remove match of pattern from beginning of $VAR (## -> longest match)
• ${VAR%pattern} or ${VAR%%pattern} -> remove match of pattern from end of $VAR (%% -> longest match)

There are other formats (see the man page), but these are the ones I use the most. Eg:

for i in *.png; do convert "$i" "${i%.*}.jpg"; done

Re:Square or Curly brackets?

[thc69]

I think somebody needs glasses ;)

Don't you mean:
I think somebody needs glasses ;}

Anyway...Curly brackets? Nyuk nyuk nyuk!

Re:Square or Curly brackets?

[Anonymous Coward]

${VAR:-text if $VAR is empty}

Or if $VAR is unset.

${VAR:=text if $VAR is empty and set VAR to this}

Or if $VAR is unset.

${VAR:+text if $VAR is set}

And $VAR is not empty.

Empty and unset are not the same thing.

unset var
test "${var+set}" = "set" &&
    echo "var is set to \"${var}\"" ||
    echo "var is unset"
var=
test "${var+set}" = "set" &&
    echo "var is set to \"${var}\"" ||
    echo "var is unset"
var=a
test "${var+set}" = "set" &&
    echo "var is set to \"${var}\"" ||
    echo "var is unset"
set -u
var=
echo "var=\"${var}\""
unset var
echo "var=\"${var}\""

Re:Square or Curly brackets?

[lmfr]

Empty and unset are not the same thing.

They are, for the uses I mentioned: :-, := and :+.

But not for the case you demonstrate: ${VAR+text} (or ${VAR-text}, etc.)

I'm not familiar with ${VAR[+-=]text} instead of ${VAR:[+-=]text}, I've been using bash from when I started using Unix. Both bash and ksh uses the form with '=', is the one without it from the original bourne shell?

Re:Square or Curly brackets?

[cas2000]

for i in *.png; do convert "$i" "${i%.*}.jpg"; done

that's neat. i've always used sed or basename for this, but your way is better. i knew bash could do useful stuff with ${...} like having default values for vars, but for some reason it never really sunk in that it could do primitive sed-like stuff.

e.g. my old way of doing it (to be replaced by the ${VAR%pattern} way from now on):

for i in *.png ; do convert "$i" "$(basename $i .png).jpg" ; done

Re:Square or Curly brackets?

[Anonymous Coward]

They are, for the uses I mentioned: :-, := and :+.

They behave the same way with :[-=+], but they aren't the same thing. You wouldn't say the only solution to x^2=4 is x=2, just because negative two behaves the same way when squared as positive two, would you? :)

But not for the case you demonstrate: ${VAR+text} (or ${VAR-text}, etc.) I'm not familiar with ${VAR[+-=]text} instead of ${VAR:[+-=]text}, I've been using bash from when I started using Unix. Both bash and ksh uses the form with '=', is the one without it from the original bourne shell?

Do you mean the form with ':'? Anyway, I'm not sure which form was originally used, or even whether only a single form was originally used, sorry. Both forms are standardised now, though, and generally, I find it a good idea to pay attention to the distinction.

Re:Square or Curly brackets?

[lmfr]

I understand what you mean now. You're right, of course. As for my first comment, the "is empty" and "is set" means that it's undefined or defined to an empty/null value in the first case, and defined to a value different than null in the second case.

Re:Square or Curly brackets?

[noz]

Very true, but it is important to note that these are extensions first provided by ksh and adopted by bash and zsh (IIRC). The article should have discussed different interpreters.

He's being pedantic

[Anonymous Coward]

He's being pedantic at times with no good reason. I do cat file | grep/awk/sed/perl -pe/whatever quite frequently because it makes visualizing the pipeline easier, and handles files bigger than 2gb better than the 32-bit version of awk/sed/whatever. The only really useful piece of advice is mkdir -p...something that doesn't work with legacy UNIX systems.

Typo

[seebs]

The quoted paragraph from the article is incorrect -- and it is in the article too -- but the example immediately following it correctly shows the use of braces ("curly brackets"), not square brackets, for variable names in shell.

cat file | grep something

[reub2000]

I will never stop piping cat! If I want to do so, I will!

Re:cat file | grep something

[netbuzz]

Yup, that's the phrase that jumps out at those of us who don't speak UNIX. Fun with piping cats on this blog post:
http://www.networkworld.com/community/?q=node/9764

Re:cat file | grep something

[AndroidCat]

Articles on terminating zombie children are always a treat too.

Re:cat file | grep something

[TheRaven64]

I will never stop piping cat! There are words for people like you

Re:cat file | grep something

[Anonymous Coward]

Will you terminate my zombie? I know we just met...

mkdir

[pfafrich]

His example of good habit with mkdir did not convince me

$ cd tmp/a/b/c || mkdir -p tmp/a/b/c

If the directory exists you end up in the directory, if it does not it creates the directory but leaves you where you first started. Hence you don't know which directory you will be in after the command is executed!

Re:mkdir

[Anonymous Coward]

If the directory exists you end up in the directory, if it does not it creates the directory but leaves you where you first started. Hence you don't know which directory you will be in after the command is executed!

Try

$ cd tmp/a/b/c || mkdir -p tmp/a/b/c && cd !#:1

With bash you can refer to words in the current command line with

!#:N

where N is the word position (starting from 0).

Re:mkdir

[plalonde2]

Save a few keystrokes and typos:

( A=tmp/a/b/c cd $A || mkdir -p $A && cd $A )
But this is silly: mkdir -p succeeds even if the directory already exists, so this suffices:
mkdir -p a/b/c && cd a/b/c
I also resent the use of zillions of little command options to "be more efficient". The -C to tar is just plain silly:
( cd a/b/c ; tar xf $ELSEWHERE/foo.tar )
does the same thing, withouth having to look up the fricking -C each time you want to use it. Three guesses which technique has the true unix nature.

Re:mkdir

[SharpFang]

Especially the habit of using || and && on command line seems ridiculous to me. These have room in two situations:
- scripts
- commands that take long enough that you go have a coffee.

This makes sense:

make install && lilo && reboot

This doesn't:

cd tmp/a/b/c || mkdir -p tmp/a/b/c

If you fail the first part, well, you typed " || " instead of pressing enter.
If you succeed the first part, you typed " || mkdir -p tmp/a/b/c" without a bloody reason.

Type first part. Press enter. Observe result.
If necessary, type the second part, otherwise correct the first without baggage of the second one hanging around.

Re:mkdir

[Russ Nelson]

In that case, do this:
mkdir -p tmp/a/b/c; cd tmp/a/b/c

Re:mkdir

[lahi]

Not only is it silly, but the example given in Listing 4 is WRONG: tar xvf -C tmp/a/b/c newarc.tar.gz will not work, unless there is a version of tar which implies -z given -C. (Someone else already mentioned the misplacement of -C as the filename for -f.) And of course, relying on -z options for tar will not work portably (including versions of AIX, though I'm not sure about AIX 5). This is a case where zcat (given a zcat that knows gzip) should be used: zcat arc.tar.gz| tar xvf -

The latter also works nicely across machine boundaries: zcat arc.tar.gz| ssh user@foo 'cd tmp/a/b/c ; tar xvf -' (or, better: cat arc.tar.gz| ssh user@foo 'cd tmp/a/b/c ; zcat|tar xvf -' ) I doubt -C does that. Of course use && instead of ; if there is any doubt whether the directory exists. This way you don't even have to move the tar file if you want to uncompress it on a different machine. (I suppose some GNU guy will read this and add another option --unpack-at-host which does the same, given the already prolific amount of unneccessary options to GNUtar. (Next thing I know, someone will tell me that option already exists.))

Another error: the command in Listing 7 also fails if tmp/a/b/c exists, because it will be in tmp/a/b/c when the tar command is executed, and thus the -C command will try to cd to a most likely nonexisting relative path. Perhaps it would be a good idea using test -d instead of the context-destructive cd for the condition, or maybe use an absolute path for the -C option. So many errors and I have only read less than one third of the "good habits" yet, wow!

I will make a note of the author's name (Michael Stutz) as a person whose statements should not be trusted without thorough scrutiny.

-Lasse

Re:mkdir

[hackstraw]

First, let me be clear that this list is bogus. These 'Good' habits are not that advantagious at all, and I've been using UNIX for quite some time now, and I'm pretty good at it.

Now, being pedantic, the cd a/b/c || mkdir -p /a/b/c is silly, just do the mkdir -p /a/b/c if that is what you want. Don't test for it, just do it!

Re:mkdir

[SuurMyy]

Or, if you think that the dir exists:

cd tmp/a/b/c ...which never gets complete, because you use tab to fill in the dir names.

Noticing this, then type:

mkdir -p tmp/a/b/c

then use command-line history and cut/paste to get there (esp. if the dir is long):

up arrow, ctrl+w, ctrl+c, ctrl+y, ctrl+a

type

cd and one space, then press enter.

This might seem tedious, but when it's in your spine, it's good.

Re:mkdir

[RoutedToNull]

[ ! -d /tmp/a/b/c ] && mkdir -p /tmp/a/b/c

Re:mkdir

[martinwguy]

You are correct, the author is a beginner for three separate reasons

1) mkdir -p succeeds silently if the directory already exists, so the test is pointless

2) In any case, good practice is to do what you wanna do and take remedial action if it fails, not to see whether you can do it and then if you can. That is for the terminally underconfident :)

3) Even if that weren't the case, what's wrong with "test -d /tmp/a/b/c"

For good Unix practice you are better off reading a random page of Raymond's collected "Art of Unix Programming", on the net at http://catb.org/~esr/writings/taoup/

Re:mkdir

[canuck57]

His example of good habit with mkdir did not convince me

$ cd tmp/a/b/c || mkdir -p tmp/a/b/c

If the directory exists you end up in the directory, if it does not it creates the directory but leaves you where you first started. Hence you don't know which directory you will be in after the command is executed!

Actually, the example is a good habit.

If the change directory fails, the script will not make the directory in the wrong place. It is a good practice.

Re:mkdir

[Anonymous Coward]

His example listing 7 was worse, as I said in my reddit comment yesterday:

What I thought was nice is the repeated use of:

"cd tmp/a/b/c || mkdir -p a/b/c && tar xvf -C tmp/a/b/c archive.tar"

There are many things wrong with that command.

1) It is dumb to "automate" mkdir that way. as a human, you can easily see if the cd failed or succeeded. why would you want to type in "|| mkdir -p tmp/a/b/c &&" if you didn't have to?

2) The "mkdir -p tmp/a/b/c" already tests if the directory exists and won't do anything if it does. If you're going to bother typing in the "mkdir -p tmp/a/b/c" command, you may as well drop the "cd tmp/a/b/c" test in the first place.

3) The -C option to tar already creates the folders if they don't exist, so there's no reason to test and create it ahead of time.

4) If the directory actually existed, you'll be in the tmp/a/b/c directory already when you run tar, (thanks to the cd) which is obviously not what was intended, since the tmp/a/b/c is specified by -C, which means that the file will get unpacked to tmp/a/b/c/tmp/a/b/c

5) The "-C" option was specified between the -f option and its argument, so that tar will try to unpack the files named "tmp/a/b/c" and "archive.tar" from the archive named "-C"

Did I miss anything? :)

Edit:

I just noticed that "Listing 11" is even funnier, since it does all sort of random things that are clearly not what the author intended, and for no good reason. I can definitely say don't do what this guy does.

Re:mkdir

[kastberg]

What's wrong with

> mkdir -p tmp/a/b/c
and then just doing
> cd !$

shorter and even easier to remember

Re:mkdir

[Anonymous Coward]

Maybe the author is using BizzaroUNIX V0.1, where mkdir -p /a/b/c overwrites /a/b/c if it exists?

No? Maybe the author is just an idiot then. Ocams razor and all that.

Re:mkdir

[Anonymous Coward]

Wouldn't it be much simpler to just do this:

mkdir -p tmp/a/b/c; cd tmp/a/b/c

if the directory's already there, no harm done. If its not there, you managed to automatically create it before cd'ing into anyway.

Re:mkdir

[BentSorenDahl]

Which would be better off without the || at all.

mkdir -p tmp/a/b/c ; cd tmp/a/b/c

or with GNU mkdir:

cd `mkdir -p --verbose tmp/a/b/c`

Re:mkdir

[FrankieV]

Especially as it fails in their example:

Listing 7. A combined example of good habit #3: Combining commands with control operators

~ $ cd tmp/a/b/c || mkdir -p tmp/a/b/c && tar xvf -C tmp/a/b/c ~/archive.tar

Re:mkdir

[SuurMyy]

Thanks. :)

Re:mkdir

[Anonymous Coward]

Yuck. Try using test operator instead:

[[ -d tmp/a/b/c ]] || mkdir -p tmp/a/b/c

--AC

Re:mkdir

[EugeneK]

sweet, makes me want to try Bizarro 0.1! who do i need to pay for a license?

Re:mkdir

[kastberg]

no worries, I found it in a screencast, bash is so cryptic sometimes it's no idea trying to figure it all out :)

Re:mkdir

[3.1415926535]

Even if he did have wonky mkdir behavior, he'd be better of doing [ -d /a/b/c ] || mkdir -p /a/b/c anyway. The mkdir(1p) manpage says of the -p option,

Each dir operand that names an existing directory shall be ignored without error.

so testing for the directory first is a waste of time anyway.

Unix is more than just a shell

[petes_PoV]

This article is really just about good (in the authors opinion) TTY shell usage. There's more to Unix than just its shell
(plus he didn't mention my favourite shortcut: shell history)

How about being more inclusive and expanding this to deal with security features (surely the single biggest benefit?) and the ease of working on remote boxes?

Re:Unix is more than just a shell

[MaoTse]

I cannot agree.

To know shell is to know unix to much degree. Comments like 'you can similar things in python' are simply
ridiculous. I say this being python user myself.

Lots of people try to somehow escape or flee from what unix really is and the article commented is highly recommended to all of them. You cannot find anything useful in shell history unless there actually is something ;-)

Other valuable UNIX 'kung fu' resources are obviously books. Like the old 'Unix shell programming' by
Ted Burns or famous 'Unix power tools'. And of course - man pages. For example for bash ...

Re:Unix is more than just a shell

[SharpFang]

You cannot find anything useful in shell history unless there actually is something

Mod parent 'redundant'.

Shell code in linux based devices

[wertarbyte]

Quote variables with caution

Hey, FON, are you reading this? Sadly, even shell code in comsumer devices is often deeply flawed. Which is a good thing, if you want to install a new firmware - and a bad thing, if an attacker does it for you.

This article...

[nevali]

...is so littered with basic errors that it really shouldn't be recommended to anybody. How is 'tar xvf -C tmp/a/b/c newarc.tar.gz' expected to work, for example? Quote variables with square brackets? Running subshell commands using ; instead of && ? No mention of 'xargs -0' ? Don't pipe from cat to grep? Does anybody actually care that people do this (primarily so that the syntax is consistent between a munged- and unmunged-grep, and also such that the order of the command-line is logical from a human point of view)? Plus, of course, it's possible that cat | grep could yield better performance than grep alone: if cat uses mmap() to efficiently read the input files, and the kernel's pipe implementation is good, then it could do better than a grep implementation alone that simply read()s the files.

Re:This article...

[delire]

Agreed, the article is incorrect regarding use of tar's -C switch.

This works: tar -C /tmp/foo -xvf /path/to/some.tar.gz

This doesn't: tar xvf -C /tmp/foo /path/to/some.tar.gz

I agree that '&&' is a better practice to encourage than ';'. Chaining commands without heeding their exit status can make a real mess and cause much confusion.

Aside from that I don't think it's a bad top-ten. I think there should've been something in there about command completion. If productivity is important to this discussion then this feature of any good shell environment can't be ignored.

Re:This article...

[shird]

I think whats really missing in the tar command is the '-z' due to it being a .gz

Re:This article...

[Anonymous Coward]

Running subshell commands using ; instead of && ?

Ah, you must be a teacher, seeing something that remotely resembles a common error, and immediately assuming that it was an error. The article explains both ; and &&...

Now why don't you go read the entire article and take back some of your criticism. I think it was a fairly nice article. (and sure, there were some errors, but I strongly prefer this over some typo-less corporate soap-opera)

Re:This article...

[dallaylaen]

Agreed. 10 useful tips and tricks? Yes. 10 good habits? No. Maybe two of them. A 'habit' is something more general than just smartly using a command.

Perhaps, all slashdotters should post about *their* good Unix habits here, and than we make another article of 10 best ones :)

Re:This article...

[Anonymous Coward]

I think whats really missing in the tar command is the '-z' due to it being a .gz

Recent GNU tar versions no longer require the -z or -j options anymore. tar -xf now guesses the proper filter.

Re:This article...

[jaakko]

> if cat uses mmap() to efficiently read the input files, and the kernel's pipe implementation is good, then it could do better than a grep implementation alone that simply read()s the files.

B.S. grep will still read(2) the stdin.

Re:This article...

[delire]

It works anyway, try it. -z is redundant in this case.

Re:This article...

[nevali]

Re-read what I said. If cat uses mmap() and the kernel's pipe implementation is good [i.e., such that grep's read() is negligible], then it could do better than a grep implementation alone that simply read()s the files.

Re:This article...

[Haeleth]

I think whats really missing in the tar command is the '-z' due to it being a .gz

This article is about Unix, where there is no such option (see the specification).

You may be thinking of GNU tar. I suggest you look up what the acronym GNU stands for, with particular reference to the N.

(This is not to dispute that the example given in the article is broken.)

Re:This article...

[lahi]

Does GNU tar have an embedded EMACS as well? (or will GNU tar users have to wait for the next version?)

Well, at least Michael Stutz compensated for the lacking -z in tar by adding a superfluous -1 option to ls at the beginning of a pipe. I think most ls implementations check whether stdout goes to a terminal to decide whether output should be columnized.

-Lasse

Re:This article...

[Anonymous Coward]

It still makes no sense. If mmap followed by a read from a pipe is faster than a straight read from a file, then why wouldn't the kernel's implementation of a straight read from a file be done using an mmap and a pipe?

Re:This article...

[treat]

You're the only one who hasn't mentioned xargs -0. I think it's important to elaborate on this. You should never do "find | xargs" or "find | cpio", you should always do "find -print0 | xargs -0" and find -print0 | cpio -0". The former will break if filenames have spaces or newlines in them. You break xargs if filenames have quotes, backslashes, or spaces in them. I never come across a large data set where you can do find | xargs without the -0 options.

If you are encountering data created by untrusted users, don't forget the strange consequences of filenames that contain newlines.

Failing to use -0 is dangerous malpractice.

Re:This article...

[nuzak]

Go take a gander at open(2) on a modern unix. It is mmap.

Re:This article...

[nevali]

In which case, the difference is going to be negligible, especially on a decent system (where launching an additional process has very little overhead).

Re:This article...

[nuzak]

> You're the only one who hasn't mentioned xargs -0.

Oh my goodness yes, but I think shells should go one further and have special syntax for passing around records, zero terminated or not. Then you could pass around not only filenames, but arbitrary anything, like images, without each end having to figure out or implement the protocol that encoded it. It's really hard to pass around objects like Powershell does if you can't even get reliable boundaries, let alone have to parse and unparse them yourself. Working with streams was a great idea of unix, no doubt about it, but abandoning records entirely shouldn't have been part of the package.

Re:This article...

[Breakfast Pants]

Just use XML. Pipe into and out of parsers etc.

Re:This article...

[shird]

It works.. but it won't zip the result.

Create two tar files with the same contents using the command given. Run gzip over one of the files and compare the files size differences. Try it.

He is creating a '*.tar.gz' filename, but only a '*.tar' filetype.

Re:This article...

[shird]

Hmm my bad, he is extracting, not creating, an archive. (been a while since I last made that comment).

Still.. with regular tar (not GNU tar) you cannot do that. But then you can't pass the -z option either. You have to manually (g)unzip first.

silly...

[Timothy Brownawell]

What would be more useful would be things like "have your prompt include the current path" and "use a text editor to write small scripts instead of trying to type really long lines". Also perhaps "diff -u <(command 1) <(command 2)" for comparing command output, although this goes more in the "useful tricks" category (along with most of the things from the article...) rather than the "good habits" category.

Re:silly...

[LiENUS]

"have your prompt include the current path"

Bad idea as 'cd /home/eviluser; ls' can end up running /home/eviluser/ls which contains 'rm -rf /*'

Re:silly...

[Anonymous Coward]

I think he meant adding \w to PS1, not adding . to PATH.

Re:silly...

[siride]

That's not what he meant. He meant having the actual displayed prompt show the current path, like the Fedora prompt: [user@host /usr/bin]$

Re:Old News

[Anonymous Coward]

Shocking. Digg is fast on news, but usually the comments will digg down anyone with even the tinest grasp of the subject matter in favour of totally incorrect information that 'feels' right. While slashdot is slow as hell to get the story up, but will usually have useful comments.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:Why?

[Timothy Brownawell]

What is Unix?

*nix is a highly modular component-based software system with a standard interface (flat byte streams) between components, and a basic set of standard components (given in the POSIX standard) that can be relied upon to always be present.

Re:Why?

[Anonymous Coward]

Why the fuck would you bother to reply to the GP? Either you have a massive and debilitating lack of social skills, or you're more drunk than I am right now. If it's the latter, I'd recommend having your stomach pumped in the near future.

What is Unix? Jeebus Christ.

Re:Old News

[Anonymous Coward]

Digg is full of posers who think using Cygwin is the same as using a *nix.

not helpful

[juergen]

This is in part about UNIX shell (bash alike) scripting, Tips & Tricks, and UNIX tools, not habits.

While in general useful on the commandline, these are very specific points, and some of them are quite weak. #10? come on, if you had "cat tmp/a/longfile.txt" in the command history, wouldn't appending "| grep and" be the most efficient way?

I also miss the most important and universial UNIX habit of all: do not login as root, use sudo or similiar tools when you have to.

This article seems to be driven by the need to publicize something, anything, bare of real content, like so many others at IBM devnet.

Re:not helpful

[fatphil]

``grep and !$''
is more efficient in every way than
``\up-arrow | grep and''
or even
``!! | grep and''

Unnecessary pipes and cats do not make things simpler. That's about the only thing in the article that is correct.

Re:not helpful

[juergen]

No it is not. Read my prior post again, and TFA. Saving 0.005 seconds as in their example is ridiculous. I rather save time by using the command history conveniently. Sometimes it really is most simple to append "| grep ..." to a command found by browsing the history visually, and this article was about command line usage after all, not scripting per se.

Even arguing this as one of 10 important points is insane and an utter waste of time.

Re:not helpful

[fatphil]

Navigating the command line is demonstrably not as simple as recalling a term from a prior command line. As I demonstrated.

And regarding the cat - did you not see the sys time _triple_? I somethimes do greps over huge files - I certainly don't want to waste vast amounts of time piping data between processes. You seem very hung up by the 0.005s figure - are you too thick to realise that that number will scale up as the input size increases? It seems so.

But what the heck - spend 3 times as long inside the OS in an I/O bound task if you want. Me, I'll do things the sensible way, and happily patronise the wastrels.

Re:not helpful

[juergen]

"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil." (Hoare and Knuth)

It might be useful to patronize the wastrels and give you a cozy feeling, but it won't get real work done when the day is over. (me)

But let's argue this for fun: Very likely most of that 0.005 seconds is one off from creating another process, and will not scale up noticeably for large files -- in which case disk I/O will dominate everything and another in memory copy for the pipe is marginal. A dual CPU (or core) system would even benefit, if these were more complex programs connected by a pipe. More complexity is another way of scaling up, a way in which your approach will loose. And don't give me a lot of if's now, those if's are exactly why advice #10 is useless to start with, it is based on too many assumptions, and not the obvious ones.

Re:not helpful

[fatphil]

I'm glad to see you base your "very likely" on complete guesswork.

I particularly like the way that you think that unnecessary transfers between non-shared caches would be _quicker_ than no unnecessary transfers.

Consider this, OS-call wise:
with cat: read + write + read
without: read

Your CV would be in the bin by now if I were hiring.

Re:not helpful

[juergen]

What about dual CPUs and paralell execution is unclear to you? These systems are quite common nowadays.

            CPU1                     CPU2, in paralell
with pipe   disk-read + mem-write    mem-read + crunch
no pipe     disk-read + crunch

disk-read + crunch > disk-read + mem-write
        if crunch > mem-write
disk-read + crunch > mem-read + crunch
        if disk-read > mem-read

Assuming both disk-read and crunch take up more CPU than the memory copy, the pipe version will be faster. Granted there are some assumptions, but they are equally likely than crunching really large files on the commandline in a performance sensitive manner.

I even tried to benchmark this. But the variance on my slow dualy testing box (P2-350 x2) is already so high there is no clear winner at all for up to 100MB files and simple grep patterns, so I decided to not waste more time doing a statistical relevance test. If ever I will need to tackle this problem, I won't need to do it on the command line.

Premature optimization indeed.

Re:not helpful

[fatphil]

Opteron Dual Core:

$ perl -e 'for($i=1;$i<10000000;++$i){print("$i$i$i\n");}' > test

$ ls -l test
-rw-r--r-- 1 Phil Carmody None 216666666 Dec 20 16:57 test

$ time grep -c '666' test
64893

real 0m0.654s
user 0m0.452s
sys 0m0.234s

$ time cat test | grep -c '666'
64893

real 0m8.858s
user 0m3.841s
sys 0m13.154s

It's not premature optimisation at all - it's chosing the right algorithm. Which is what Knuth recommends.

Re:not helpful

[juergen]

== First:

Choosing the right algorithm has to be seen in the context of TFA, 10 good unix habits on the command line. In this context avoiding pipes is premature optimization because:

- 99% of the time the performance will be irrelevant.
- In edge cases it may improve performance (when you crunch really large files on the cmd line).
- In edge cases it may decrease performance (scale up to more complex programs instead of only larger files).

If ever I've seen a harmful optimization example, there's one. Optimize for the right edge cases when you actually get them.

== Second:

Your values look really dodgy. Are those repeateable with the cache seeded? Even on my box the test file fits into RAM. What shell are you using? If time is not a built in command, in "time cat test | grep -c '666'" it will only apply to "cat", not the whole pipe.

Here are my values on a slightly loaded small box (dual p2-350 512M RAM, encrypted /home). And my zsh's built in time command gives timings for both parts of the pipe. test and test2 are identical files to your own.

First time file reads:

~/tmp/% time grep -c '666' test
64893
grep -c '666' test 2,20s user 1,80s system 16% cpu 24,256 total

~/tmp/% time cat test2 | grep -c '666'
64893
cat test2 0,05s user 2,52s system 18% cpu 13,652 total
grep -c '666' 2,24s user 1,23s system 25% cpu 13,649 total

It is quite hard to outguess the linux buffer cache, so take this with a big grain of salt. Repeating this with other files gave very different results.

After repeated file reads the timings stabilized to this:

~/tmp/% time grep -c '666' test
64893
grep -c '666' test 2,24s user 1,27s system 98% cpu 3,547 total

~/tmp/% time cat test | grep -c '666'
64893
cat test 0,10s user 2,26s system 57% cpu 4,080 total
grep -c '666' 2,37s user 1,24s system 88% cpu 4,078 total

Looks like after the buffer cache is seeded, you win 0.5 sec for this already extreme and untypical case, and slightly lesser CPU utilization. Before that, results are inconclusive and disk I/O dominates everything. Wow, this mis-advice really deserves #10 on good unix command line habits?!

Please don't bother trying even larger files, it'll be just more off topic. On the other hand, if you can find real world exmaples of things done daily where this really matters, I'd be interested to learn of them. But i will not respond to further contrived examples.

Re:not helpful

[fatphil]

Someone call the wahbulance - you've just proved yourself wrong with your own data.

In a public forum too.

Want a bandage for your foot?

Feet, that is.

Very helpful

[EvanED]

I really like this example:

~ $ time grep and tmp/a/longfile.txt | wc -l
2811

real 0m0.097s
user 0m0.006s
sys 0m0.032s
~ $ time grep -c and tmp/a/longfile.txt
2811

real 0m0.013s
user 0m0.006s
sys 0m0.005s

I am so glad that he showed what a difference can make, because I was *really* getting annoyed at having to wait that extra .084 seconds.

Re:Very helpful

[trip11]

There is no such thing as .084 seconds. Surely you mean .084 hours.

Re:Very helpful

[Anonymous Coward]

I wouldn't dismiss a speedup of factor 7.5 so easily.

Re:Very helpful

[je ne sais quoi]

Yeah I was just thinking, gosh that hint could be really helpful when hashing through a gigabyte sized file, which I have to deal with all the time. I don't know why slashdot is so down on these hints, they all seem like good practices to me. *shock* I even learned something (except I knew enough to not pipe cats already).

Re:Very helpful

[martin-boundary]

Especially since that example doesn't account for filesystem caching effects. There's no way of knowing if the bulk of the gain is because of the changed command or because the file is already in RAM, some background process was running, etc.

When timing commands, it's best to repeat the command several times and see if the times change significantly.

Re:Very helpful

[McNihil]

Haha... true but let say you have 50+ GByte file to grep (which I regularly have) that 7 times speed difference is a whole lot.

Re:Very helpful

[canuck57]

I am so glad that he showed what a difference can make, because I was *really* getting annoyed at having to wait that extra .084 seconds.

But if in perl, python or java??? Worse yet, for for loop over 1 million files?

Re:Very helpful

[multipartmixed]

Yeah!

Can you show me point 0 8 4 of a second? I didn't think so!

Don't blame me, I'm not a mathematitian!

Re:Very helpful

[multipartmixed]

I will almost certainly not be a 7 times speed difference on a modern UNIX platform. Most of that 0.084 seconds is probably taken up by fork..exec and pipe connecting.

Depending on the kernel, cat, and grep implementations, it's actually possible that it would be faster with the extra cat; but that's unlikely IME. The big thing, however, is that idiot boy committed the CS measurement faux-pas of the un-leet who understand nothing.... he applied ONE measurement to an entire class of problems, and that measurement was made only to back up his intuition and get people like you, to make wildly insane assertiations based upon single-datum linear extrapolation.

Incidentally, don't talk to me about non-modern UNIX platforms w.r.t. your example, because they would all break 48 gigs before your file made it into their filesystems anyhow.

Re:Very helpful

[Sentry21]

Facetious though this post may be, when you're grepping 8.2 gigs of logs, those milliseconds add up.

On top of that, another trick that no one ever uses is alternate grep modes - fgrep specifically. When you're just grepping for a string and not a regex, use 'fgrep' or 'grep -F', and grep will just do simple string matching instead of regular expression matching. On huge logfiles, this can result in some pretty astonishing speed increases.

As with the rest of the article, it's another example of being aware of your tools, and learning to optimise your workflow.

Re:Very helpful

[MoogMan]

-c is a GNU extension. i.e. if you work with both Linux and UNIX (e.g. Solaris), then you wont be using -c with /usr/bin/grep. Using grep file | wc -l would work on both machines, therefore is probably the better one to use.

Re:Very helpful

[nuzak]

> When you're just grepping for a string and not a regex, use 'fgrep' or 'grep -F', and grep will just do simple string matching instead of regular expression matching. On huge logfiles, this can result in some pretty astonishing speed increases.

I'm fairly sure that GNU grep knows enough to switch to a BMH or KMP or like algorithm when it gets a string with no regex metacharacters. If it doesn't, it's clearly broken.

(speaking of broken, what is ubuntu's problem with enabling the -P flag to grep like every other damn distro does?)

Re:Very helpful

[McNihil]

true and one should actually use "grep -P" if possible these days which is in my experiance two magnitudes faster than "normal" grep for my purposes. But then again parsing big logs is better done with perl in any case... if the parsing needs to be identical (which they aren't when bug hunting.)

Things I had to learn the hard way

[gd23ka]

1. Don't rm with an absolute path because you could easily

#rm -r -f / tmp/dir

when "all" you wanted was

#rm -r -f /tmp/dir

instead do this:

#(cd /tmp ; rm -r -f dir)

or even better use sudo if you have it:

$(cd /tmp ; sudo rm -r -f dir)

2. When logged on as root or when using sudo on a production system think things over
at least twice before hitting enter.

3. Make sure at all times you're on the right machine, logged on as the right user in the right directory.
Set up your shell prompt to look like this user@host /path$

Re:Things I had to learn the hard way

[SharpFang]

Also DO remember 'su -' changes working directory.

% cd ~
% cd somejunk
% ls
just junk, junk...
% rm -Rf *
somejunk: permission denied
% su -
Password:
# rm -Rf *^H^H^H^H^H^H^H^Hpwd
/

Re:Things I had to learn the hard way

[sarathmenon]

you could easily
#rm -r -f / tmp/dir
when "all" you wanted was
#rm -r -f /tmp/dir

You are forgetting one thing - there's no solution for stupidity and lack of common sense. While tips like these are generally useful, the person who's going to screw up a system will ignore you, and the zillion other tips that people have taken efforts to write. I've seen people who've run a rm -rf /bin to clean the recycle bin and then wonder what happened.

Its hard to bring in any improvement in the average unix admin, and part of the reason is that unix is unlike windows an OS that expects people to think and run sane commands. Its hard to cultivate a habit like that especially when the average-joe-fng-admin is used to next-next-next install processes and right-click-select-click-click operations. I'd love to see a change, but I feel that the IQ of the average person using a computer (any computer for that matter) is dropping sharply these days.

(I am not trolling, but I *am* sure that this will be voted down as one)

Re:Things I had to learn the hard way

[Anonymous Coward]

Don't use bash, use zsh instead. (Yeah, I got bitten by that "rm -rf some_dirname *" as opposed to "rm -rf some_dirname*" once too many. zsh asks first, without resorting to asking about bloody everything like rm -i does.

Re:Things I had to learn the hard way

[tendays]

1. Related advice:

I sometimes make the typo of pressing enter instead of for instance shift ...

So let's say I want to do rm -r a/b/c. On my keyboard layout I need shift to type a slash, and it has happened to me that when trying to do the first slash I actually press enter, and execute rm -r a :-(

So when doing something potentially dangerous I start by typing a #, and when done, control-A delete to remove the hash and run the command. If I accidentally press enter then it will try to run something like #rm a and ignore it or fail (depending on the shell).

Of course, cd-ing to the parent directory first is even safer but after a while you get lazy (I do) :-)

Re:Things I had to learn the hard way

[TeknoHog]

On my keyboard layout I need shift to type a slash

This is exactly why I prefer the US/UK keyboard for unix stuff, even if I live in Finland, Finland, Finland, the country where slash needs a shift.

There are similar annoyances with a few other punctuation characters in the Fi keyboard. Besides, it's easy to setxkbmap when you need to type with your native language.

Re:Things I had to learn the hard way

[maxwell demon]

When I do an rm which removes more than a single file, I quite often first type it prefixed with echo, look at the output, and after having verified that it is really what I want I go back in the command history, remove the echo and press enter. This is especially good with wildcards, which unintentionally might match something I didn't intend to remove.

Re:Things I had to learn the hard way

[GreyKnight]

1. Don't rm with an absolute path because you could easily

#rm -r -f / tmp/dir

instead do this:

#(cd /tmp ; rm -r -f dir)

That's an improvement, but there are still lots of ways to shoot yourself in the foot with that pattern.

Assume you want to rm -rf /tmp/lib, and your working directory is /. A simple mistyping (cd /tpm ; rm -r -f lib) just wiped out your /lib directory.

(cd /tpm && rm -r -f lib) would solve this, since the second command wouldn't be executed unless the first succeeded, but even that isn't really safe:

Let's say you are in *any* directory, and you mistyped: (cd / tmp && rm -r -f lib)

On the version of linux I'm using, cd ignores the second parameter, and you just ran rm -rf /lib.

You should really do this sort of thing in two steps...

kanazuchi ~ #
kanazuchi ~ # cd / tmp
kanazuchi / #
***(Oops)***
kanazuchi / # cd /tmp
kanazuchi tmp # rm -rf lib

There's no substitute for slowing down and reading what you type. A healthy dose of paranoia helps.

Re:Things I had to learn the hard way

[physicsnick]

What the?? All of the replies to this post surprise me. Everyone has all these elaborate ways to prevent themselves from accidentally using rm -rf on an incomplete command; subshells, pound signs, ls... Why do you all want to type -rf right away? Here's something that should have been in the guide:

If you're using rm recursively, always type the modifier *AFTER* the path.

#rm /a/b/c -rf

If you accidentally hit enter after typing "rm /", you get "rm: cannot remove directory `/': Is a directory".

Re:Things I had to learn the hard way

[SuurMyy]

> Set up your shell prompt to look like this user@host /path$

And colour-code it. I have different colours for different hosts so that I won't fek up.

Re:Things I had to learn the hard way

[derubergeek]

Don't rm with an absolute path because you could easily

#rm -r -f / tmp/dir

We had someone high up in IT do exactly that from a root cronjob. His solution to prevent this from happening again was obvious and quite clever. He disabled cron for non-privileged users.

Seriously.

Re:Things I had to learn the hard way

[nightfire-unique]

If you're using rm recursively, always type the modifier *AFTER* the path.

#rm /a/b/c -rf

If you accidentally hit enter after typing "rm /", you get "rm: cannot remove directory `/': Is a directory".

This behavior isn't standard across unix platforms. Sun file utilities, for example, will not accept parameters in just any particular order. This is something mainly the GNU tools do well.

It would surprise me if this method worked on GNU tools, as GNU getopt has a very specific protocol for command line interpretation.. this would be a nonstandard way of handling it (not like it would be the first, however).

Re:Things I had to learn the hard way

[houghi]

What I have learned to do was type rm whatever -rf instead of rm -rf whatever.
It gives me that bit of extra time to think between what I want to delete and what I actualy do delete.

Also it helps me to not delete whatev by accident instead of whatever

Re:Things I had to learn the hard way

[multipartmixed]

I learned that the hard way, too, back in my first compiler construction course...

# rm file.y file.l
# ls

file.y.out file.y.cc file.o blah blah
you get the picture

# rm *

Permission Denied: You were building as root, shithead

# su -
Password:
# rm *

<whirr whirr>

Me: Why is my harddrive going crazy?

<whirr whirr>

Me: oooooh SHIT!

^C^C^C^C^C^C^C^C^C^C^C^C^C^C

CTRL-ALT-DEL

System: INSERT BOOT DISK

Me: Hello? Faculty of Arts and Sciences? Yes... This is student 338-xxxx. I'd like to drop CISC 458..

(Glad I re-took it the next year, though. VERY good course, better prof too!)

Re:Things I had to learn the hard way

[ucblockhead]

If you use a limited set of machines, also use color in the prompt to differentiate them.

Re:Things I had to learn the hard way

[multipartmixed]

FWIW, GNU has their own version of getopt, for handling long arguments.

*shrug*

I've never actually used it, except maybe without knowing about it.

The odd case where I need to handle -- arguments, I just use string-to-character lookup function and shortcut the getopt loop with a goto during the the '-' case. :)

Re:Things I had to learn the hard way

[dirtyepic.sk]

1. Don't rm with an absolute path because you could easily
 
#rm -r -f / tmp/dir
 
when "all" you wanted was
 
#rm -r -f /tmp/dir
  alias rm="rm --preserve-root"

Re:Things I had to learn the hard way

[Anonymous Coward]

Don't rm with an absolute path because you could easily

#rm -r -f / tmp/dir
when "all" you wanted was
#rm -r -f /tmp/dir

Just learn to stop, go back, and re-read your command line before you hit return whenever you use 'rm'. That's how I learned the hard way.

Re:Things I had to learn the hard way

[jZnat]

This is why I use tab-completion ALL THE TIME. I'm sure that tab is the most used key after space when I'm in a shell.

Re:Things I had to learn the hard way

[jZnat]

...which is why I prefer sudo. You can just press up a couple times to get back to the command you wanted to do, prefix with sudo, and be done with it. Of course, you could also prefix with su -c " and suffix with ", but I still prefer sudo over su any day.

Re:Things I had to learn the hard way

[physicsnick]

nick@nick:~$ lsb_release -d
Description: Ubuntu 6.10
nick@nick:~$ apt-cache policy coreutils|grep Installed
    Installed: 5.96-5ubuntu4
nick@nick:~$ mkdir -p a/b
nick@nick:~$ find a
a
a/b
nick@nick:~$ rm a -rf
nick@nick:~$ find a
find: a: No such file or directory

Definitely works on this version of GNU tools. I've never used any other Unix variants, but this is always a good practice to use on Linux machines.

Re:Things I had to learn the hard way

[multipartmixed]

...unfortunately, sudo did not exist when I started with UNIX.

Well, more specifically, it did not exist outside of CU-Boulder, and certainly did not ship with the version of slackware (which I downloaded at school and brought home on FLOPPY DISKS because 2400bps was too slow) I was using at the time.

Point taken, though -- sudo *is* a valuable tool, I should really integrate it into my daily habits... but I've got the "su XXX ^D" pattern pretty much hardwired into my mental pathways.

Re:Things I had to learn the hard way

[MoogMan]

Set up your shell prompt to look like this user@host /path$

I showed this trick to my collegue at work. Unfortunately, he names all his machine `machine`.

Re:Things I had to learn the hard way

[phliar]

I only use absolute paths with "rm -rf". Using an absolute path means you can recall it from the history and re-execute it, no matter what the current directory happens to be. At the same time I want to reduce the number of keys I have to type.

(cd /tmp ; rm -r -f dir)

Also, those commands are in a sub-shell. I use sub-shells sparingly, because any changes you make in the environment in a sub-shell do not propagate back to the caller. I have suffered many hours of trying to figure out why the damn variable "kept getting reset" when it was set in a sub-shell. But I suppose it makes sense to use two "pay attention!" operations together.

Many years of being a sysadmin put in place a low-level reflex so that every time my eyes see "rm -rf " I immediately step back, reset, and think about the arguments and environment. That is the one command that I cannot hit <enter> on reflexively. (Individual files I do accidentally delete by accident, just no "-rf".)

If you're ever in an environment where you have a large number of hosts that you login to -- you're a sysadmin, or working on clusters -- you'll learn you cannot always rely on crutches like aliasing rm etc. with "-i". You have to pay attention and reflect before you hit CR.

Re:Things I had to learn the hard way

[that this is not und]

I've seen people who've run a rm -rf /bin to clean the recycle bin and then wonder what happened.

You mean they were confused by all the 'permission denies' messages they got, or were they already doing something terrible, terribly wrong before issuing the command?

Re:Things I had to learn the hard way

[Phaedrus420]

#rm -r -f / tmp/dir

Why does that look so familiar?

Oh, yeah;
rm -rf /home /phaedrus.old

Thank freshmeat for unrm

Re:Things I had to learn the hard way

[Anonymous Coward]

Quick, someone give this man the Medal of Freedom he deserves.

Re:Things I had to learn the hard way

[Anonymous Coward]

Having the path in the prompt is a waste of space IMO. If I need to remember where I am, I type
  three letters:

pwd

Re:Things I had to learn the hard way

[lahi]

Ah, this reminds me of another good habit.

DON'T use aliases to protect yourself from nasty behaviour. If you get to rely on it, even the slightest bit, you will become careless and one day you login without your normal profile, and thus witout aliases, and *whoosh* you just messed up. And now you probably messed up a system that you are not used to work on (because otherwise you'd have had your comforting aliases), which makes it even harder to clean up the mess.

Anything involving rm should never be habitual or "safety-enhanced".

-Lasse

Re:Things I had to learn the hard way

[ralph.corderoy]

#(cd /tmp ; rm -r -f dir) No! Do "cd /tmp && rm -rf dir" else if the cd fails you remove ./dir in error.

People use && far too little, e.g. the number of `how to install from source' guides that say "./configure; make; make install" instead of "./configure && make all check install".

Re:Things I had to learn the hard way

[dirtyepic.sk]

That's like saying you shouldn't use an umbrella in the rain because one day you'll forget it and get wet.

Re:Things I had to learn the hard way

[gd23ka]

"I only use absolute paths with "rm -rf". Using an absolute path means you can recall it from the history and re-execute it, no matter what the current directory happens to be."

So you actually !rm from history and then take whatever you got coming like a man?

(Individual files I do accidentally delete by accident, just no "-rf".)

Well if that's the case I suppose you should get a second opinion before hitting enter... :-)

Where did you get all that from? The "BOFH Best Practices Manual on UNIX Systems" ?

Re:Things I had to learn the hard way

[phliar]

So you actually !rm from history and then take whatever you got coming like a man?

That's right. And I build my own computers too. Starting from refining the ores.

Actually I don't use history substitution, just command-line editing -- ^Rrm to bring back the last rm, and hit return. If there's a "-rf" on the line the alarm goes off and I examine the command before hitting return; if not -- well, that's why we have backups, and the few hours of work I just lost is just a valuable lesson in the use of power tools. At least with our power tools we don't lose fingers.

(I learned my lessons being the lowest grunt on the totem pole (aka tape monkey). When I screwed up and needed to restore something from the backups, I'd be the one doing the shit-work. Very liberating to be able to screw up and not have someone yell at you.)

Re:Things I had to learn the hard way

[gd23ka]

1. I prospect for and mine my own ores and rare materials.

2. "Going back" with ^R.. (so you're using some sort of gnuish shell that uses libreadline) is okay I suppose.

"if not -- well, that's why we have backups, and the few hours of work I just lost is just a valuable lesson in the use of power tools. At least with our power tools we don't lose fingers."

I think very on to something here. The other extreme to that would be some people who cut off their fingers intentionally
with a knife after they have failed dishonorably. The problem with that however is that most of my customers accept
neither apologies nor amputated limbs. They do take cash however :-)

Whoever you're working for, happy monkeying around with their systems. Why I even envy you.

Re:Things I had to learn the hard way

[phliar]

The problem with that however is that most of my customers accept neither apologies nor amputated limbs.

For those, you just need a different kind of customer!

Whoever you're working for, happy monkeying around with their systems. Why I even envy you.

That was a long time ago. I was only an apprentice-BOFH; sure it was a lot of fun and I got to really learn Unix, and in all flavours (VAXen, Suns, Sequent Symmetry, AT&T 3B5, HP-UX, etc.), but I think being a sysadmin is the worst kind of job: people only notice you when things are fucked up. I don't envy sysadmins.

Re:Things I had to learn the hard way

[kayditty]

actually, it's like holding a tractor which functions as an umbrella over your head, but, one day, you're at a tractor factory and you are confused about which tractor belongs to you. having picked up the wrong tractor, the umbrella functionality is no longer functioning-ality, and you get wet (assuming rain is falling at this time). it also requires a pretty light tractor.

absolute drivel

[Anonymous Coward]

This is, without a doubt, the most worthless article I have ever seen, both on Slashdot and on ibm.com, of which I thought better. It is not that the article is boring, but that it is factually incorrect in some places.

"the only excuse to define directories individually was that your mkdir implementation did not support this option, but this is no longer true on most systems. IBM, AIX®, mkdir, GNU mkdir, and others that conform to the Single UNIX Specification now have this option."

This is nonsense. The expansion of the path components in the {braces} is not a function of mkdir(1), but of the shell, and how its argument expansion is configured. I cannot believe that anyone "with 20 years of experience" is brazenly quoting names of standards in an effort to give his ramblings an air of credibility. Actually, wait a minute...

Another bad usage pattern is moving a .tar archive file to a certain directory because it happens to be the directory you want to extract it ...

Better is to check what's in the archive before extracting it in case some inconsiderate fool has failed to put a top-level directory in it.

His research interests include digital publishing and the future of the book.

Let me give you a couple of hints.

Re:absolute drivel

[Anonymous Coward]

This is nonsense. The expansion of the path components in the {braces} is not a function of mkdir(1), but of the shell, and how its argument expansion is configured.

RTFA. He was referring to the -p parameter which automaticly creates the parent directories. That has nothing to do with the shell. If this is the only flaw in the article you could come up with then I guess its not as bad as you claim afterall.

Re:absolute drivel

[Anonymous Coward]

He wasn't talking about the expansion of the braces, but the -p option of mkdir, instructing it to create parent directories. Just because someone else posts drivel doesn't require you to compound it.

Re:absolute drivel

[portnoy]

He's not talking about the braces construct, he's talking about the -p option to mkdir. Just because you can't read doesn't mean the article is factually incorrect.

Re:absolute drivel

[SanityInAnarchy]

Better is to check what's in the archive before extracting it in case some inconsiderate fool has failed to put a top-level directory in it.

Which is why I always extract to a temporary directory first, then either rename that directory (if they're an inconsiderate fool) or move the top-level directory out of it (if they aren't an inconsiderate fool).

Personally, the habit I always get into is moving the tarball to wherever it makes sense to have it, then cd'ing to where I'm going to unpack it. I never use -C.

Re:absolute drivel

[Anonymous Coward]

I'm in the strong habit of using t to see what will be created when I x the file. This holds for ZIP files, as well. Although on the bright side, the number of inconsiderate fools seems to have droped off in the last few years, thankfully.

Mostly BS

[gweihir]

I don't know whether it is complete BS, as I stopped reading after the 3rd or 4th "good habit". Maybe AIX has an exceedingly slow ''cat'', Linux does not. Maybe the other stuff is valid in some OSes, but then I have not encounterd them. By the same token, really important stuff is missing completely. This looks a bit like a Unix-noveces initial list of things he/she thought about in the first week. Really not worth your time.

Re:Mostly BS

[TheRaven64]

The cat thing is not likely to make a difference on small files, however, there are some reasons why it is bad practice:

1. If you are using it in a find -exec command, then an extra process creation/destruction for every result will create a lot of overhead.
2. By using cat, the utility examining the file just thinks it is looking at stdin. This prevents it from scanning backwards (unless it buffers it internally, which is inefficient), and it prevents it from reporting anything related to the filename. Again, bad if it's being run from something like find.
3. Running something like grep on a very large file can be a lot faster if it is reading the data directly, rather than having it read through cat and a pipe.

In practice, it's often easier to use cat, although unless you are actually concatenating files it is usually easier not to; < is less to type than cat. Unless you have to enter HTML into your shell...

Re:Mostly BS

[TheLink]

I suggest not ever using < for that. It's next to > , and the shell is likely to allow the command if you accidentally use one instead of the other

I'm guessing it's not funny when you use > when you intended to use < . ;)

I don't recall ever making that mistake yet, but that's probably because I rarely use < to read files.

Re:Mostly BS

[TheRaven64]

Hmm. I've never managed to type > instead of <. I have managed > instead of >> a few times though. That can be quite distressing.

Re:Mostly BS

[TheLink]

Ouch... I must have done that before, can't remember tho. Or maybe I've seen people do it :p.

I think someone should come up with a typo friendly ">>" :).

Anyway, I'd prefer learning clever tricks to do unix commandline stuff safer.

Ever got the sinking feeling of realizing you now have an extra 1GB space on your filesystem? ;)

I heard a story of a product called star. A customer was told over the phone that the very simple uninstall procedure was rm -rf star

There was a fair bit of distress when it was realized that the uninstall process was taking longer than expected ;).

Hmmm, I wonder what the uninstall procedure is like for that popular open source PBX software... Hehe.

Re:Mostly BS

[gweihir]

You seem to assume bad == slow.

That is a seriously wrong approach! What about the probability to run into errors? What about that the 'cat' normalizes the data stream to STDIN?

Re:Mostly BS

[gweihir]

And in addition, it hardly matters anyways. 5% slower is not a justification to call something ''bad''. It is however a complete waste of time to even think about it.

Re:Mostly BS

[TheRaven64]

I think someone should come up with a typo friendly ">>" :). You can do this with zsh. It supports 'global aliases,' which are like aliases except that they can go anywhere on the command line. One example I have seen is to alias G to '| grep', so you can pipe the output from any command to grep with one character. The same mechanism could be used to create an alias for appending data to a file.

Anal Unix Guy

[jellomizer]

The title should be 10 Good Unix Hints. Not Habits.

These things he is stating is like when an other Unix guy is looking over your shoulder and you type in a command that works perfectly fine but has to correct you to do it his way, to save the extra millisecond in CPU while wasting about 5 Seconds to retype the command.

Secondly depending what you are doing on Unix you may or may not need to use these commands often enough to remember all the parameters for every tool. like the mkdir -p, While I know mkdir could create the entire directory Tree with some parameter, I just never bothered using it, or looking it up. I may see it in a script made by someone and realizing mkdir creates paths and that I know the directory doesn't exist yet. So I figure that is what it is doing. But for myself I rarely create an entire directory tree like he stated first. I like to create a directory find what file and folder I need and put them there when I need them.

Re:Anal Unix Guy

[jgrahn]

The title should be 10 Good Unix Hints. Not Habits.

Yes -- and habits is what people desperately need. The people I know primarily need three habits: RTFM when they don't understand something; adjusting their behavior based on the FM; and managing their use of the current directory (i.e. you don't have to cd into a directory to use a file which lives there).

Re:Anal Unix Guy

[TeknoHog]

like the mkdir -p, While I know mkdir could create the entire directory Tree with some parameter, I just never bothered using it, or looking it up.

Interesting... I'm used to using mkdirhier for the same effect. I thought it was simply an alias to mkdir -p, but it turns out to be a shell script (part of X11 imake package) that uses plain mkdir. I better update my scripts to the more portable version now...

Eh? There's more to Unix than shell scripting

[Taagehornet]

"10 good habits that improve your UNIX command line efficiency" would probably have been a better title.

The title did however bring back fond memories of Eric Raymond's The Art of Unix Programming. The book is available online, and if you were hoping for something a bit more substantial as well, then the section Basics of the Unix Philosophy might be worth a read.

Re:Eh? There's more to Unix than shell scripting

[Russ Nelson]

Yes, some idiots (e.g. Jim Thompson) claim that Eric isn't a good programmer, and yet he somehow has gotten the insight to write this book. Wonder of wonders!

Yeah, parent post is written in sweat and blood!

[dallaylaen]

I'll dare to continue.

4. Log out of the root session as soon as the moment's job is done.

Better type password yet again than clean the mess.

5. Never use cat to view files. There are tail (1), head (1) and less (1) (and even more) for that. Pumping zetabytes of logs over the network *is* a bad habit.

6. The good old 'rm -rf * .o' joke is so well known... I type
% ls -l ... ,
then arrow up and replace 'ls -l' with 'rm -rf'. Or even use Midnight Commander to delete files (heresy, yeah).

Also, I often use constructs like
% for i in foo bar baz; do echo somecommand $i; done
before removing the 'echo' part.

Just to see what I'm going to do.

Gonna read other people's advices now...

Re:Old News

[Anonymous Coward]

Digg is full of posers who think using Cygwin is the same as using a *nix.

Speaking as a regular and experienced user of both, I have to wonder exactly what you're trying to say. Cygwin comes close enough to POSIX to make the comparison meaningful. It's about as close to Unix as, say, OS X is.

(What is "using Unix" supposed to mean, anyway? I know people who've used Solaris desktops for years who still treat the command line as an arcane interface to be used with fear and trembling as a last resort when all GUI-based alternatives have failed.)

ls -l | xargs(?!)

[glas_gow]

I'm still trying to figure out what listing 16 does:

ls -l | xargs
-rw-r--r-- 7 joe joe 12043 Jan 27 20:36 December_Report.pdf -rw-r--r-- 1 \
root root 238 Dec 03 08:19 README drwxr-xr-x 38 joe joe 354082 Nov 02 \
16:07 a -rw-r--r-- 3 joe joe 5096 Dec 14 14:26 archive.tar -rwxr-xr-x 1 \
joe joe 3239 Sep 30 12:40 mkdirhier.sh

When I do this it doesn't include the \ continuation, and even if did, why would anyone want that on a single line?

Re:ls -l | xargs(?!)

[Anonymous Coward]

I think he's using a number one (-1), not a lowercase L (-l).

Re:ls -l | xargs(?!)

[Jesus_666]

Maybe to have heaps of cryptic-looking data scrolling across the terminal window to fool your boss into thinking that you're just playing Tetris because you're waiting for this incredigby important script to finish crunching all that data...

Re:ls -l | xargs(?!)

[glas_gow]

Listing 15 uses hyphen 1 (-1). But listing 16 definitely uses hyphen l (-l).

I'm sure it was just an arbitrary example of xargs ability to reduce multi-line input to a single line. If it was that useful a facility I'm sure he'd have come up with a better example of its use.

Re:ls -l | xargs(?!)

[multipartmixed]

This jackalope probably once something that LOOKS like this, didn't understand it, and is now spouting it as "Wisdom".

He probably saw somebody type:

ls -1 | xargs

Which, of course, is the same thing as ls | xargs on a non-broken ls.

By non-broken, I mean that ls -1 is the default behaviour of ls when isatty(STDIN_FILENO) or isatty(STDOUT_FILENO) return non-zero.

Or, perhaps, he's planning to use xargs to parse, say, filesizes out of ls? But that would be incredibly annoying, especially if your filenames contained spaces.... or something.

I dunno. I have always found xargs in newb books, but I have VERY RARELY used it in the real world. I am a big fan of using find to key iteration.

find dir -type f | while read file; do echo "$file"; done

Want the filesize instead?

find . -type f -ls | while read line; do set -- $line ""; echo $7; done

You could also use ls similarly, but ls gets slow for LARGE directories, as it must sort the results. Find will just traverse the directory, spitting stuff out in disk order, meaning you can do processing on each file while the next directory entry is being read. I think ls (at least on Solaris) must be implemented with bubblesort, it's SLOW for any N over 100,000 or so, no matter how big the box is.

Try it the other way 'round....

[Anonymous Coward]

there is this thing called a buffer cache.

You see,

[Anonymous Coward]

we never write "(cd /tmp ; rm -r -f dir)", because the first operation might fail.

We write "(cd /tmp && rm -rf dir)". ;)

But I guess that's different on linux, and the system actually is built to know that you didn't mean to remove anything from your current working folder in case the first operation fails...

sudo is indeed better, because it allows you to do this kind of mistake on _every_ folder. Including the important system folders.

Another advise: if you make a lot of mistakes, TYPE SLOWER. You'll save a lot in the end.

tar comment

[thomasa]

In their example with tar they did

tar xvf

without the dash. (E.g., tar -xvf)

While that does work, I prefer to add
the dash as it makes it more consistent
with the other commands. So I consider
that a bad example. tar is one of the
older commands like dd that have weird
command line syntax.

Re:tar comment

[multipartmixed]

Entertainly enough, the linux b0yz of the late '90s thought that the dash was superfluous everywhere, and so made ps bitch if you used it.

echo 'export I_WANT_A_BROKEN_PS=fu' >> .bashrc

oughtta fix that for ya. :)

Now, if only I could figure out how to quickly fix the linux xterm termcap on every linux box I use....

Re:tar comment

[Anonymous Coward]

How is this "informative"? thomasa doesn't like the way Michael Stutz uses the "tar" command. Great. There's nothing wrong with the example in the article. If thomasa likes to stick dashes in where they're not required, that's fine. But it's not informative. Certainly not +4 informative.

Re:tar comment

[kjamez]

... yeah, like that pain in the ass 'ps' command always TELLING you not to use the - ... those bastards.

Actually useful hints

[Artraze]

As has been pointed out, this article is riddled with errors. It's also not very interesting. So in the interest of perhaps actually providing some interesting tips:

In scripts, prefix dangersous commands with an 'echo' for a test run (So you can catch all those rm -rf /).

Single quotes are the best quotes for plain strings. The only reasion to use double quotes is if you need to quote a variable or a single quote.

Completion is fun, but using wildcards is more flexible (though you'll only want to use benign commans like cd, less, etc):
    nano /etc/modules.autoload.d/kernel-2.6
    nano /etc/m*a*d/*6

Note that the use of subpaths reduces the amount of flexibility.
    cd /etc/m* -> /etc/mail
    cd /etc/m*d -> /etc/modules.d
    nano /*/m*/*6 -> /etc/modules.autoload.d/kernel-2.6, and /etc/modules.d/i386 (not quite!)

Finally, as a comment for the article, using:
    test -e $DIR || mkdir -p $DIR
is much better than their suggestion and probaly faster anyway. Though I'd just do "mkdir -p $DIR" and maybe "&>/dev/null" under most circumstances anyway.

That's all I can think of at this point. Anyone else have tips?

Re:Actually useful hints

[multipartmixed]

IMNSHO, wildcards should only be used where shell completion won't work at all. This is part of instilling good habits. :)

I prefer [ -e $DIR ] || mkdir -p $DIR, but this is exactly the same solution as yours. (/bin/[ is a soft link to /bin/test on most unices).

And I think you meant 2>/dev/null >/dev/null. :)

Your solution backgrounds the command and redirects stdout to /dev/null.

Well, assuming you're not using bash. That's the case with sh and ksh.

More to the point

[metamatic]

I'll gladly trade 0.1 seconds of CPU time if it means I don't need to remember an extra command line option for the wc command.

That's why I'm of the v7 Unix philosophy. Make a tool that does one thing, and does it well.

If I later need to optimize for speed, then I'll read the man page.

lowercase uppercase

[dheera]

i wish there was an easy way to make filenames lowercase in mass. because i transfer images from my camera and sometimes they come up as IMG_1234.JPG and sometimes as img_1234.jpg and it screws with my website scripts if they're not all the same extension (.JPG != .jpg). so often i want to make an entire directory of filenames lowercase. the only trouble is there's no clear-cut way to do this, doing it in one line is hard, and googling for it always seems to give me a bunch of shareware or a complicated .tar.gz file instead of a single line to do what i wanted. i usually end up writing a Perl script, but it seems such an overkill and it seems that it ought to be easy and simple to do as a command.

Re:lowercase uppercase

[Scorpio]

for i in *.JPG ; do mv $i `basename $i .JPG`.jpg ; done

Re:lowercase uppercase

[Anonymous Coward]

rename old-word new-word *JPG ... so:

rename IMG img *JPG; rename JPG jpg *JPG

Alternatively (more general lowercase-izer:)
for i in *JPG; do mv "$i" "`echo $i | tr '[:upper:]' '[:lower:]'`" ; done

Re:lowercase uppercase

[dheera]

blah, bash scripts. okay. well, i can handle memorizing that, but bash syntax, especially of things like for loops and if statements beats me (see my comment earlier). what beats me is why there is a 'do' in there, why there is a 'done' rather than the would-be bash-ish 'rof' (for backwards) ? and why there are semicolons rather than some sort of { construct...

Re:lowercase uppercase

[dheera]

how i wish i could type @dir=~tr/A-Z/a-z/ at the prompt and be done with it...

Re:lowercase uppercase

[Anonymous Coward]

for file in $*
do
      mv "$file" $(echo $file | tr '[:upper']' '[:lower:]')
done

Re:lowercase uppercase

[Anonymous Coward]

sanity.pl

http://www.splitbrain.org/projects/sanity

Also,

http://filerenameutils.sourceforge.net/

http://sourceforge.net/projects/miscutils/

Re:lowercase uppercase

[Chandon Seldon]

Why is the perl script so hard? The command line would have a bunch of sed going on, whereas the perl script only requires running perl.

I'm guessing... perl -e 'for(`ls`) { chomp; $n = lc $_; system("mv $_ $n"); }'

Re:lowercase uppercase

[Anonymous Coward]

#!/bin/sh

for x in *
    do newx=`echo $x | tr "[:upper:]" "[:lower:]";`
    mv "$x" "$newx"
    echo "$x --> $newx"
done

Re:lowercase uppercase

[TheRaven64]

The example was not bash-specific, it was Bourne shell script syntax, which will work on any POSIX-compliant system (the existence of /bin/sh and certain behaviours are mandated by POSIX and SUS).

I presume the reason for the loops using do ... done blocks is so that for and while loops can have the same syntax. There are only a few things to remember. Take a look at the if statement syntax:

if {expression} ; then
echo true
fi

Note that in shell scripting, ; is a separator, equivalent to a newline, so this can also be written as:

if {expression}
then
echo true
fi

Actually, there is no need for the line break between the 'then' and the 'echo.' Effectively, this is inserting a null statement here. You could write the same thing again as:

if {expression} ; then echo true ; fi

Unfortunately, inserting a semicolon between 'then' and 'echo' won't work; being able to put a line-break after then is just a little syntactic sugar added to make multi-line if statements easier to read.

Now, moving on to while loops, we have quite similar syntax:

while true ; do
echo foo
sleep 1
done

The structure is very similar to that of an if statement; you have a keyword, a condition, and then two markers for defining a block (then and fi, do and done). A for loop is almost exactly the same. Just replace 'while {expression}' with 'for {variable} in {list}' If you are familiar with other scripting languages, it may help to remember that the POSIX-shell for loop here is closer to a foreach loop in other languages. The '{list}' part is a list of values, and each iteration will set the value of the variable to the next value in the list.

Re:lowercase uppercase

[value_added]

for i in *.JPG ; do mv $i `basename $i .JPG`.jpg ; done

Bonus points for avoiding cat, but there's no need to call basename.

$ man -P 'less -p Parameter\ Expansion' bash

An overly simplified example would be:

$ for i in *JPG ; do mv $i ${i/JPG/jpg} ; done

Obviously, you could rewrite that any number of ways (dealing with spaces in file names would be just one reason), but you get the idea.

Re:lowercase uppercase

[multipartmixed]

Oh, for the love of God, stop bitching about how hard this is to do under UNIX without package x.y.z installed and imagine doing it under Mac OS9 or Windows.

Break it down into its constituent parts: Iterate, rename. Whoo! Simple now, eh?

# cd cameraDir
# find . -type f -prune | while read file
> do
> mv "$file" "`echo \"$filename\" | tr '[A-Z]' '[a-z]'`"
> done
#

... should do the trick. If you don't have tr, you can use sed with the y command. But I can't think of the last time I saw a box w/o tr.

Re:lowercase uppercase

[dheera]

on a related note, if you try to mount a windows SMB share with Samba in linux ...

dheera@may-the-force-be-with-you /mnt/photos/test $ ls
ibm.JPG
dheera@may-the-force-be-with-you /mnt/photos/test $ mv ibm.JPG ibm.jpg
mv: `ibm.JPG' and `ibm.jpg' are the same file ----- AAAAAAAARGH
dheera@may-the-force-be-with-you /mnt/photos/test $ mv ibm.JPG ibm.foo
dheera@may-the-force-be-with-you /mnt/photos/test $ mv ibm.foo ibm.jpg
dheera@may-the-force-be-with-you /mnt/photos/test $ ls
ibm.jpg

grumble...

Re:lowercase uppercase

[multipartmixed]

Why is it that you perl weenies always bitch about how hard it is to solve problems in shell... Then implement the same solution, only with bugs?

My perl is weak, but I believe your solution fails address the cases where filenames have spaces, or where then are subdirectories in the directory hierachy which should be left alone.

Never mind the fact that you're escaping to the shell TWICE in one line of perl to the damned job done. Not to mention spawning a new shell for each and every file to be renamed!

You have written the equivalent of this program:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
 
void lc(char *out, const char *in)
{
  while (*out++ = tolower(*in++));
  *out = (char)0;
}
 
int main (int argc, const char *argv[])
{
  const char **list = NULL;
  size_t listSize = 0;
  char line[FILENAME_MAX + 1];
  FILE *pipe;
  const char *filename;
  char systemBuf[64 + sizeof(line) + sizeof(line)];
 
  pipe = popen("ls", "r");
 
  do
  {
    list = realloc(list, (listSize + 1) * sizeof(*list));
    if (!list)
      panic(); /* probably ENOMEM */
    if (read(pipe, line, sizeof(line)) > 0)
      *list[listSize++] = strdup(line):
    else
      *list[listSize++] = NULL;
  } while (!feof(pipe) && !ferror(pipe));
 
  for (filename = *list; filename; filename++)
  {
    lc(line, filename);
    snprintf(systemBuf, "mv %s %s", line, filename);
    system(systemBuf);
  }
 
  return 0;
}

.... hopefully re-writing it like this (in a real programming language) makes your problems more obvious.

A note for the pedants: I have not tried to compile the above C program, and so definately do not warrant that it will not shoot off your foot!

Note of course, that there are a few minor I/O bugs in this program that are not present in yours. I would normally never use a non-abstracted read() statement from a pipeline and assume it's going to completely fill the buffer. (Although it would in 99.999999% of reasonable cases). I don't know what perl does in these odd situations (for example, a pipeline stall, EAGAIN, ENOLCK, EIO, or (especially) EINTR).

Re:lowercase uppercase

[Teppic_52]

$ for i in *JPG ; do mv $i ${i/JPG/jpg} ; done
This isn't the first time I've seen this, but it will result in a file MYJPG.JPG being called MYjpg.JPG, ${i//JPG/jpg} would be better as at least the it would end up with the .jpg at the end, but ${i%.JPG}.jpg would be best.

Re:lowercase uppercase

[multipartmixed]

I am no SMB expert; in fact, I would NEVER think of a windows machine as being a "server" to a UNIX anything.. but (and I am also not a Linux expert, rather a UNIX guy), I seem to recall that there is an option for mount_smbfs to not do upper/lower case crapfulness. There certainly is in the other direction (windows machine mounting partition from a real OS).

If there isn't, it actually wouldn't be too hard to fix what you want, assuming that your smbfs is implemented a fairly normal unixish vfs switch. (If you don't know what this is, you really gotta read the samba docs, they did a really nice job explaining this concept).

Take a look at the rename system call which is hopefully addressed by this switch, detect the condition, and repair it.

Hmm. Scratch that idea, a little bit of debugging reveals that it's a red herring:

# truss -f mv w w
 
15730: execve("/usr/bin/mv", 0xFFBEF694, 0xFFBEF6A4) argc = 3
[ boring stuff removed due slashdot lameness filter ]
15730: lstat64("w", 0x00024540) = 0
15730: lstat64("w", 0x00024540) = 0
[ Argh... Anyhow, it immediately writes the error message to stderr ]
15730: llseek(0, 0, SEEK_CUR) = 263473
15730: _exit(2)

...The problem here is that mv checks for identical-ness before calling rename() -- it's probably checking the inode numbers in lstat. (In fact, I just manned rename, it doesn't even have an errno for same-name).

SO. Looks like the solution is to rewrite mv. That should actually be easier!

Find the lstat checks (probably actually calls to stat). If the inode number comes up the same, see what type of filesystem the files are on (statvfs?). If it's an smbfs, open your new file, carefully, like this:

open(path, O_WRONLY | O_CREAT | O_EXCL, 0600)

If THAT succeeds, go through to the rename call, otherwise it really *is* the same file.

I think that will work.

It would probably piss off POSIX though. Fixing the mount options of the smbfs would be more portable.

Re:lowercase uppercase

[newt0311]

oh, I ran into this problem too many times so I set up the following function:

function pmv ()
{
        local src dest oifs
        oifs=$IFS
        export IFS=$'\n'
        src="${1:? 'Error: input pattern not specified'}"
        dest="${2:? 'Error: destination pattern not specified'}"

        for i in $src ; do
                mv "$i" $(sed -e "s/$dest/" <<< "$i")
        done
        export IFS=$oifs
}

Now I can move files by regexps by typing pmv <file-match-pattern> <sed-matcher>/<replacement>
very very convenient.

Re:lowercase uppercase

[value_added]

$ for i in *JPG ; do mv $i ${i/JPG/jpg} ; done

This isn't the first time I've seen this, but it will result in a file MYJPG.JPG being called MYjpg.JPG, ${i//JPG/jpg} would be better as at least the it would end up with the .jpg at the end, but ${i%.JPG}.jpg would be best.

Again, there's lots of ways to do it. To use the trivial JPG -> jpg, example, yes, you're correct in that using the shortest match at the end would be a better approach (excluding other issues). I just wanted to illustrate the redundant (and typically overused) use of basename with a simple example, and remind the folks that using parameter expansion is preferrable both in interactive form, and in scripts.

Me, I've always relied on Larry Wall's script exclusively to rename files interactively. Scripts, on the hand, are often best written with /bin/sh in mind, and should as a rule be as simple, clean and efficient as possible.

Re:lowercase uppercase

[a.d.trick]

Me, I've always relied on Larry Wall's script [fsu.edu] exclusively to rename files interactively. Scripts, on the hand, are often best written with /bin/sh in mind, and should as a rule be as simple, clean and efficient as possible.

Wait, first you mention perl, and then you start talking about scripts being simple and clean?

Re:lowercase uppercase

[k8to]

Indeed, using perl to use the shell to rename the file isn't a really impressive point for perl. I'm sure perl has some module for sane renaming. Why do perl users inist on doing things the wrong way?

Re:lowercase uppercase

[Anonymous Coward]

I pretty frequently use Perl as a "command" language (in addition to writing longer scripts). Here's how I would lowercase all files in the current directory:

perl -e 'for (grep { -f } <*>) { rename $_, lc; }'

Re:Old News

[thomasa]

While it is not the same, it is the best thing on Windows that is close to UNIX.

Just gubberish.

[Gearrion]

Everyone is talking a different language. No wonder why nobody but nobody uses this operating system.

Re:Just gubberish.

[canuck57]

Everyone is talking a different language. No wonder why nobody but nobody uses this operating system.

To biased and pig headed to.

I always stress to new to xNIX types, learn the basics first before getting wrapped up in a perl/java/python or some other cult. Often these other languages are like taking elephants into china shops. And besides, they are not always on a system -- ditto vi versus emacs.

asserting that a variable is set

[Anonymous Coward]

The article didn't mention a tip for that can prevent you from shooting yourself in the foot when you dereference a variable that isn't set (or set set to an empty string). Here's a dangerous way to clean out a directory:

$ rm -rf $FOO/*

If FOO isn't set this command will try to delete every file in your system (and network mounts).

Here's a better way to clean out a directory.
$ rm -rf ${FOO:?}/*

If FOO is unset then the shell will not execute the rm command and gives a nice error
$ rm -rf ${FOO:?}/*
-bash: FOO: parameter null or not set

wget options to read urls from a file

[pikine]

According to man wget, you can use -i or --input-file options to read URLs from a file.

No actual habits in the article

[xyloplax]

Here are some more important general IT rules (Unix rules can easily be OS and version dependent and frequently come from usage in YOUR environment)

• Copy before edit
• Tape backup before delete/decommission
• READ YOUR COMMAND before hitting return
• Check where things are symlinked to
• Echo in your scripts instead of destructive commands as a test run
• Test your changes on a lesser-importance box
• Use proper Change Control procedures
• Cover your ass and capture your terminal output
• When taking something out of service, turn it off for a few days/weeks before deleting/purging it

Re:No actual habits in the article

[Anonymous Coward]

The above are indeed extremely valuable rules and much more valuable than anything in the posted article.

Even that's wrong / Ich bin Grammatiknazi

[The Monster]

The latter also works nicely across machine boundaries:
zcat arc.tar.gz| ssh user@foo 'cd tmp/a/b/c ; tar xvf -'

Only that isn't portable. Some flavors of unix zcat will try to uncompress $1.Z ; clearly there wouldn't be a arc.tar.gz file.

Fortunately, there's an easy fix for that: Use redirected stdin rather than a filename for zcat :

zcat <arc.tar.gz | ssh user@foo 'cd tmp/a/b/c ; tar xvf -'

Some people call me a 'Grammar Nazi' for being so picky about how people use English. I get frustrated about things like commmercials that say "check us out on the web at dubya dubya dubya dot ${COMPANY} dot com backslash ${PRODUCT}", the inability to distinguish single quotes, double quotes, and backticks; or parentheses, square brackets, and curly braces. As a tech-support person, I expect non-techinical people to make that sort of mistake, but in technical writing it's inexcusable. Calling something by the wrong name is where we get VerizonMath, or crash a space probe because someone didn't convert between feet and meters.

Re:Even that's wrong / Ich bin Grammatiknazi

[multipartmixed]

You should be using gzcat, not zcat, anyhow. zcat is only portably able to be compress -d.

gzcat will never be broken in the way described, hence the following is fine and portable IME:

gzcat arc.tar.gz | ssh user@foo 'cd tmp/a/b/c && tar -xvf -'

HOWEVER, I find that even vaguely modern CPUs are much faster at gunzipping than typical internet speeds. So, I would use this myself:

cat arc.tar.gz | ssh user@foo 'cd tmp/a/b/c && gzcat | tar -xvf -'

On the otherhand, I would never actually write that, because if I had the archive in place, I'd just transfer it with scp and untar it myself on the remote end. Unless, of course, cat in the example is just a place holder for 'arbitrary cool shit in the pipeline'.

HEY! PYTHON WEENIE! YEAH YOU, UP THERE!

Let's see you do this in your bloatware:

(cd /updateDir && find . -newer timestamp -type f | tar -T - -zcf -) | ssh user@foo 'cd /stagingDir && tar -zxvf -'

Incidentally, dropping the "z" flags and adding "-C" to ssh will make this totally cross platform, even to non-gnu-land, back as far as ssh 0.99 without significant penalty or performance difference. A reasonable alternative, before about 1992, would have been:

(cd /updateDir && find . -newer timestamp -type f | tar -T - cf -) | compress | rsh -l user foo 'cd /stagingDir && compress -d | tar -xvf -'

Moo hoo hahaha

What would you python weenies do if confronted with an AIX 2 or a SunOS 4 box? Go home and backport the behemoth? Any one of my sysadmins -- who have never used either of those OSs but know shell -- could solve that problem in five minutes flat.

Re:Even that's wrong / Ich bin Grammatiknazi

[FuzzyFox]

Hear hear!

I am really tired of helping someone over their shoulder as they type, and I tell them "you need to put parentheses around that part," and they ask "What kind?" Or worse yet, they just blithely put curly braces around the text without even asking.

There aren't different kinds of parentheses! There is only one kind! The round kind, found on your shifted number keys! The others are called brackets, or braces!

If I tell you to put braces around it, or brackets, you should know which one is meant!

Preview is my friend!

[The Monster]

clearly there wouldn't be a arc.tar.gz.Z file

So what about filenames with spaces

[Marrow]

Though I loath filenames with embedded spaces, his "good practices" fail to even mention the problem.
And I believe his xargs examples will not work with them.

And this:

ls -1 | xargs file
December_Report.pdf: PDF document, version 1.3

The file command will explode with the long output of ls. WTF.

The count on grep could work though.

Best practices:

Use directories; dont keep too much in one directory.
When copying to a directory: cp -av file1 file2 file3 file4 directory/. # the trailing path is a good habit

Use small shell scripts to get work done on the command line:

$ for i in images/*
> do
> convert -resize 640x480 $i 640/$i
> done

Re:So what about filenames with spaces

[Marrow]

sigh, well

the 640 directory should have an images directory
in it too.

Re:So what about filenames with spaces

[canuck57]

Though I loath filenames with embedded spaces, his "good practices" fail to even mention the problem. And I believe his xargs examples will not work with them.

This one should.

ls *\ * | xargs -l1 -i echo 'mv "{}" `echo {} | tr [\ ] [_]`' | sh

Have fun, when some Windoze user dumps all that garbage spaced files I fix it. The key is -l1 -- process by line not by spaces.

Re:So what about filenames with spaces

[pthisis]

Valid point, but that still breaks on files with newlines in the filename. Use a combination of find's -print0 and xargs' -0 option to deal with all filenames safely (and no, null isn't a valid filename character so there's no problem there).

Re:So what about filenames with spaces

[sparcnut]

Notice that he typed ls -1, not ls -l (ls -ONE, not ls -ELL). Big difference there, although the default for ls when outputting to a pipe is already -1.

Piping cats?

[jrobinson5]

Piping cats? Is that like kitten huffing?

In ZSH it's easy

[Anonymous Coward]

$ cat ~/bin/lowercase.zsh
#!/bin/zsh
for i in $@ ; do
        mv $i $i:l
done
$ ls
TEST1
tEsT2
test3
$ lowercase.zsh *
$ ls
test1
test2
test3

you *nix people are assholes

[Anonymous Coward]

*nix doesn't incluse linux, solaris, aix, a/ux, and many more. drop it.

cat file|grep regex|awk '{print $1}'

[mediis]

finally someone has said what has annoyed the living hell out of me for years. And I've been seeing it in multiple professional articles: remove cat. How many times to you type "cat file|grep regx" for a regex and when you can just run "grep regex file".And needless to say "awk '/regex/ {print $1}' file" works wonders as well. I think there is a good point to his article... there more than one way to skin a cat, just don't make a habit of it.

Re:cat file|grep regex|awk '{print $1}'

[kjamez]

whereas i typically use cat foo | grep that

doing something like:

> grep function\ thisFunction *.php

in a directory of source files will print the filename of the file where the grep matched, allowing you to know where the function or whatever is.

> cat * | grep function\ thisFunction

just returns the lines ...

Re:cat file|grep regex|awk '{print $1}'

[nuzak]

> How many times to you type "cat file|grep regx" for a regex and when you can just run "grep regex file

Because it's much easier to replace with a line edit or history replace when I later want to go back and replace 'cat' with something else, like oh, netcat or wget. It's amazing how bent out of shape some unix geeks will get about a spurious cat. If I was that obsessed with speed, I wouldn't be using a shell script in the first place.

That list is bull.

[taskiss]

It reads like "Hey! I'm so uber cool and I know a trick and you can be cool too if you act like me!"

which is utter crap.

The biggest thing to do to adopt "good" habits is to use syntax that'll work on every version of every flavor of UNIX. As the author of the article doesn't stress that, he's not a good person to use as an example.

Re:Yeah, parent post is written in sweat and blood

[multipartmixed]

I think you and I are from the same school of sysadmin-dom. But you'd never catch me using mc. Wuss!

Here is a common pattern I use:

a="`echo filepattern`"
echo $a
ls -lrt $a | head
ls -lrt $a | tail
export a
su
echo $a
rm $a
^D

This is quick to type and accurate for certain VERY sensitive directories where
a) errors are unforgiveable
b) MANY files exist
c) Files DO NOT contain spaces

> % for i in foo bar baz; do echo somecommand $i; done

for i in foo bar baz; do echo somecommand "$i"; done

Quotes make this quite a bit safer, in case foo, bar, or baz have space in them. And don't forget this construct:

find /dir filepatter\* | while read file; do echo "$file"; done

The backslash, BTW, is more important that most junior sysadmins think. But it is only used if something in your local directory matches filepatter*....

Re:Yeah, parent post is written in sweat and blood

[TheRaven64]

The good old 'rm -rf * .o' joke is so well known If you use zsh, you can just hit tab a couple of times, and it will give you a list of options; hit control-c to return to the wildcard form. I do, however, tend to just use echo {pattern} first and then replace echo with rm once I've checked it is sensible...

Doesn't matter.

[Junta]

I mean, they're both the same if you look at them on paper-wise

You must not have read the article...

[Junta]

Don't use semicolons, use &&.

Actually, this case is a very good example of why && are better as a default over ; to delimit commands on a single line. If you were in /, wanted to delete /tmp/root, and you typed cd /tm;rm -rf root. You'd be in trouble, but the && would cause the failed cd to not hose things up.

Still, my advice in general would be to avoid stringing commands just for the sake of being clever and run each command serially. More can go wrong than the return code will indicate that output will.

I always use 2 or more args for grep

[benhocking]

E.g.,

grep someFn `find . -name "*.cpp"`

Personally, I was suprised he didn't mention the backtick. Now, that's useful. (Although they can be annoying while camping.)

Re:I always use 2 or more args for grep

[mOdQuArK!]

I prefer to use the xargs version of that line where I can get away with it:

      find . -name *.cpp -print | xargs grep someFn

especially if find reports a LOT of file names,
since the backtick syntax can often run afoul of
any command-line length limits that the shell
might have after it has been expanded.

As far as I know, xargs doesn't have any such
limit (other than virtual memory) when it is
constructing the command-line that it is going
to execute.

Re:I always use 2 or more args for grep

[McDutchie]

As far as I know, xargs doesn't have any such limit (other than virtual memory) when it is constructing the command-line that it is going to execute.

Actually, if the size of the arguments exceeds the maximum command line length, xargs splits them into multiple command lines and executes the utility once for each of them. Depending on what utility you're invoking, that can lead to some interesting unexpected effects if you're not aware of that.

Re:I always use 2 or more args for grep

[grumbel]

If you want it solid, you'd better insert some -print0 -0 in there, else it will barf at the first encounter of filename with some 'funny' characters such as a newline:

  find . -name "*.cpp" -print0 | xargs -0 grep someFn

The most interesting thing however is when you want to do more with a filename, so that you can't get away with a single xargs line, then you can use (the -r -d $'\0' makes it save against 'funny' characters again, $'' is worth a lookup in the bash manual as well, can come in very handy at times):

  find . -name "*.cpp" -print0 | while read -r -d $'\0' filename;
  do
    echo "$filename";
  done

Re:I always use 2 or more args for grep

[mOdQuArK!]

I usually do a little reconnaissance on the files that I'm going to be searching through (I don't want to deal with some joker naming a file "-rf" or something like that), but I had forgotten about those options - thanks.

Re:I always use 2 or more args for grep

[grumbel]

### I don't want to deal with some joker naming a file "-rf" or something like that

'--' helps with that on GNU tools, every argument after '--' is handled as non-option argument (i.e. file in most cases):

$ rm -- -rf
rm: cannot remove `-rf': No such file or directory

Learn shell scripting

[benhocking]

The better I have learned how to write scripts in bash, the easier it has become for me to quickly write a "script" at the command line. I know, you can do this with Python, etc., but for some jobs it's just easier and quicker to use bash. The more tools you have, the quicker you can solve your problems.

Also, I believe that many default distros don't have Python in them. So, if you're on a strange computer you can't count on it being there. Perl or bash, however, you'll almost always find. (This is the same reason I've tried to improve my vi skills even though I'm an emacs fan. I know, I know. Yes, I said emacs.)

Suggestions

[Anonymous Coward]

Would it be considered bad form to send this to my boss?

My top 10 list for Unix Admins

[Anonymous Coward]

1. Learn about man & google. man is your friend, google is his big brother.

2. Learn vi. It may not be the best editor, but it is everywhere.

3. Learn your shell & shell tricks. All should have history & script. Up arrow or escape - k? Use & and ; when warranted: "cp big-file /somewhere &" can save you time!

4. Don't be root unless you need to. Mistakes happen. Being root makes a mistake's potential impact larger.

5. Be very careful with rm -rf. Watch out for "rm -rf /$SOMEPATH"... are you sure that $SOMEPATH is defined?

6. Before editting a file, keep an original. Before editting a config file, cp foo_config foo_config.orig.20061219. Same goes for directories du -k dir ... cp -r dir dir.orig.my_name. Alternately, if you have the file in source control, use it.

7. Be smart, clear & lazy when solving problems. Solutions often get attacked with brute-force ignorance instead of elegance. Be clear - exactly what problem are you trying to solve? Be lazy: stop when you have solved the problem. Also, If you can change the problem statement slightly and achieve the same results with a simpler solution, do so.

8. Expect the unexpected. When writing scripts, be defensive - check error codes, verify that variables are set and have sane values, etc...

9. Simplicity is sometimes more elegant than efficiency. If it looks like line noise and you wrote it, split it up. BAR=${${FOO:-$ZED}-y} is not as clear as using a nested if/then/else.

10. Save values or cut & paste, do not retype. Retyping is slow & error prone.

Not very useful at all

[TheLink]

They seem to just be tips for being "clever".

You don't need clever for Unix commandline. You need paranoid. Make backups but work so you won't need them.

Typing backslash to have extra long command lines? Don't you guys make typos?

Here are my tips:

Don't type complex commandlines into the shell directly. Copy them to a text editor first (that's what the GUI is for ;) ), edit, check, then make sure you know how to copy and paste stuff correctly ;). Only then paste the commandline to the shell... When the commandline starts to look very complex, you may wish to consider writing a script instead.

Consider that many keyboards put the backslash or pipe rather close to the Enter key, this means if you slip (or someone nudges you) you could prematurely terminate the commandline. Depending on how you do things, this could be not a big problem or it could be a catastrophe ( e.g. you end up running an SQL UPDATE without a where clause ;) ). If you are typing the command to a text editor first, this is less likely to be a problem.

The tip about not using cat is stupid. If you use cat, it is easier to reedit the command line to repipe the file output to some other command instead. e.g. cat file | head, then up arrow replace head with grep or strings | head, or whatever.

To those who use "<", using "<" to read files is asking for trouble, because "<" and ">" are next to each other and you could be very unhappy if you used ">" instead of "<". So make it a habit of NOT using "<" at all.

Basically, when you use the Unix (or SQL) commandline, always be paranoid. Yes, don't move the tar file before untarring, but don't do it the suggested way either. Change to the directory you want to unpack the file to first, then ls -al to see if it's really the right directory ;). Then only use filename completion for tar -xvf <path to tar file> to untar. This way if you screw up you are less likely to mess up the wrong directories.

When you are using tar, always keep in mind that with the US and UK keyboard layouts the x and the c keys are next to each other and mean rather different things ;).

Try not to do rm -rf <long path name>. cd to the directory first. Then do the rm -rf with filename completion. That way the dreaded rm -rf / is less likely to happen.

If you are using bash and you like to use ctrl-r, don't assume you'll get the command you are thinking of. Look and confirm before you press enter ;).

Re:Not very useful at all

[cas2000]

The tip about not using cat is stupid. If you use cat, it is easier to reedit the command line to repipe the file output to some other command instead. e.g. cat file | head, then up arrow replace head with grep or strings | head, or whatever.

yes, this is really useful.

i do it all the time when i have a very large file, and i'm e.g. iteratively developing the exact regular expression i need.

e.g.

start off with

head filename | grep "reallycomplexbutnotperfectpattern"

and keep editing the regexp until it gives me exactly what i want, nothing more and nothing less.

then, when i've got the pattern right, just hit up arrow and change "head" to "cat". or a sequence of cat, grep, sed, awk, whatever in the pipeline if you need to transform the input before the final grep.

similarly, you can insert "| head |" into a pipeline to run a quick test before you do the real thing.

10 Reasons Unix Won't Be Adopted

[justthinkit]

I love the command line. Really. But not the Unix command line. Only a computer could ever love C, or a C shell. Humans are forever tortured by both.

Exhibit A, the tar command. Never was so inscrutable and impractical a monstrosity presented to so many grovelling supplicants. No, it does not even compress the archive. Why? Well you see unix must be modular, even when it means that every bullet gets hand-packed before being inserted in the musket. If a finger or two is lost due to a blocked chamber, so be it.

Unix, the triumph of difficulty over usability.

Re:10 Reasons Unix Won't Be Adopted

[Anonymous Coward]

Windows: When all you have is a hammer, every problem starts to look like an OK Button.

Re:Yeah, parent post is written in sweat and blood

[lahi]

Also, I often use constructs like
% for i in foo bar baz; do echo somecommand $i; done
before removing the 'echo' part.

Me too. Except I'm lazy (used to code Perl a lot, now mostly Korn shell) so I just type ^P^E|sh^M
I know it only saves a keystroke or two. However it also makes it easy to type |sudo sh instead, or
add another grep to filter out unwanted commands, etc. I guess that counts as a habit.

-Lasse

ObRMS pedantry

[Sloppy]

Maybe AIX has an exceedingly slow ''cat'', Linux does not.

Linux doesn't have a cat. GNU does, though.

Some good, some bad, some plain pointless.

[Junta]

I'm evaluating the tips based on them being prescriptions for things to do in interactive shell behavior, since that seems to be the theme. Writing scripts changes the situation to make some tips valuable. My number one tip as a response to these is don't try to be too clever (particularly when the biggest benefit of the approach is to say 'look how clever it is!'). Maybe it's because I don't work in a vacuum and all too many times have been called in to clean up where an administrator tried to do something too complicated for their understanding.

mkdir -p is a convenience people should be aware of, but telling people to start getting overly creative with the shell expansion behavior is asking for mistakes/trouble. Also, having a mkdirhier script in case the example isn't supported on all shells is an indication that you shouldn't get overly cozy if you are going to be dealing with a lot of different systems/users with different default shells. The amount of time a lot of people take to figure out the 'clever' way in terms of how to phrase the expansion so the shell will expand it right is often longer than just typing the two lines more that the less thought takes. Not saying this isn't useful, but in my experience too many people mess things up too frequently or take too long to think up the expressions to tell them trying to be clever ends up taking more time than they think they are saving.

Change the path instead of the archive is not that dire to do normally, but if you avoid it, to me it's just easier to be in the target directly and use full path to the archive.

On combining commands, I second that ; can be dangerous and && as a default will make the chain more ready to break, but again I say not trying to be so clever as to put all you want on one line. Some things go wrong that aren't reflected in return codes, doing it one at a time let's you think of those. True, though, that the && never assume the first command works, while your fingers may keep moving and hit enter on next command before your brain realizes the command failed, so && may have merit, but then again taking your time may have more merit.

On the quotation thing, true enough, you must understand how quoting works to do remotely complex things, particularly nested circumstances (i.e. ssh to a system to run a command, where the output will be parsed by two shells.)

On the breaking up long lines thing, in a shell script it may be more necessary, but on an interactive command line it could also indicate you are trying too hard to do things in one chunk. I admit sometimes it does get too wide, but particularly less experienced admins should consider if there were a simpler way to do it in smaller chunks they won't screw up.

Grouping commands is important to know, and harmless (better than repeating the same pipe over and over and more powerful).

I will say xargs is way way over-rated. Too many people, particularly dealing with directory trees containing spaces, get into trouble piping the output of find into anything when IFS causes something like "/tmp/Monthly Report" to be parsed as two different files. find has a competent filtering mechanism (-type, -iname, -name, etc...) and it's own -exec. find is well aware of the state of each file. You could assign IFS to try to avoid it, but using find's built-ins where possible alleviates it.

When you are talking about interactive shell operation, picking the .01s instead of the .09s operation is a bad example. He could have set up a
much larger demonstration that would have been useful, but this just makes people mock the example. In any event, this seems like an okay thing to convey, but I dunno if it would've made my top 10.

Probably a more valid point about using awk, and a common trap I do see people stuck in.

On piping cat, that seems like more an annoyance than anything constructive. Some people use the cat | grep construct because it is so unambiguou

I don't think I ever use xargs

[Sloppy]

Why

foo | xargs bar

when I can

bar `foo`

?

Re:I don't think I ever use xargs

[Anonymous Coward]

xargs can split up the args to execute bar multiple times.
bar `foo` can fail if `foo` happens to make the argument list too large.

Also use find -print0 | xargs -0 instead of find -exec if you prefer not forking a jillion times.

What!?

[Xenographic]

I thought that all-natural kitten huffing was PETA approved?
Or have they changed that recently? :]

Re:What!?

[Pink Tinkletini]

Good Christ, I remember Uncyclopedia. I remember it as the unfunniest so-called humor site I'd ever had the misfortune to visit. It called to mind an autistic Carson Daly, if Carson Daly were two hundred pounds overweight and covered in grease.

For God's sake, please, don't encourage others to visit and risk entrapment in that land of dweebish comedic failure. One shit-riddled wiki is enough.

Re:Old News

[Anonymous Coward]

No, it is not "the best thing on Windows that is close to UNIX", both Interix/Microsoft SFU (freeware) and AT&T/Bell Labs UWIN (open source) are much, much, MUCH better.

We work in windows, not on teletypes

[DavidHumus]

I continue to be amazed at how many people do command-line work in non-scrollable windows and don't run the command-line within an editor like emacs.

Using a non-scrollable window is simply idiotic: what are you saving - the few thousand bytes of (potentially useful) session history?

Using a scrollable window alone is only a little dumb - you've vaulted yourself into the era of teletype printing terminals: if you need to look at a previous part of the session, you can scroll backwards and search for it using your eyeballs, just like manually searching through a long piece of teletype paper. Why run in an editor which has search commands built-in? Think of the microseconds of computer time you save by wasting seconds and minutes or your own time.

Re:We work in windows, not on teletypes

[Anonymous Coward]

I continue to be amazed at how many Unix installations hand me a terminal in a non-scrollable window, or a window with finite history, by default. By the time I've realized the terminal app is maldesigned or malconfigured it's too late.

Piping cats

[this great guy]

People who argue that piping a single file via cat is the best method are wrong. The following method has all of the advantages you cite, but is also shorter to type, uses less system resources (no cat process, no pipe(2) object), and doesn't require you to "rething the input method" in case you want to change the grep command:

    $ <file.txt grep foobar

Few people know that input-redirection can be established before the command name :)

comments

[cas2000]

1. first impressions: the article is mostly simple, obvious stuff, i'm surprised to see it on IBM's developerworks site.

2. re: xargs end-of-file warning. as of version 4.2.9, GNU xargs defaults to not having an end-of-file string - worth knowing about, but not really a problem.

3. a far more common "gotcha" with xargs is dealing with space characters (and other characters with special meaning to the shell) in file names. this is especially common if you are working on directories containing HTML (and other) files uploaded by Mac & Windows users. to deal with it, use find's "-print0" arg and xargs' "-0" arg, for null-terminated input.

e.g. to set permissions to 644 on all the html files in a user's public_html directory:

find /home/USER/public_html -name "*.htm*" -print0 | xargs -0 chmod 644

most of the GNU utils support null-termination for input & output - so if you need to run grep or whatever on the output of find before passing it to xargs, you can.
e.g. to find all copies of Matt Wright's scripts such as FormMail.pl in all users' web sites and disable them:

find /home/*/public_html /home/*/cgi-bin -name "*.cgi" -o -name "*.pl" -print0 | xargs -0 grep -Zzli "matt.*wright" | xargs -0 chmod a-x

make your users use secure alternatives instead - such as the NMS versions of the same scripts. even Matt's Script Archive recommends the use of the NMS versions instead of his own, these days.

"Stop piping cats"

[IntergalacticWalrus]

Is that like Kitten Huffing?

Use of backquote

[darkonc]

Xargs gets messy if you want the filenames to be in the middle. It's also quite a waste when you know that you're not going to get more than 10,000 values... eg:

grep / ` which someprog`

Then again, there's also the more recent replacement for the backquote -- the $( structure ).. Because it has a syntactically distinct start and end indicator, it's far easier to nest them (when the need arises).

Bad unix habid

[j_sp_r]

I always did

for test in $(find .)
do if cat $test 2>/dev/null | grep togrep; then echo $test; fi
done

But you can also do

grep -R test .

less == more

On what planet is gzip 'portable' across unices?

[The Monster]

You should be using gzcat, not zcat, anyhow. zcat is only portably able to be compress -d.

Unfortunately, in the context of Unix, there's nothing 'portable' about using gzipped files. At work, I spend most of my time working with SCO OpenServer 5.0.x (I know), a little bit with AIX 4-5, and rarely get to do anything with Linux. The first two don't ship with gzip, leaving as the common denominator of all three the LZW of compress . On all three of those flavors of *nix, zcat < filename handles those files with no problems. Therefore, when I build a tarball of shell scripts designed to run portably, that's the format I'm using to package it, and the scripts are written to the LCD with some case `uname`... constructs where necessary.

Re:On what planet is gzip 'portable' across unices

[darkonc]

gzip is portable to most 'modern' unides. Even many of those systems that don't have it as a default distribute the GNU tools as a separate (but free) add-on.
As an absolute worst case, you can always compile and install the GNU tools (in your home 'bin' directory, if necessary), including, possibly, gcc.

If you're really worried about portability, you should also remember that -C is also a 'relatively' recent add-on to tar, so you'd be better off with ' (cd a/b/c ; tar -xf - ) < somefile '

However, once you have gnu tar and gzip installed, you can then go 'tar -C a/b/c -xzf somefile.gz'

If I wanted to upload binaries...

[The Monster]

However, once you have gnu tar and gzip installed

But that defeats the purpose of shell commands that use binaries that are part of a stock *nix installation. An awful lot of people who have an account on a Unix box can't just throw gcc on it and compile their own binaries. As hard as it is to believe, some administrators actually frown on such things.

I'm usually writing shell scripts that invoke tools like sed/awk/grep that are built into all the flavors of Unix I have to work with. I often have to access a machine over a 9600 bps modem connection.

My most popular utility tarball weighs in at 44K (LZW) compressed, or 127K uncompressed. If I had to upload a 50K gzip binary and a 184K gnu tar just to be able to uncompress my tarball, I'd be more than doubling the time it takes to do the install. The improved compression from gzip (about 7% compared to LZH under SCO compress -H , or 25% compared to LZW from SCO/AIX compress would take a long time to pay back.

A recent project is working with much larger tarballs, so it might be a good idea to pursue getting GNU tools out to the point that they become the norm. If my employer would install them on all systems we have to support, it would be a different situation; I'd use all GNU goodness and love every minute of it. I'd also probably go back to writing utilities in perl, which is what I used to do at a prior job where I had only a handful of local systems to support instead of thousands across the whole country. But since that isn't likely to happen any time soon, I'm sticking with what is truly portable, as POSIXly correct as I can, with the absolute minimum of quirk adjustments.

Re:If I wanted to upload binaries...

[lahi]

Right. So 10 *really* good Unix "habits" would be:
1. Never use csh or any derivative thereof.
2. Know the portable behaviour of your Unix tools.
3. Learn to use ed, one day you'll be glad you did. You can also use ed and ex from scripts or from a command.
4. A shell command is a small program. If you are unsure about a command, test it first, like you would any program.
5. Learn to use the standard shell on your system.
6. Learn useful nonstandard extensions of utilities, but use them with care.
7. Never rely on an extension to the point that you forget how to do it portably. The definition of "portably" is up to you.
8. Learn to use csh enough that you can make do in an emergency, and learn *why* you shouldn't use it.
9. If your standard shell is Bash, learn Korn too. And vice versa. Learn both, how they differ, and how they differ form your standard shell.
10. Sometimes a real C program or a script in a different language is better than using shell.

-Lasse

Re:If I wanted to upload binaries...

[multipartmixed]

Quick point of order, there is no requirement to use GNU tar with gzip!

gzcat filename | tar -xvf -

will work exactly the same as

tar -zxvf filename

That said, point taken -- my gzcat (Solaris 9 sparc stripped no non-standard DSOs) weighs in at a hefty 63K. I could probably get it down 30% from there with TFS and a C compiler by throwing out some features, but it would take a couple of days, and really, there would be no point.

Re:If I wanted to upload binaries...

[darkonc]

Have you tried stripping it?

bad ibm no cookie

[illuminatedwax]

Great, IBM, way to ignore the dreaded "xargs" security bug! Seriously, IBM notices some kind of obscure danger about underscores, but completely ignores the fact that xargs separates arguments by newlines??

Let's say I'm a sysadmin and I'm running as root, trying to remove all the files in the /tmp directory by a certain user for some reason:
find /tmp -user 1001 | xargs rm

User 1001 has a directory in /tmp called "haxor\n". Inside there he puts another directory "etc" and inside there he puts a file called "passwd."

Can you guess what happens?
find prints: /tmp/tmp43cc91 /tmp/haxor /tmp/haxor /etc/passwd

xargs sees: ["/tmp/tmp43cc91","/tmp/haxor","","/tmp/haxor","/e tc/passwd"]
Oops!! You just hosed your system!

The correct way to use xargs is to use the -0 switch, which will separate the input by null characters, which cannot appear in filenames. find has a handy -print0 option which will output the correct output:

find /tmp -user 1001 -print0 | xargs -0 rm

And your system is safe.

Re:bad ibm no cookie

[OzPixel]

The correct way to use xargs is to use the -0 switch

... which only applies if you have a GNU-derived find/xargs/cpio. Not very helpful if you're stuck on (e.g.) HP-UX a couple of versions behind the current one, and with no chance of installing a suite of GNU tools. Welcome to the wonderful world of Unix portability ...

Re:bad ibm no cookie

[illuminatedwax]

...welcome to the world of the 1990s...

Huh?

[fm6]

Ie there is NOTHING bad about piping cats. While you might indeed get a ~30% performance increase if you skip the cat, the complexity increases.

Say what? You actually think that

grep 'pattern' file1 file2

is more complex than

cat file1 file2 | grep 'pattern'

Your logic is not like our Earth logic!

Incorrect benchmarks!

[Vince]

Does anybody else notice these benchmarks are flawed? For an article discussing the shell, we should know that in this first benchmark, time is only counting the execution time of grep, and not wc, and is thus undercounting how much CPU time is actually used. How about a neat shell trick to correctly run that benchmark?

> ~ $ time grep and tmp/a/longfile.txt | wc -l
> 2811
>
> real 0m0.097s
> user 0m0.006s
> sys 0m0.032s
> ~ $ time grep -c and tmp/a/longfile.txt
> 2811
>
> real 0m0.013s
> user 0m0.006s
> sys 0m0.005s
> ~ $

Exactly

[h2g2bob]

I know it's not proprietory 'cos Debian has it

$ bash --version
GNU bash, version 3.1.17(1)-release (i486-pc-linux-gnu)
Copyright (C) 2005 Free Software Foundation, Inc.

some UNIX habits

[GodLived]

Clearly this post is too old for anyone to care about, so I'll do it just to mark the article.

While I agree with /. reader comments, I wanted to point out some other problems with the article:

1. Avoid the use of || and && on the command line. They are squirrely between different systems, and as some other reader pointed out, you should just run a command and see what happens. You're not in a script. Also, I am afraid of A || B && C - what happens? A and C, B and C, or just A? Operator precedence matters.
2. Be careful about tars. I spent 2 years managing open source, and found that while most authors correctly name a root of the extraction ./my-package-1.2.3, some just put all their files in the current directory. You need to "jail" your tars carefully.
3. Why not do "grep 'Dec '" for that grep example, instead of invoking awk? Point is, look at your data, then decide the simplest way to handle it.
4. Piping cats is a loopy comment. Nobody cares. Also, the wc -l saving 0.84 seconds is loopy, as well.
5. If you find yourself using many backslashes to continue a line, then you should be writing a script. Actually, the best thing you can do is open a notepad-like editor and build your command line in that, then paste it to your shell window.

Now, for my tips:

1. Never depend on a variable being set when doing a remove. That is, NEVER do "rm -rf $SOMEPATH/*" because if SOMEPATH is not set, guess what gets removed? Uh huh.
2. Always back up your MBR before making serious changes to your filesystem. Do this using "dd if=/dev/hda of=mbr.bin bs=1 count=512" and ideally put it on a USB stick. It's free, takes little time, and you can always restore your MBR from a bootable Linux distro such as SysRescueCD (my fav) if you really hose things up.
3. Quit using CSH, like some other poster said. Use bash. It's everywhere.
4. Pipe error output into a file, not just stdout. Somecommand >output.txt 2>&1 will write everything including errors into output.txt. If you ignore the 2>&1 (which says, duplicate stdout and route stderr into it), you only get the normal output.

Hope this helps.

Mod Parent WAY up!!!

[multipartmixed]

I'm a UNIX guy who's been using UNIX and derivatives for something like 2/3 of my life.

This post is bang on the money, and, in my estimation, approximately 3113.7% more useful than TFA.

Re:On what planet is gzip 'portable' across unices

[multipartmixed]

> Unfortunately, in the context of Unix, there's nothing 'portable' about using gzipped files.

You're absolutely correct, but missing the mark.

The moronic FA author was trying to portably gunzip files. Given those requirements, gzcat is a reasonable prerequisite. Thus providing gzcat with arguments which would be portable to zcat is unnecessary, as zcat could not complete the task required in the first place.

BTW, I don't know what kind of machines you admin, but I have the luxury of administering primarily sparc/solaris boxen these days. This means that I can ship a Solaris 2.5.1 binary version of GNU tar and gzip with my install packages, and it will run on any of my machines, assuming they are not more than 11 years old. Sun ABI compatibility is quite good. I could even ship 2.4 binaries without issue, extending my box-age to 12 or 13 years, but that seems kind of pointless since all the boxes are less than 5 years old, with the exception of the workstation I use to build Solaris 2.5.1 binaries (an ultra 5 running 2.5.1 hardware 11-97).

I suppose I'm going to have issues at some point the future, though, as we are considering moving to Opteron machines. (If they would rip out the BMC and replace it with OpenBoot, we would have moved long ago). In that case, I'll probably have to snag gzip and tar from an old Solaris 7/x86 box I still have around.

bash

[ThePhilips]

Get a real shell, man!!

so ... be polite ... OR !

[freaker_TuC]

I replace you with a small shell script