Essential UNIX Tricks and Tools?

Chris Lesner asks: "What handy UNIX tricks/tools do you use everyday? I'm asking for stuff that amazes your friends and makes you wonder how they use UNIX w/o them. Some simple examples include: job control (with fg, bg/&, jobs, Ctrl-Z); moving login sessions between machines with Screen for vt100 and VNC for X11 and using screen and VNC to share login session b/w users for demos etc.; using find, xargs -i and echo to build command strings which after inspection can be piped back though bash e.g. `find . -type f | xargs -i{} echo "cp {} {}.bak" | bash` I'm asking b/c my source for this kind of information has dried up as my UNIX skills have matured. I'm guessing other Slashdot readers have the same problem. By the way, if you think the examples I give are lame I challenge you to better them!"

Comments

Bash Completion Project

[Bouncings]

One of the most useful gadgets I use is the bash completion project. It's a handy-dandy tool where tab-completion does more, oh, so much more than filenames. When I do a Debian apt-get install python-, I get a list of Debian packages to install starting with python-

There's more fun too. It completes tons of crazy stuff. I'd check it out.

Re:dusort

[Phexro]

But why bother with the awk, when you can just do:

$ du -s /path/to/wherever/* | sort -rn | head -10

to get the top ten hogs in /path/to/wherever?

Also, sometimes there are some big files, and you are only interested in the directories full of crap:

$ du -s `find /path/to/wherever/* -type d` | sort -rn | head -10

While we're on the subject, you can use this handy-dandy snippet to find the disk usage of one user in any part of the filesystem:

$ find /path -type f -user jsmith -exec ls -l {} \; 2>/dev/null \
> | awk '{ sum += $5} END { print sum }'

command line tricks

[caca_phony]

Here is a trick for my favorite shell, es the Extensible Shell, a derivitive of plan9's rc shell. The es shell has the most logical set of syntax rules I have ever seen in a shell. It is also the only shell where I have figured out how to do automated file renaming (very handy). Here is a transcript of a session as an example:

;; touch 123
;; touch 124
;; touch 125
;; touch 126
;; touch 127
;; touch 128
;; touch 129
;; ls
123 124 125 126 127 128 129
;; echo <={~~ (*) 1*}
23 24 25 26 27 28 29
;; for (i = <={~~ (*) 1*}) {mv 1$i one$i}
;; ls
one23 one24 one25 one26 one27 one28 one29

this functionality is great for preventing name collisions when consolidating files from two directories into a single directory.

as some explanation <={...} is like typing the return value of the command inside the braces, ~~
returns the part of the second aruments matches that were expanded from the the first argument's glob.

Regular Expressions

[sam the lurker]

Regular expressions aren't so much either a trick or a tool exactly, but you can use them with all the "good" tools.

Get the book "Mastering Regular Expressions," by Jeffrey E. F. Friedl. http://www.oreilly.com/catalog/regex2/

Read it slowly, a couple of pages every day. I didn't understand much of what he was trying to say until I read the book the second time.

But why make up my own clever things to say... From http://www.oreilly.com/catalog/regex/desc.html "There can be certain subtle, but valuable, ways to think when you're using regular expressions, and these can be taught."

I find that books that teach you how to think about problems and solutions are few and far between, and books that do it well are almost impossible to find. This books is one of those.

Once learned regular expressions are one of those things that can profoundly effect the way you work. And once your there "you wonder how they [other people] use UNIX w/o them".

Re:Regular Expressions

[Permission Denied]

Here's a cute little perl script that's useful for developping other perl scripts:

#!/usr/bin/env perl
$st = "\033[7m";
$en = "\033[m";

print "Enter regex: ";
$re = <>;
chomp $re;
while () {
print "Enter string: ";
$s = <>;
chomp $s;
exit unless $s;
$s =~ s/$re/$st$&$en/g;
print $s . "\n";
}

This will highlight what parts of a string match a given regular expression. (I modelled this script on something I found in a Ruby tutorial).

Also, for those learning regular expressions, I would highly recommend Introduction to Automota Theory, Languages and Computation by Hopcroft et al. This is a more mathematical/theoretical introduction to the subject. Regular expressions/automata are a very nice part of CS in that the basic theory is immediately applicable.

What I can think of right now:

[crucini]

1. Tcpflow - read contents of tcp traffic in real time. Great for watching browser/webserver interactions.
   
2. Netcat - connect Unix pipes to TCP sockets. Should have been part of Unix since the advent of TCP/IP. Great for rigging a temporary "server" to see if a client is connecting as advertised: nc -lp 80.
   
3. X Resources (as seen in ~/.Xdefaults) - you can make xterms really dark, even when running colored apps like mutt, with dark Xresources like: XTerm*color9: #690000 - man xterm for meanings of color0-15.
   xrdb -merge .darkXres to use.
   
4. Xmessage - useful in crontabs to remind you of periodic things - like remembering to go home. With the right params, it can take over the whole screen, which is hard to ignore.
   
5. perl -pi.bak -e's/chocolate/vanilla/g' *.recipe - change a bunch of files, leaving backups.
   

chmod, jot, grep, *, ls, less

[realdpk]

To make all of your files and directory world-readable with one command:

$ chmod -R go+rwX .

(the X is the key, you don't have to worry about executables vs. directories)

Need a list of numbers of letters or anything? Check out jot (on FreeBSD, maybe on others):

$ jot 3
1
2
3

$ jot -w %c 3 65
A
B
C

(This is especially useful with for loops - "for i in `jot 20`;do touch foo.$i;done" will generate foo.1 through foo.20. Extra hint: -w %02d will give you a leading zero on 0-9)

grep's -A, -B, and -C flags can be very useful. Using them, you can have grep display the lines immediately before and/or after a match.

If you have a lot of files in a directory, so many that "*" complains of the argument length, bash (and probably other shells) can let you get around this by changing your command to one using a for loop. It won't be as fast, but it won't churn for several minutes only to tell you it won't work. ;)

instead of:

$ rm *

do

$ for i in *;do rm $i;done

You can get a list of all files in your directory excluding . and .. with 'ls -A'.

One of the most annoying things about 'less', at least in Debian and probably other dists, is that it clears the screen when you exit. Ugh! You can fix this problem by setting PAGER='less -X' in your environment.