Essential UNIX Tricks and Tools?

Chris Lesner asks: "What handy UNIX tricks/tools do you use everyday? I'm asking for stuff that amazes your friends and makes you wonder how they use UNIX w/o them. Some simple examples include: job control (with fg, bg/&, jobs, Ctrl-Z); moving login sessions between machines with Screen for vt100 and VNC for X11 and using screen and VNC to share login session b/w users for demos etc.; using find, xargs -i and echo to build command strings which after inspection can be piped back though bash e.g. `find . -type f | xargs -i{} echo "cp {} {}.bak" | bash` I'm asking b/c my source for this kind of information has dried up as my UNIX skills have matured. I'm guessing other Slashdot readers have the same problem. By the way, if you think the examples I give are lame I challenge you to better them!"

Comments

Bash Completion Project

[Bouncings]

One of the most useful gadgets I use is the bash completion project. It's a handy-dandy tool where tab-completion does more, oh, so much more than filenames. When I do a Debian apt-get install python-, I get a list of Debian packages to install starting with python-

There's more fun too. It completes tons of crazy stuff. I'd check it out.

How about

[Hard_Code]

find . -type f | xargs -i{} echo "cp {} {}.bak" | bash

find . -type f -exec cp '{}' '{}'.bak ';'

Re:How about

[Zapman]

The slight problem with this (not in this case though) is you have the expense of forking and killing a lot of processes (equal to the number of files +1 for the find). On the mv or cp case, you can't (easily) get around it, but if you were to do:

find . -type f -exec chmod 644 {} \;

vs

find . -type f | xargs chmod 644

you'll find that the second runs amazingly faster, since it will group a lot of the commands together into 1.

I did 'cd /usr/bin ; time find . -type f -exec ls {} \;' and got:

find . -type f -exec ls {} \; 0.98s user 3.28s system 88% cpu 4.831 total

I did 'cd /usr/bin ; time find . -type f | xargs ls -1 ' and got:

find . -type f 0.00s user 0.02s system 31% cpu 0.063 total
xargs ls -1 0.03s user 0.04s system 74% cpu 0.094 total

That's a BIG difference, especially when you have a LOT of files.

(NOTE: if I did a bare 'ls' with the xargs, the output would be different due to the way xargs works (man xargs for more details). It may or may not make a difference depending on how you're using the output. If you're shleping the whole line at once, it could make a huge difference... if you're spliting the lines by $IFS or something, then it's probably alright.)

As for my contribution, I find these 3 find | xargs commands, wraped together in a script I call "makereadable" help me a LOT (for example, if you install from source, and the permissions get borked due to forgetting to set root's umask to 022):

find . -type d -print | xargs chmod 755
find . -type f -perm +0100 -print | xargs chmod go+x
find . -type f -perm +0200 -print | xargs chmod go+r

The first makes all directories 755. 99.99% of the time, that's what you want... just don't do it with /tmp (sticky bit will get wacked). The second finds all files that are executable by the user owner, and makes them executable to others too. The third finds what is readable to the user, and makes it readable too.

If you want to do this with write permissions, you're probably doing something stupid you will regret later. Figuring out the command to do this is then left as an exercise for the reader. :-)

some stupid shell tricks

[Permission Denied]

I use zsh on all my non-Linux machines and just use the default bash on all my Linux boxes. First of all, I have one common .zshrc and .bashrc which all my machines get. The last line sources .zsh-local or .bash-local, where I keep all my machine-specific settings. This way, when I think of something clever to put in .zshrc, I can distribute it to all my machines at once.

Here are some stupid shell tricks I've found moderately useful:

1. bash: HISTCONTROL=ignoreboth, zsh: setopt HIST_IGNORE_SPACE; This means when you type a command with a space in front of it, it doesn't go in your history. Useful if you do something that you don't want others to see (eg, xv ~/.pr0n/*.jpg).
2. The zsh FAQ has an entry which describes how to get your xterm title to describe all sorts of fun stuff. Here's a bash equivalent:

   # see the document /xc/doc/hardcopy/xterm/ctlseqs.PS.gz
   # on any XFree86 mirror; the only other cool thing you
   # can do is change the font with "ESC ] 5 0 ; fontname BEL"
   xtitle() {
   echo -ne "\033]2;$*\007" > /dev/tty
   }
   if [ "$TERM" = "xterm" -o "$TERM" = "xterm-color" ] ; then
   PROMPT_COMMAND="echo -ne '\017'"
   fi
   if [ "$TERM" = "xterm" \
   -o "$TERM" = "xterm-color" \
   -o "$TERM" = "rxvt" \
   -o "$TERM" = "cygwin" ] ; then
   settitle() {
   xtitle "`date +'%H:%M:%S'` - ${HOSTESS}: $PWD"
   }
   PROMPT_COMMAND="settitle ; $PROMPT_COMMAND"
   fi

3. PATH management:

   pathdel() {
   PATH=`echo $PATH | sed -e "s/\(.*\)\(\:\)$1\(\:\)\(.*\)/\1:\4/g" \
   -e "s/^$1://" -e "s/:$1\$//"`
   }
   pathadd() {
   if echo $PATH | fgrep "$1" > /dev/null 2>&1 ; then
   :
   else
   if [ -d $1 ] ; then
   case $2 in
   "append") PATH=$PATH:$1 ;;
   "prepend") PATH=$1:$PATH ;;
   *) echo fixme ;;
   esac
   fi
   fi
   }
   
   pathdel '\.'
   
   if [ "$UID" -eq "0" ] ; then
   pathadd /sbin "append"
   pathadd /usr/sbin "append"
   pathadd /usr/local/sbin "append"
   for i in /usr/local/*/sbin ; do pathadd $i "append" ; done
   for i in /opt/*/sbin ; do pathadd $i "append" ; done
   fi

4. This next one is absolutely necessary on any Linux machine:

   alias kilmoz='killall -9 netscape ; rm -f ~/.netscape/lock'

5. Recursively do something for each subdirectory:

   dirs() {
   # follows symlinks, prints dot-dirs
   ls -A $@ | while read i ; do [ -d $i ] && echo $i ; done
   }
   recurse() { # DFS
   dirs | while read DIR ; do
   $@
   cd $DIR
   recurse $@
   cd ..
   done
   }

Here's a dumb shell script that I call "randsort" - it prints out lines from stdin in a random order:

#!/bin/sh
randsort() {
perl -e 'srand(time() ^ ($$ + ($$ << 15)));
print sort {rand 10 <=> rand 10} <STDIN>;'
}
randsort

This is useful 'cause you can do something like this:

xv `ls -t ~/.pr0n/*.jpg | head -200 | randsort`

and this randomization, for some reason, "feels" better than xv's -random option. It also has a number of "legitimate" uses. For instance:

playm3u() {
randsort < $1 | while read i ; do mpg123 "$i" ; done
# heh - 'mpg123 -z --list "$i"' does same thing nowadays
}

Anyway, I got lots of stupid little crap like this. I try not to get too far into the customization stuff, as it's definitely a timesink and it's very important to be able to use a "standard" unix setup. For example, my .emacs is circa 80 KB (I'm big into Lisp), but I'm extremely proficient with vi, because there are lots of situations where emacs is not available, or vi is better-suited to the job.

The lameness filter is such a piece of horseshit. I mean, are the editors deliberately trying to prevent us from discussing code and technical matters? You'll need to re-indent all the snippets above because of this.

Lameness filter fodder:

C este un limbaj de programare cu scop general ale carui caracteristici sint economia de expresie, structuri moderne de control al fluxului si de date, precum si un set bogat de operatori. C nu este un limbaj de nivel "foarte inalt", nici "mare", si nu este specializat vreunei arii particulare de aplicatii. Dar absenta in restrictii si generalitatea sa il fac mai convenabil si mai de efect pentru mai multe scopuri decit limbaje presupuse mai puternice. C a fost la inceput proiectat si implementat pe sistemul de operare UNIX pe DEC PDP11 de catre Dennis Ritchie. Sistemul de operare, compilatorul C si in mod esential, toate programele de aplicatii ale lui UNIX (inclusiv software-ul folosit pentru a pregati cartea aceasta) sint scrise in C. Compilatoare de C exista deasemenea si pe mai multe alte calculatoare, intre care IBM System/370 Honeywell 6000 si Interdata 8/32. C nu este legat de nici un hardware sau calculator anumit si e simplu de scris programe care se pot executa fara nici o modificare pe diferite calculatoare care au limbajul C implementat. Aceasta carte are drept scop sa-l ajute pe cititor sa invete sa programeze in C. Ea contine o initiere, pentru ca noii utilizatori sa poata incepe cit mai repede posibil, capitole separate pentru fiecare caracteristica majora, si un manual de referinta. Marea parte a textului nu se bazeaza atit pe expunerea de reguli si propozitii cit pe citirea, scrierea si revizuirea de exemple. In cea mai mare parte exemplele sint programe reale si sint complete si nu fragmente izolate. Toate exemplele au fost testate direct din text, care este intr-o forma citibila pe calculator. Pe linga faptul ca am aratat cum se utilizeaza efectiv limbajul, am incercat in plus, acolo unde era posibil, sa-l ilustram cu algoritmi utili si cu principii de bun stil in programare si proiectare sanatoasa. Aceasta carte nu este un manual introductiv de programare. Ea presupune anumite familiaritati cu conceptele. de baza din programare, ca variabile, instructiuni de asignare, bucle, functii. Cu toate acestea, un programator novice va fi in stare sa citeasca cartea si sa-si insuseasca limbajul, chiar daca ajutorul unui coleg cu experienta mai mare i-ar usura munca foarte mult. In experienta noastra, C s-a dovedit un limbaj placut, expresiv si adaptabil pentru o mare varietate de programe. Este usor de invatat si "se poarta bine" pe masura ce experienta in programare cu el creste. Speram ca aceasta carte va va ajuta sa-l folositi bine.

C este un limbaj de programare cu scop general ale carui caracteristici sint economia de expresie, structuri moderne de control al fluxului si de date, precum si un set bogat de operatori. C nu este un limbaj de nivel "foarte inalt", nici "mare", si nu este specializat vreunei arii particulare de aplicatii. Dar absenta in restrictii si generalitatea sa il fac mai convenabil si mai de efect pentru mai multe scopuri decit limbaje presupuse mai puternice. C a fost la inceput proiectat si implementat pe sistemul de operare UNIX pe DEC PDP11 de catre Dennis Ritchie. Sistemul de operare, compilatorul C si in mod esential, toate programele de aplicatii ale lui UNIX (inclusiv software-ul folosit pentru a pregati cartea aceasta) sint scrise in C. Compilatoare de C exista deasemenea si pe mai multe alte calculatoare, intre care IBM System/370 Honeywell 6000 si Interdata 8/32. C nu este legat de nici un hardware sau calculator anumit si e simplu de scris programe care se pot executa fara nici o modificare pe diferite calculatoare care au limbajul C implementat. Aceasta carte are drept scop sa-l ajute pe cititor sa invete sa programeze in C. Ea contine o initiere, pentru ca noii utilizatori sa poata incepe cit mai repede posibil, capitole separate pentru fiecare caracteristica majora, si un manual de referinta. Marea parte a textului nu se bazeaza atit pe expunerea de reguli si propozitii cit pe citirea, scrierea si revizuirea de exemple. In cea mai mare parte exemplele sint programe reale si sint complete si nu fragmente izolate. Toate exemplele au fost testate direct din text, care este intr-o forma citibila pe calculator. Pe linga faptul ca am aratat cum se utilizeaza efectiv limbajul, am incercat in plus, acolo unde era posibil, sa-l ilustram cu algoritmi utili si cu principii de bun stil in programare si proiectare sanatoasa. Aceasta carte nu este un manual introductiv de programare. Ea presupune anumite familiaritati cu conceptele. de baza din programare, ca variabile, instructiuni de asignare, bucle, functii. Cu toate acestea, un programator novice va fi in stare sa citeasca cartea si sa-si insuseasca limbajul, chiar daca ajutorul unui coleg cu experienta mai mare i-ar usura munca foarte mult. In experienta noastra, C s-a dovedit un limbaj placut, expresiv si adaptabil pentru o mare varietate de programe. Este usor de invatat si "se poarta bine" pe masura ce experienta in programare cu el creste. Speram ca aceasta carte va va ajuta sa-l folositi bine.

Re:dusort

[Phexro]

But why bother with the awk, when you can just do:

$ du -s /path/to/wherever/* | sort -rn | head -10

to get the top ten hogs in /path/to/wherever?

Also, sometimes there are some big files, and you are only interested in the directories full of crap:

$ du -s `find /path/to/wherever/* -type d` | sort -rn | head -10

While we're on the subject, you can use this handy-dandy snippet to find the disk usage of one user in any part of the filesystem:

$ find /path -type f -user jsmith -exec ls -l {} \; 2>/dev/null \
> | awk '{ sum += $5} END { print sum }'

Real shell timesaver...

[stevey]

The biggest single command which saves me time is 'cd -' which changes to your previous directory under bash.

It doesn't sound terribly useful, but it is... Take my word for it.

best grep trick ever

[novarese]

Instead of
$ ps aux | grep foo | grep -v grep
use
$ ps aux | grep [f]oo
The brackets will show up in the ps output but don't match your pattern, so your grep is automaticly excluded from your final output.

Re:Regular Expressions

[Permission Denied]

Here's a cute little perl script that's useful for developping other perl scripts:

#!/usr/bin/env perl
$st = "\033[7m";
$en = "\033[m";

print "Enter regex: ";
$re = <>;
chomp $re;
while () {
print "Enter string: ";
$s = <>;
chomp $s;
exit unless $s;
$s =~ s/$re/$st$&$en/g;
print $s . "\n";
}

This will highlight what parts of a string match a given regular expression. (I modelled this script on something I found in a Ruby tutorial).

Also, for those learning regular expressions, I would highly recommend Introduction to Automota Theory, Languages and Computation by Hopcroft et al. This is a more mathematical/theoretical introduction to the subject. Regular expressions/automata are a very nice part of CS in that the basic theory is immediately applicable.