What's Your Command Line Judo?

lousyd asks: "We all have our CLI amor. That two- or three-letter command that fiddles our heartstrings. 'mv' is pedestrian, but 'mmv' is so cool. 'cp' can lug bits around all day to no applause, but 'cpio' will excite even the most jaded of command line linguists. 'How perfect!' you exclaim. So what's your poison? What turns you on? What little known command performs just the right function for you?"

Comments

hrm.

[grub]

foad works every time.

lh and hog

[newsblaze]

My favourites are

lh ls -lt !* | head -15

which shows me the newest 15 stories in the current or specified directory

hog "ps -eo pcpu,vsz,args,time | sort -rn | head -11"

which shown the 10 most cpu-intensive processes

can't eat just one

[yagu]

Command line judo? Sheesh! Where to start?

Okay, the tool I'm using now: vim, derivative of Bill Joy's vi, with color syntax and a bounty of enhancements. (Yes, I prepare my comments in vi, then cut and paste, don't even try to make me use some GUI text widget editor and claim it can be productive.)

And then there's:

• sed, or even ed. ed is great for scripting edit changes if you don't have the heart to master sed. I once used "sed" to create nroff output of document versions with delete markers (asterisks), change markers, and insertion markers from our technical documentation library.
• awk. I couldn't have done half the work I've done in my career without this one. I watched entire teams scramble to solve some problem in Visual Studio, and would crank out a twenty line awk script that afternoon. Problem solved.
• [ef]grep. If you're going to pipeline some shit, it's nice to filter it first (and sometimes second, then third, etc....). The grep family is second to none for this. (Funny story: I once worked at Microsoft. Was doing some stuff in DOS. Asked what command I'd use to find a string in a file or files. They showed me "FIND". Okay. I typed in:
  find mystring *.*
  and literally got back:
  mystring found in *.*, and then a listing of all the lines found. No reference to the files they were in... Shit, after all, I asked to find the lines in '*.*'.

  I complained. They showed me their (Microsoft's purchased from IBM) unix, "xenix", and their "grep" command. Ahhhhh, better. I typed: grep -i mystring *.*
  and it replied "unknown option -i". I complained about not having an "ignore case" option. They looked at me like I was crazy... "Why would you ever want to ignore case?"

• diff. A life saver. 'Nuff said.
• find. Freak! When I first met this command I said "go away!". Talk about an obtuse syntax and paradigm. Learn it! It's worth it.
• xargs. Nice way to get around the line length restrictions of some of the unix shells.
• zsh, bash, (but not csh)... Any interactive shell with a good history and command line completion mechanism. I never have to remember command syntax anymore... usually I can find a usable and editable example somewhere in the last 2000 commands I've executed.

I could go on, and probably will in some subsequent posts. When you have so many well written, well evolved, well crafted, and well behaved tools all flying in

Re:can't eat just one

[ATomkins]

I complained. They [Microsoft] showed me their (Microsoft's purchased from IBM) unix, "xenix"...

Sorry to be picky, but Microsoft bought a license for UNIX from AT&T, developed Xenix, then sold it to SCO who would transform it into SCO UNIX.

Re:can't eat just one

[bergeron76]

I actually use a shell script in my .bash_logout that appends my .history file to a master "history" file. It's helped me numerous times when I've need to recall a command I ran ages ago. I just use grep succesively until I find the obscure command string.

I don't mean that I was looking for a single command; I use bash pretty hardcore, and I've used pipes to do some pretty extensive things - that's where I think history has it's greatest use.

(and yes, I'm using bash 3 and my HISTSIZE envvar is set to 10000)

Here's mine

[Henry V .009]

My most important tools as a system admin:

alias rm 'rm -rf \!*'
alias hose kill -9 '`ps -augxww | grep \!* | awk \'{print $2}\'`'
alias kill 'kill -9 \!* ; kill -9 \!* ; kill -9 \!*'
alias renice 'echo Renice\? You must mean kill -9.; kill -9 \!*'

cp -g

[ReKleSS]

I don't know how much information your script gives, but cp -g gives you a progress bar, transfer rate, completion percentage and other stuff when a transfer is going to take more than a few seconds (similar to what scp gives you). It's sometimes a handy feature to have, sounds like what your script is doing...
-ReK

CTRL + L

[Kizzle]

Every person I see using linux clears the screen by typing CLS. The key combo "CTRL + L" is so much sexier.

Re:CTRL + L

[pclminion]

CTRL+L in almost any terminal application forces a terminal redraw. Comes in handy when the application doesn't explicitly detect SIGWINCH, and you want to resize the terminal window. Resize, then hit CTRL+L to inform the app that the window size changed.

I wish the behavior was the same in bash, but for some reason they chose to make it clear the screen instead.

skill

[cgenman]

I'm not much of a linux guru, so my favorite is simple.

skill

s-kill, basically, kills a process by name. "skill netscape" will kill netscape, no finding proc ID required. It's what kill should have been from the beginning.

The only command I love more than skill is apt-get, but that doesn't really count.

Re:skill

[Fweeky]

Looks like killall to me:

SYNOPSIS
    killall [-delmsvz] [-help] [-j jid] [-u user] [-t tty] [-c procname]
            [-SIGNAL] [procname ...]
 
DESCRIPTION
    The killall utility kills processes selected by name, as opposed to the
    selection by pid as done by kill(1). By default, it will send a TERM
    signal to all processes with a real UID identical to the caller of
    killall that match the name procname. The super-user is allowed to kill
    any process.

Though you need to be careful with that if you admin other systems like Solaris ;)

pkill/grep are nice too, and are standard on a fair few systems now.

ls -d without the subcontents

[bergeron76]

I never figured out a way to use 'ls' to show only directories (and not their subcontents), so I created an alias called 'lsd':

alias lsd='\ls -l | grep "drwx"'

and placed it in my .bash_profile

It's quite useful, but it doesn't work well with shell scripts.

Re:ls -d without the subcontents

[pclminion]

Dude...

ls -d */

The final slash is key.

Re:ls -d without the subcontents

[Nutria]

alias lsd='\ls -l | grep "drwx"'

What if a file has the string "drwx" in it?

And you forgot about dot directories.

alias lsd='ls -al | grep ^d'

Re:ls -d without the subcontents

[sootman]

try ls -F | grep /

'-F' shows a slash after each entry that's a directory, an asterisk after each executable, and some other things. Read the man page for 'ls' sometime, it's like a whole freaking operating system.

NAME
          ls - list directory contents

SYNOPSIS
          ls [-ABCFGHLPRTWZabcdfghiklmnopqrstuwx1]

By the way, my favorite script fu is `du -sh ./*` to show how much room each file takes up, and it adds up directories also--just like 'calculate folder sizes' has done in Mac OS for a decade.

perl

[Turmio]

First, I must comment the article. Question goes What little known command performs just the right function for you? I hope all sane people here (haha) would answer "None". There's no command that does just the right function since there's no one The Function. It depends on the situation what the function is. And in that case, per Unix philosophy, where one tool does one simple job, but does it well, you should choose the tool accordingly.

Enough of that. If you really must name something, then, in my opinion, there's one gizmo above others. And that is

perl

Perl one-liners is a damn powerful concept when you get it. Say one of your boxen switches IP address. You want to replace all references in files under /etc to this new IP address. First you might think searching all the files under /etc with find(1), then passing the list of files to grep(1) and then manually editing the places where the old IP address was found with your $EDITOR. That's fine and will get the job done and all but what if you could just edit the files in place? With perl, you can.

perl -pne's/oldip/newip/g' -i `find /etc -type f`

and you're done (better be extra careful with commands like that for obvious reasons!). Of course you're able to do the same thing with other tools too, but I don't think it could be much easier than that. And naturally you're not just limited to simple search and replace of text, you have the full power of Perl (and CPAN!) at your disposal.

Besides being my number one choice for creating complex scripts and small applications, Perl has very special place in my command line toolbox just next to the old friends such as grep(1), cut(1), wc(1), etc. and a huge pile of pipes :)

Re:perl

[davegaramond]

Second that. "Traditional" Unix tools sometimes just don't cut it (or they do, but requires significantly longer typing and multi-piping work).

I use this all the time:

$ perlrename -e's/.\[www.descargasweb.net\]//' *
$ perlrename -e's/.\[www.\S+?\]//' *
$ perlrename -e'$_=lc' *
$ perlrename -de's/Sex And The City - S(\d+)E(\d+)/SATC - $1$2/ig' Sex*
etc etc

where perlrename is a 30-something perl script.

groff

[(1 -sqrt(5))*(2**-1)]

groff -ms -t -Tascii <file>.ms

is how I code all my outgoing emails; that's GNU troff with the ms macro package and tbl preprocessor.

Justifies nicely said paragraphs; provides lists and sigart.

Junction for Windows

[alan_dershowitz]

Junction lets you make symlinks in Windows without installing the entire Windows Resource Kit tools. Also, CACLS.EXE for changing ACLs in Windows via the command line, since I have no fucking clue where you do this in the GUI. Some of the more usefule CLI commands in Windows, IMO. I hope this discussion wasn't limited to Unix or anything.

Re:Junction for Windows

[ConceptJunkie]

I hope this discussion wasn't limited to Unix or anything.

It shouldn't be. Another source of good command line tools is sysinternals.com.

The pstools let you do all kinds of nifty things that are sometimes covered by RK utils, but the sysinternals stuff is usually way better... ...with one exception that drives me crazy. pslist is their version of ps, which I just alias using 4NT, however they don't echo the banner to stderr like most utils, so if you sort the list looking to see if you need to kill and errant process of Firefox 1.5 beta 1, you get the banner sorted with the list of processes.

Other than that, they are great tools.

You can easily control services on other machines ("net" might let you do this but I don't know how), you can even defrag with "contig". Heck there's even a blue screen screen saver. At one place, years ago I used to run this to encourage people not to mess with my machine.

Of course no Windows command line would be complete without cygwin in the path.

Re:Junction for Windows

[BrynM]

without installing the entire Windows Resource Kit tools

There are several RK tools in my MS toolbox, but the best thing is having real unix utils. Pop those in your %PATH% and enjoy some of the same fun that's being spoken of here. Of course there's always Cygwin, but these native ports are handy to keep on a USB drive and don't need any configuration/installation at all.

Re:Junction for Windows

[sudog]

Why waste your time fighting with DOS commands?

UNIX Utils, Natively Compiled for Windows

dusort

[RGRistroph]

in bash:

function dusort ()
{
  du -s "$@" | sort -r -n |
    awk '{sum+=$1;printf("%9d %9d %s\n",sum,$1,$2)}' ;
}

in tcsh:

alias dusort 'du -s \!* | sort -r -n | awk '"'"'{sum+=$1;printf("%9d %9d %s\n",sum,$1,$2)}'"'"

The most common way to use those commands would be:

cd /home/
dusort *

It's useful for tracking down what's using up your space, for example finding a sub directory deep in a source tree that isn't cleaned by make clean.

hmm...

[bbrack]

perl -pi -e "s/x/y"

ever had to make a change to every line in a test vector (up to several million lines long), but didn't have the half hour it would take to retranslate the whole thing? - has saved my ass more than anything I can think of
also fun to do something like

perl -pi -e "s/(alias \w) \'.+\'/$1 \'echo \"DFU DFU DFU\"/g ~user/.aliases

ls -t | head

[ArmorFiend]

ls -t | head
This is nice for listing the 10 most recently modified files. Usually the one you want is the first or second one.

We're not talking about cool commands, here!

[pclminion]

I don't think the point of the article is to discuss which commands are coolest, rather, what ways can you combine those commands to achieve powerful results?

Here's something that I run via cron on a nightly basis. See if you can decode its function :-)

for F in $(find $SRCDIR); do echo $(basename $F) $F; done | sort | rev | uniq -c -f 1 | grep -E ^[[:space:]]*1[[:space:]] | awk '{print $2 " " $3}' | rev > $CACHEFILE

screen

[pauljlucas]

Multiple multiplexed ttys that stay running even after disconnect and you can reattach to them later.

Re:screen

[vrai]

I'd add disown to that list - under bash it detaches background jobs from the tty and allows them to continue running after session is closed (like nohup).

Example:

vrai@lenin $ complex_task
^Z
[1]+ Stopped complex_task
vrai@lenin $ bg %1
[1]+ complex_task &
vrai@lenin $ disown %1
vrai@lenin $ exit

The complex task continues to run in the background; though the output will be lost to /dev/null unless redirected.

the one that impresses most people

[elmegil]

Though I have no idea why, it's pretty basic:

find . -type f -exec grep regexp {} /dev/null \;

The tricky bit is that adding /dev/null means I will always see the filename in the output; otherwise grepping against one file only returns the matched strings.

Re:the one that impresses most people

[_randy_64]

If you're using gnu grep, you can use the -H option (or --with-filename) instead of the dummy /dev/null argument.

Give credit where credit is due

[mister_jpeg]

That's the Technical Fascist's .cshrc.

http://www.gnu.org/fun/jokes/know.your.sysadmin.ht ml

unison

[buffy]

Based on the rsync protocol, unison can maintain a bi-direction file sync. I use it to sync my personal data between home and co-lo'ed server, and work and co-lo'ed server. I can update files in any of the three locations and they'll be replicated across all three.

-buf

xargs and for loops

[photon317]

There's millions of tricks, but if I had to a couple simple powerful techniques that anyone should learn that doesn't know them already, it would be xargs and commandline "for" loops.

xargs takes whatever is piped into it, and executes a command with those things as arguments. It can do it all at once, or it can break them up in chunks, or it can execute your command once per input. Consider:

rm -f `find . -name "*.o"`

This normally works fine, and will forcibly remove all .o files anywhere underneath the current directory. However, if there are too many .o files to fit on a single commandline, it will barf with "argument list too long" or some such sounding error. The xargs way to do this would be:

find . -name "*.o" | xargs -n 50 rm -f

Which will execute a seperate "rm -f" for each chunk of 50 filenames. Take a look at the "-i" mode as well, read the whole man page. It's a great little peice of glue.

On to for loops. You've seen them in sh/ksh/bash shellscripts like so:

for fn in *.c
do
    echo Sending $fn ...
    rcp $fn remotehost:/tmp/
done

You can of course do this straight from the commandline, which is indispensable for complex looped operations. To do it all in one line, you just have to get the semicolons in the right place. Just remember there's a semicolon before the "do", but not immediately after it:

for fn in *.c; do echo Sending $fn ...;rcp $fn remotehost:/tmp/;done

Re:xargs and for loops

[photon317]

Oh a couple more good simple ones popped into my head for beginners:

Use "&&" between commands, and the second command will only execute if the first command succeeded. For example:

make && make install

That will only run the "make install" if the "make" succeeded (which is kinda contrived, since for most packages a simple "make install" would have done the same all on it's own).

And also, did you know that you can group commands with parentheses, and you can pipe into parenthesized groups of commands? Piping into a group of commands will send the pipe data to the first command in the group that actually wants it. You can copy a directory over from one place to another using tar all in one line like so:

tar -cf - . | (cd /otherdir/; tar -xvBpf -)

The two tar commands are simply piping a tar-format file between them without using the disk, and the second tar runs with it's current directory set to /otherdir/.

make yourself a few aliases

[frn123]

Need to find that file named somethingfoosomeotherthing?
alias ff='find . |grep -i '
>ff foo

./somedir/somethingfoo
./somedir/something foo2
./somedir/somethingfoo3

Need to find that pesky configuration file for printer?
~/bin/gr:
grep -i -r $1

~/bin/fgre:
grep -i -r -l $1

>cd /etc/
>fgr laserjet

./cups/ppd/hp.ppd:*ShortNickName: "HP LaserJet Series"

Just interested which files to check?
>fgre laserjet

./cups/ppd/hp.ppd

Midnight Commander.

[Inoshiro]

'mc' is your gateway to a much more powerful shell. Combine your command line shell with quick changing to bookmarked dirs (ctrl+\); easy adding to that via alt+a; F2 menus that let you easily compress subdirs, or run other things from your actions file based on the mime-type or file pattern; two-panes at two different FS locations; undelete FS, FTPFS, and other meta-FS interfaces; good mouse support; built-in editor (Cool edit); prompts you for arguments for makefiles you hit enter on...

The list goes on. Midnight Commander is a shell designed to speed up all your common tasks. It has sane defaults you don't really have to change much. It'll make you more productive!

Re:Midnight Commander.

[fuzza]

Not to mention, it's really useful for temporary (ie. until you can reinstall) cleanups of compromised systems, since (a) it uses its own internal code for things like ls, chmod, etc, and (b) rootkits don't generally replace it :)

My favourites

[cowbutt]

strace/truss/ktrace for Linux, Solaris and BSD/MacOS respectively. If a program fails to start, or terminates abnormally, this will usually give me the heads up on why (it's usually a missing file, or bad permissions) without having to break out gdb.

lsof. Useful in so many ways; for debugging situations similar to the above, as well as hardening systems and building chroot environments for specific programs.

tcpdump/snoop/tethereal/ethereal. If you can see what's really on the wire between two network applications, you can probably figure out what's going wrong. Ethereal is particularly nice.

hexdump/khexedit. If you can see what's really in the file used by an application, you can probably figure out what's going wrong. :-) khexedit also has a bonus feature of being able to perform statistical logical operations across the file; useful if you have a file which you suspect has been encrypted with some lame substitution cipher.

After those, the usual - sed, awk, grep, find. It's rare that I can't turn any problem into an awk-shaped nail. :-)

scp is your friend

[DrSkwid]

scp -C -r * user@system:target

and

scp -C -r 'user@system:/target/*' .

ZTree and UnixTree

[Glonoinha]

Yea, Though I compute through the user interface of the shadow of death, I shall fear no evil, for ZTree is with me.

No joke - ZTree is a character for character re-write of an old utility called XTree Gold v2.0 or 2.5 - and it is by far the most effective and influential interface between me and my data. The entire file system is but an extension of my mental processes, and I can slice and dice through the multi-dimensional (time, space, attributes, multi-layered directory structure, multi-drive architecture,) in effect creating a virtual directory within which I control the parameters driving what is listed, in what order - then copy, view, move, delete, diff (file compare), view in hex mode (and edit it in hex mode), search for text in lists of files, compare directory trees for like or different files (binary, time stamp, etc.)

It is totally CUI, about like Midnight commander but a ton better. Take time to get fluent in ZTree (UnixTree for Unix / Linux, a bit older with a few quirks, but still pretty damn good) and you will be like the guys in the Matrix sitting at their green screen terminals.

ZTree Don't leave $HOME without it.

Re:ZTree and UnixTree

[Glonoinha]

Ya - there is a Unix version too (not from the same guys, but following the same exactly layout.)

And I got excited about TC - but alas it is mouse clicky centric. ZTree was designed and implemted to be 100% character based (although I guess you could use the mouse, I haven't actually tried.)

When you can use TC to find the files located somewhere on your hard drive, directory unknown, name unknown, extension unknown, dated somewhere around Christmas of 2003, containing the phrase 'Total Commander' but not containing the phrase 'Midnight Commander', sized between 50k and 53k, excluding .dll, .exe, .js and .html files, including files that are flagged system or hidden but excluding files that are flagged 'system and hidden and read-only', and then create a text file with a list of the full path/name.ext of all the files sorted reverse chronologically - without touching the mouse - in under 20 seconds - and then copy all of those files yet only those files (twice, one including directory structures, and again just dumping all of them into some new directory) to another hard drive ... then I will give it another look, maybe even drop ZTree / UnixTree (which run from the command line / command prompt / korn shell w/o a need for any GUI or mouse) for it. Until then, it looks cool, but I'm going to keep on recommending my CUI.

You are frickin' insane.

[Just Some Guy]

find . -name "*.o" | xargs -n 50 rm -f

That's great. Now, create a directory called " " (empty space). Inside that, create a directory called " -rf " ("-rf" with an empty space on either side). Inside that, create a file named " " (yet another empty space). Now, watch in horror as find prints "./ -rf /", which it passes dutifully to "rm -f". Since xargs by default passes each word as an individual argument, that expands to:

rm -f ./ -rf /

Hope you weren't running as root! The moral of this story is to never, never! use find/xargs without the "-print0" option whenever the command you're executing is destructive. "ls" is probably OK. "rm" definitely isn't.

Re:assign them to a short name

[newsblaze]

Hello Robert.
My name is Alan, no reference to Tron

in my .cshrc, I have this:
alias lh "ls -lt !* | head -15"
alias lt "ls -lt !* | more"
alias hog "ps -eo pcpu,vsz,args,time | sort -rn | head -11"
alias ff "find . -name !* -print"

I use these and many more when I'm working on the newspaper.
You can type these aliases on the commandline if you don't want to add them to your .cshrc and it will work for just that session.

So then I type lh in the CLI and it shows the newest 15 files - you can change that number to whatever you like.

I love the SELECT command...

[Richard Steiner]

...but it's limited to JP Software shells (4DOS, 4NT, 4OS2), and the authors of Linux shells don't seem interested in implementing it.

SELECT is a command which provides a fullscreen point-and-shoot file selection interface for other commands, allowing for the easy arbitrary selection of target files (point-and-shoot) in command aliases or batch/script files. In essence, it provides a mini filemanager interface which can be used to select the targets for *any* command on the command line, used in aliases, etc.

Some simple examples (from my 4OS2 alias file) are as follows:

TCOPY=select copy (*.*) %1
TMOVE=select move (*.*) %1
TDEL=select del (*.*)
TRUN=select %1 (*.*)
TEDIT=trun fte
TLIST=trun list
UNZ=f:\unz.cmd
TUNZ=select unz (*.zip)

The filespec in parenthesis limits the files which appear in the selection list, so the TUNZ command will only show .ZIP files as potential targets.