(Useful) Stupid Unix Tricks?

So the other day I messaged another admin from the console using the regular old 'write' command (as I've been doing for over 10 years). To my surprise he didn't know how to respond back to me (he had to call me on the phone) and had never even known you could do that. That got me thinking that there's probably lots of things like that, and likely things I've never heard of. What sorts of things do you take for granted as a natural part of Unix that other people are surprised at?

Comments

Tab

[computersnstuff]

I'm sure everyone at some point is surprised of tabbed completion.

Re:Tab

[Craig Davison]

With bash, you can even get tab completion for hostnames. Try this:

ssh user@l[tab]

Everything after the @ is filled in from /etc/hosts.

Listing directory contents without the ls command

[thepacketmaster]

echo *

I discovered if you give the echo shell command an asterisk as a parameter, it dumps out the file names of the current directory. (The sad thing is I had a practical use for this when a less-than-clueful-collegue deleted the /bin directory, leaving the system without an ls program).

X-forwarding

[mikeb]

I've seen Windows people go slack-jawed in astonishment as I ssh to the other side of the world and run X programs over forwarding.

Some refuse to believe it, others shake their heads and walk away.

Re:X-forwarding

[Chris Pimlott]

You could easily have an entire Ask Slashdot just on ssh, perhaps the greatest unix command ever invented.

One of it's many great uses is creating secure tunnels:

ssh user@remotehost -L123:example.com:456

Open a tunnel on your local machine, port 123, to example.com, port 456, via the remote host

ssh -R lets you go in the opposite direction (tunnel from remote end to local end), but if your application supports SOCKS, it's even easier:

ssh user@remotehost -D8080

Creates a secure tunnel supporting the SOCKS protocol.

Talk / DD / Mount

[p14-lda]

People seem to be losing the ability to use all the older manual ways of doing things.

On the older systems, talk was a great utility.

dd, device duplicator / disk destroyer

mount, what I can't have a desktop icon?

also managing disk volumes and the old conventions of /opt, /u, /usr, /usr/local

This new fangled Linux craze with all of the UI tools is feeding it. Redhat is training admins that are dependent on a given release of their enterprise software (which I am a huge fan of) but not teaching them how it works under the hood.

How about slirp? scp?

The one ray of hope seems to be a new generation hacking their bsd and linux based (iPhone/Android) phones and having fun in a somewhat embedded (but full blown) *nix environment.

cd -

[JustinOpinion]

In terms of navigation directories efficiently, I find that "cd -" is often forgotten (changes directory to your previous directory). I personally find it very useful, and couldn't live without it!

Job control.

[Craig Davison]

fg, bg, kill, Ctrl-Z, &. Learn it. Know it. Live it.

Even if they do know about job control, I've seen people look for a background job with ps, and then kill it using the PID. In most shells you can just do kill %, e.g. kill %1

Re:Listing directory contents without the ls comma

[marcansoft]

You're not giving echo an asterisk as a paratemer. You're giving the shell an asterisk, which it dutifully expands. echo (which in this case is a shell builtin, but it doesn't have to be then just echoes them back.

This isn't some echo peculiarity. It works for anything, even commands that don't normally take files, or even with files that look like switches (conversely, if you want to treat all subsequent arguments as files, not switches, most programs have a '--' switch):

$ ls
a -l b c
$ ls *
-rw-r--r-- 1 marcansoft users 0 2008-11-05 21:58 a
-rw-r--r-- 1 marcansoft users 0 2008-11-05 21:58 b
-rw-r--r-- 1 marcansoft users 0 2008-11-05 21:58 c
$ ls -- *
a -l b c

In the second example, ls sees "ls a -l b c" and takes -l as a switch instead of a filename.

Re:This one always surprises people for some reaso

[interiot]

• Optimized version of that: find / | grep -i $SOMETHING
• Even more optimized: find / -iname $SOMETHING
• However, most systems support locate/updatedb already, and that's much faster.

-exec as a test

[Chris Pimlott]

One great feature of find that many people are unaware of is that you can use -exec as a test, not just as an action. For example, this is equivalent to your command above:

find . -exec grep -q {} \; -print

The "-print" action is only executed if the -exec command returns success.

You can do a lot of handy things with this. Here's a real-world example from earlier today. I wanted to change the mime-type of all the xml files in my svn repository from "application/xml" to "text/xml":

find . -name \*.xml -exec sh -c "svn propget svn:mime-type {} | grep -q application/xml" \; -exec svn propset svn:mime-type text/xml {} \;

Re:Show attached block devices

[duguk]

Some more common ones I've thought of:

screen - too useful, run apps in a virtual console which you can attach, deattach and share

cd `pwd -P` - Jump into the real directory (from a linked directory).

history - use it with grep if you forgot what you did

strings - just show the printable strings from a file

tail and head - tail -f is a lifesaver

sftp - i really shouldn't need to explain this.

file - do magic stuff

Hope that's some help.

short list of shell tips

[James Youngman]

Assuming you already know the simple stuff like how to use shell quotes correctly, what you can do with ps and top, ...

1. Using awk '$3 ~ /foo/ { bar }' to grep just one column of a file
2. reset
3. find . -blah -exec quux \+
4. Adding : to the front of complex commands you just typed but realise you don't want to execute yet so that they get into your shell history
5. Meta-T in Bash for swapping arguments
6. find . -printf X | wc -c for counting files (since find |wc -l would miscount files with newlines in the name)
7. set, shift and implicit shell loops (for without in)
8. "${foo:-bar}" and similar
9. "${x%%.ext}.newext"
10. comm -3 <(sort long) <(sort short)
11. unalias rm

Share mouse and keyboard

[pieleric]

When I pop up with my laptop to discuss with a colleague, after a while I might do on their computer:
xhost +mylaptopname

and on my laptop I do:
x2x thecomputername:0 -west

Then suddenly my mouse can go over the two computers, my keyboard works on both as well, and I can even copy-paste between the two computers. It looks like the two computers got united. In a flash, newbies get a new idea of what means unix and X ;-)

Re:Show attached block devices

[HTH NE1]

This shows all attached block devices (it also errors like crazy, hence the | more)

blockdev --report /dev/* | more

Redirect stderr much?

blockdev --report /dev/* 2> /dev/null

Re:A simple search

[AKAImBatman]

Xargs is much more fun with complex data processing. e.g.

Convert all PSDs to PNGs:

ls *.psd | cut -d . -f 1 | xargs -L1 -i convert {}.psd {}.png

Parse out and sort column 2 from a semicolon delimited file:

cat myfile.txt | cut -d \; -f 2 | sort > output.txt

Oh, I almost forgot about one of my favorite tricks. Count the number of items:

wc -l
(paste the list into the window and then type CTRL-D)

It even works when the list of items has oddities. e.g. I had a list where every other line was blank. So I needed to count n/2 the value. Except that one of the blank lines wouldn't copy, so I actually needed (n+1)/2.

echo $(($((`wc -l`+1))/2))

Want to make sure your sig is under 120 characters? Type "wc -c" in, paste it into your terminal, then press CTRL-D. Instant character count.

Ah, all the fun stuff you can do with Unix tools.

Re:A real time saver!

[Mish]

DISCLAIMER: Don't run this!
I didn't think I needed to say this, but I just showed someone this and they thought it was a legitemately helpful command...

Re:Show attached block devices

[MrMunkey]

cd -

Change to your previous directory. It's great for going from dev to test environments from time to time.

grep --color

[krappie]

grep --color

For some reason, many people are greatly surprised when they figure out that grep will highlight matches for them.

lsof

[pak9rabid]

lsof is a LIFE SAVER for trying to find what's still using something in a mounted resource when trying to unmount something. For example:

lsof /mnt/myMount

That will list which processes have anything under /mnt/myMount open

It's also useful to find who's accessing what device. For example, say you're trying to listen to an mp3 and Amarok bitches about the sound device not being available. In that case, you could do something like this (assuming you're using ALSA):

lsof /dev/snd

That will list what processes are accessing any of your ALSA sound devices.

Re:Show attached block devices

[aniefer]

ctrl+r (in bash?): reverse incremental search through history.
pushd/popd , change directory saving the old one on a stack.

Re:Show attached block devices

[jonaskoelker]

sftp - i really shouldn't need to explain this.

I much prefer sshfs. Diff doesn't work so well over ftp ;)

Re:Show attached block devices

[cain]

tail and head - tail -f is a lifesaver

I use tail -F, which is the same as tail -f, but works on non-existent files. Useful when tailing log files from programs that start a new log file every time it runs. Using tail -F in this case, you can just leave tail running while you start and restart the program overwriting the log file.

Shell history tricks

[steveha]

There are a whole bunch of "history" tricks, to recall old commands without using the mouse.

When I started college, I studied the shell's man page until I knew them all. Some are so obscure I have forgotten them.

Generally, these involve an '!' character in some way.

Here are a few I use:

!! # run again the last command that was run
!9 # run again the command with history number 9
!v # run again the last command that started with a 'v'
!vi # run the last command that started with "vi"
!?foo? # run the last command that had the string "foo" anywhere in it

diff oldfile newfile
mv !$ !^ # same as "mv newfile oldfile"
# !$ is last arg of previous command, !^ is first arg

ls foo bar baz
rm -f !!* # same as "rm -f foo bar baz"
# !!* repeats all arguments from previous command

There are actually some baroque tricks that recall a previous command and perform a search-and-replace on it, but for anything that complicated I just recall the line and edit it. The baroque tricks would have been pretty darn cool back in the paper teletype days, though.

By the way, the Bash shell can be configured to edit command lines using vi or Emacs commands. I described how to do it in an article I wrote for Linux Journal magazine. It's the last section, "vi or Emacs Mode in the Shell".

http://www.linuxjournal.com/article/8361

Oh, not exactly a history trick, but here's something I use all the time:
ls -1 > /tmp/files
vi /tmp/files # edit list to include just the files I want
rm `cat /tmp/files`
# `cmd` inserts the standard output from cmd into the command line as if you typed it

ls -1 > /tmp/files
vi /tmp/files
# edit list to include just the files I want
# now run this command: :%s+.*+mv & /some/directory/path/&+
# save file and quit vi
source /tmp/files

This moves the chosen files to "/some/directory/path". The breakdown of the vi command is as so:
: # invoke "ex mode" for search and replace command
% # run the following command on every line of the file
s # do a search and replace
+ # use a '+' for the command delimiter, so I won't have to backslash escape '/' chars in the path

.* # all characters on the line
+ # end the match pattern, begin replace pattern
# & refers to the match pattern, thus all chars on the line
mv & /some/directory/path/& # replace "foo" with "mv foo /some/directory/path/foo"

Takes less time to do it than to explain it!

The above is perhaps overkill if all the files are going to the same place. It's great if you want to send some files one place, some to another, because you can just edit the destinations until it looks right.

steveha

Re:directory stack

[Chris Pimlott]

In tcsh (I don't know other shells), you can do directory stack substitution. =0 is current directory, =1 is one up, =2 is two up, and so on.

In bash, it's ~0, ~1, ~2, etc.

Re:Well

[Hatta]

You only need a tiny bit of entropy to seed the pseudorandom number generator in /dev/urandom. Once it's seeded you get all the pseudorandom numbers you want. /dev/random gives you truly random numbers, and is highly dependent on the amount of entropy the system has. It will block if you run out of entropy, urandom will not.

Re:Show attached block devices

[EvanED]

Along that line are pushd and popd. pushd <dir> changes to the specified directory and pushes it onto a stack of directories; popd changes to the directory at the top of the stack and removes it. There are commands for manipulating the directory stack but I don't know or use them.

With zsh, and I think with Bash as well, you can setopt AUTO_PUSHD and setopt PUSHD_SILENT and then cd behaves like pushd.

(Both of these commands, along with cd -, work in the Windows command interpreter too.)

Useful tricks.

[ElizabethGreene]

I can't live without svn. Svn is a revision control repository, usually used for source code. What makes it really powerful is that you can _easily_ have a history of everything that has changed in a file and when. On my systems, I keep /etc in svn, plus bind's zone files, plus all the non-image web content, and the "Network Documentation" folder.

Second trick, rsync. I use it to backup my home directory to another box. Very nice when you go through a hard drive/year.

Screen -x was my next pick, but somebody already mentioned it.

"echo ProtocolKeepAlives 120 >> /.ssh/config" No more dropped ssh sessions because of stupid nat boxes.

su -u Username -s .. become Username, but keep the current shell. Good for diagnosing permissions problems when the user has a /bin/false shell. (named/www-user/backup/etc).

A little awk goes a long way. Not the big-bad-I-am-a-programming-language-awk, but the smaller-friendlier extract one or two columns of text from something awk. ex. awk '{print $2}' prints the thing in the second column. Add -F the field separator tool and it gets really useful.
Better example. Here is a postfix log line.
      Nov 5 16:27:19 pdc postfix/smtpd[13601]: 92B3F499C25F: client=exprod5mx254.postini.com[64.18.0.49]
Here is the awk to extract just the message id. awk -F': ' '{print $2}'

And here is the "I didn't get this message your mailserver must have eaten it" disprover. It searches the maillog for every message from or to a given address and extracts the full email transaction for that message id.

grep -i user@domain.com /var/log/maillog | grep smtpd | awk -F': ' '{print $2}' | sort -un > temp.fil && grep maillog -f temp.fil

Next trick, back ticks. `` Back ticks substitute the output of a command within a command.
Ex. Name a file after the date. echo "hi" > `date +%Y%M%d`.txt

On the subject of dates. date -d'yesterday' or 'last week' or '-4 hours' can be handy.

Last one. Loopback nat with Iptables, so you can access local hosts by their external ip. (Instead of setting up split dns.)
iptables -I POSTROUTING -s $local_network -d $local_network -j SNAT --to $lan_interface

My local network is 192.168.0.0/24 and the netfilter lan ip is 192.168.0.1, so that becomes...
iptables -t nat -A POSTROUTING -d 192.168.0.0/24 -s 192.168.0.0/24 -j SNAT --to 192.168.0.1

-ellie

Re:Show attached block devices

[ip_fired]

I actually like less better than tail -f. If you less a file, and then hit SHIFT-F it will tail the file, but you can break out of it and scroll around and search for terms. Very handy while looking at log files.

Time warp

[SchmellsAngel]

cal 9 1752

Re:Show attached block devices

[geminidomino]

Organizing them all is left as an exercise for the reader.

Cakewalk. Put the following in ~/.bash_profile or ~/.profile

For ubuntu:
SHELLID=(echo `tty` | sed 's!/!.!g')
HISTFILE=$HISTFILE$SHELLID

Logout. Log back in. Bada-bing.

Re:Show attached block devices

[BorgCopyeditor]

Your data is corrupt: only the current sig is ever shown.

Re:Show attached block devices

[Isomer]

More awesomely, if you have found something in your history with ^R or up arrow or whatever, then you can press ^O to "execute this line and put the next line in the history onto the command line". Thus:
vi foo.c
make
./foo
^Rvi^O^O^O^O^O^O^O^O^O^O^O^O^O^O
and so on.

Re:Show attached block devices

[FrangoAssado]

I think you missed a '$' before the open parenthesis, it should be:

SHELLID=$(echo `tty` | sed 's!/!.!g')

Also, you could replace "echo `tty`" with simply "tty":

SHELLID=$(tty | sed 's!/!.!g')

Re:rm -rf /

[Curtman]

when I told a guy over the phone to 'rm -rf /bin' but unknowingly he was at "/". Oooops

I don't get it. You told him to give rm an absolute path, it doesn't matter where he was.

Re:rm -rf /

[VShael]

Pshaw! All 1337 sysadmins just live as root!

Yes, but all the really stupid ones do as well.

Re:session-sharing with screen -x

[quarterbuck]

eject is a useful tool, If you have a rack of servers all alike and you need to identify one of them . Some servers have blinking lights etc. but mine had no audio nor lights. But it had a CD tray.
I simply put eject and "eject -t" in a loop and go look in the server room -- the hyperactive server is the one I was looking for.