Slashdot Mirror
(Useful) Stupid Unix Tricks?
So the other day I messaged another admin from the console using the regular old 'write' command (as I've been doing for over 10 years). To my surprise he didn't know how to respond back to me (he had to call me on the phone) and had never even known you could do that. That got me thinking that there's probably lots of things like that, and likely things I've never heard of. What sorts of things do you take for granted as a natural part of Unix that other people are surprised at?
rm -rf /
Well.
dd if=/dev/zero of=/dev/sda1
screen is awesome.
The World Wide Web is dying. Soon, we shall have only the Internet.
... Generally people are surprised by the fact that you could type some strange incantations into a black window like awk grep etc and make the computer do things without touching the mouse. Yeah, some are surprised by that thing.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
So much easier for me to use than perl. I presume the modern unix user prefers perl.
I once worked with an admin that wrote a program that wrote directly to a users terminal and prompted in the same way "write" did. One notable exception is that he let you put whatever username@hostname you wanted.
I got quite a few requests from "yourmom@pronindustry.com" to chat...
I'm sure everyone at some point is surprised of tabbed completion.
And I've been administering Linux systems for awhile now. Step back for a moment and you'll find that "man pages" and "info" are actually a pretty awful way to distribute documentation. As a supplement they'd be fine, but as the main source of information on how to use many commands... not so much.
I discovered if you give the echo shell command an asterisk as a parameter, it dumps out the file names of the current directory. (The sad thing is I had a practical use for this when a less-than-clueful-collegue deleted the /bin directory, leaving the system without an ls program).
--
Luck is just skill you didn't know you had.
A sys admin was recently surprised that I didn't use screen. My explaination was that all that C-x stuff reminded me too much of using Emacs.
Moderators are free to mod this Flamebait or Insightful, depending on personal bias.
Not a typewriter
I've seen Windows people go slack-jawed in astonishment as I ssh to the other side of the world and run X programs over forwarding.
Some refuse to believe it, others shake their heads and walk away.
On the older systems, talk was a great utility.
dd, device duplicator / disk destroyer
mount, what I can't have a desktop icon?
also managing disk volumes and the old conventions of /opt, /u, /usr, /usr/local
This new fangled Linux craze with all of the UI tools is feeding it. Redhat is training admins that are dependent on a given release of their enterprise software (which I am a huge fan of) but not teaching them how it works under the hood.
How about slirp? scp?
The one ray of hope seems to be a new generation hacking their bsd and linux based (iPhone/Android) phones and having fun in a somewhat embedded (but full blown) *nix environment.
Search and replace inplace in files, using perl:
perl -pi -e "s/searchme/replaceme/g" *
In all .cpp anh .h files:
perl -pi -e "s/searchme/replaceme/g" `find . -name \*.cpp -o -name \*.h`
Or if you're a bash person:
perl -pi -e "s/searchme/replaceme/g" $(find . -name \*.cpp -o -name \*.h)
In terms of navigation directories efficiently, I find that "cd -" is often forgotten (changes directory to your previous directory). I personally find it very useful, and couldn't live without it!
Showers
Help! I'm a slashdot refugee.
The rev command has got to be one of the most useless Unix commands I've ever come across. It's almost as if someone's first c program somehow got taken up as a part of standard Unix! Maybe in the days before sed and awk and perl it had some function in pipes that I can't grok, but nowadays other than making hints for video game websites I can't imagine what it's for.
Unhackable encryption of course.
fg, bg, kill, Ctrl-Z, &. Learn it. Know it. Live it.
Even if they do know about job control, I've seen people look for a background job with ps, and then kill it using the PID. In most shells you can just do kill %, e.g. kill %1
Hands in my pocket
You're not giving echo an asterisk as a paratemer. You're giving the shell an asterisk, which it dutifully expands. echo (which in this case is a shell builtin, but it doesn't have to be then just echoes them back.
This isn't some echo peculiarity. It works for anything, even commands that don't normally take files, or even with files that look like switches (conversely, if you want to treat all subsequent arguments as files, not switches, most programs have a '--' switch):
$ ls
a -l b c
$ ls *
-rw-r--r-- 1 marcansoft users 0 2008-11-05 21:58 a
-rw-r--r-- 1 marcansoft users 0 2008-11-05 21:58 b
-rw-r--r-- 1 marcansoft users 0 2008-11-05 21:58 c
$ ls -- *
a -l b c
In the second example, ls sees "ls a -l b c" and takes -l as a switch instead of a filename.
:(){ :|:& };:
The World Wide Web is dying. Soon, we shall have only the Internet.
Hell, I think it's probably a GNU extension, because it's still not in Solaris.
I think rgrep appeared around BSD 4.4, though.
Oh well. I still surprise people with backticks. *sigh*
Do daemons dream of electric sleep()?
One great feature of find that many people are unaware of is that you can use -exec as a test, not just as an action. For example, this is equivalent to your command above:
find . -exec grep -q {} \; -print
The "-print" action is only executed if the -exec command returns success.
You can do a lot of handy things with this. Here's a real-world example from earlier today. I wanted to change the mime-type of all the xml files in my svn repository from "application/xml" to "text/xml":
find . -name \*.xml -exec sh -c "svn propget svn:mime-type {} | grep -q application/xml" \; -exec svn propset svn:mime-type text/xml {} \;
I have a theory that find + xargs + grep is Turing-complete. Can't prove it, but it feels right.
Some more common ones I've thought of:
screen - too useful, run apps in a virtual console which you can attach, deattach and share
cd `pwd -P` - Jump into the real directory (from a linked directory).
history - use it with grep if you forgot what you did
strings - just show the printable strings from a file
tail and head - tail -f is a lifesaver
sftp - i really shouldn't need to explain this.
file - do magic stuff
Hope that's some help.
chmod a+x /bin/laden
I sometimes use rev to sort text by the end of the line, not the first. This is often useful when comparing two similar file structures.
For example:
$ wc -l foo/* bar/*
6 foo/dead.letter
86 foo/xorg.conf
6 bar/dead.letter
54 bar/xorg.conf
$ wc -l foo/* bar/* | rev | sort | rev
86 foo/xorg.conf
54 bar/xorg.conf
152 total
6 foo/dead.letter
6 bar/dead.letter
(Yes, I'm aware you can use sort -k to specify the sort key, but this is quicker and easier)
Assuming you already know the simple stuff like how to use shell quotes correctly, what you can do with ps and top, ...
Need to know the serial number of a server or bios version or many other things dmidecode is your friend.
When I pop up with my laptop to discuss with a colleague, after a while I might do on their computer:
xhost +mylaptopname
and on my laptop I do:
x2x thecomputername:0 -west
Then suddenly my mouse can go over the two computers, my keyboard works on both as well, and I can even copy-paste between the two computers. It looks like the two computers got united. In a flash, newbies get a new idea of what means unix and X ;-)
The advantage of the directory stack over "cd -" is that the directory stack always remembers where you last were. "cd -" only remembers until you change directories again.
In tcsh (I don't know other shells), you can do directory stack substitution. =0 is current directory, =1 is one up, =2 is two up, and so on.
I also use bindkeys to bind Control-G to 'dirs -v' so I can look at the directory stack with ease, even in the middle of a command.
Personally, I think directory stack commands are the least-known, but most useful feature in tcsh.
The ultimate goal of science is to unify all forces of nature to a single law that can be silk-screened onto a T-shirt.
!! in bash uses the last command you entered. $ make_me_a_sandwich What? No. $ sudo !! okay. $
ls -l | sort -n +4 -- sorts files in size order, good for finding big files in a directory
du -s * | sort -n -- similar to above, find the biggest files & subdirectories of the current dir
du | xdu -- only when you're in X, obviously. Better grain than above, with the ability to drill down into subdirectories
This shows all attached block devices (it also errors like crazy, hence the | more)
blockdev --report /dev/* | more
Redirect stderr much?
Oh, say does that Star-Spangled Banner entwine / The myrtle of Venus with Bacchus's vine?
Xargs is much more fun with complex data processing. e.g.
Convert all PSDs to PNGs:
Parse out and sort column 2 from a semicolon delimited file:
Oh, I almost forgot about one of my favorite tricks. Count the number of items:
It even works when the list of items has oddities. e.g. I had a list where every other line was blank. So I needed to count n/2 the value. Except that one of the blank lines wouldn't copy, so I actually needed (n+1)/2.
Want to make sure your sig is under 120 characters? Type "wc -c" in, paste it into your terminal, then press CTRL-D. Instant character count.
Ah, all the fun stuff you can do with Unix tools.
Javascript + Nintendo DSi = DSiCade
BTW: How does one ork a cow?
Very, VERY, carefully.
DISCLAIMER: Don't run this!
I didn't think I needed to say this, but I just showed someone this and they thought it was a legitemately helpful command...
cd -
Change to your previous directory. It's great for going from dev to test environments from time to time.
My previous place of employment was a Mac shop, where I discovered the wonderful pbcopy and pbpaste commands. Why they aren't a standard part of every X windows distribution, I'll never know, but they are damned handy.
What they do is allow you to read and write from the cut-and-paste buffer from the command line. "pbpaste" will print the currently copied text to stdout, while "pbcopy" will replace the buffer with stdin.
Fortunately, there are some third-party X equivalents for this, such as xsel or xclip, which can be adapted to work in the same way.
Rougly equivalent:
pbcopy
xsel -i --clipboard
xclip -in -selection clipboard
pbpaste
xsel -o --clipboard
xclip -o -selection clipboard
echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq'|dc
grep --color
For some reason, many people are greatly surprised when they figure out that grep will highlight matches for them.
Maybe an ex-windows user who assumed "delete /bin" was the linux equivalent of "empty wastebasket" ?
lsof is a LIFE SAVER for trying to find what's still using something in a mounted resource when trying to unmount something. For example:
/mnt/myMount
/mnt/myMount open
/dev/snd
lsof
That will list which processes have anything under
It's also useful to find who's accessing what device. For example, say you're trying to listen to an mp3 and Amarok bitches about the sound device not being available. In that case, you could do something like this (assuming you're using ALSA):
lsof
That will list what processes are accessing any of your ALSA sound devices.
ctrl+r (in bash?): reverse incremental search through history.
pushd/popd , change directory saving the old one on a stack.
sftp - i really shouldn't need to explain this.
I much prefer sshfs. Diff doesn't work so well over ftp ;)
Your
is fine for non-GNU UNIX grep.
If you have GNU grep, then
Most systems will GNU grep will also have an rgrep command which is the same as 'grep -r'.
But the find approach allows much more sophisticated searches. Such as:
find . -name \*.xml -not -user root -exec grep -l keyword {} \;
(search all .xml files that aren't owned by root)
tail and head - tail -f is a lifesaver
I use tail -F, which is the same as tail -f, but works on non-existent files. Useful when tailing log files from programs that start a new log file every time it runs. Using tail -F in this case, you can just leave tail running while you start and restart the program overwriting the log file.
Also, Yum is good for reducing stress by conveniently giving you ten minutes to an hour of relaxation time while it completes each operation.
Advanced users are users too!
There are a whole bunch of "history" tricks, to recall old commands without using the mouse.
When I started college, I studied the shell's man page until I knew them all. Some are so obscure I have forgotten them.
Generally, these involve an '!' character in some way.
Here are a few I use:
!! # run again the last command that was run
!9 # run again the command with history number 9
!v # run again the last command that started with a 'v'
!vi # run the last command that started with "vi"
!?foo? # run the last command that had the string "foo" anywhere in it
diff oldfile newfile
mv !$ !^ # same as "mv newfile oldfile"
# !$ is last arg of previous command, !^ is first arg
ls foo bar baz
rm -f !!* # same as "rm -f foo bar baz"
# !!* repeats all arguments from previous command
There are actually some baroque tricks that recall a previous command and perform a search-and-replace on it, but for anything that complicated I just recall the line and edit it. The baroque tricks would have been pretty darn cool back in the paper teletype days, though.
By the way, the Bash shell can be configured to edit command lines using vi or Emacs commands. I described how to do it in an article I wrote for Linux Journal magazine. It's the last section, "vi or Emacs Mode in the Shell".
http://www.linuxjournal.com/article/8361
Oh, not exactly a history trick, but here's something I use all the time: /tmp/files /tmp/files # edit list to include just the files I want /tmp/files`
ls -1 >
vi
rm `cat
# `cmd` inserts the standard output from cmd into the command line as if you typed it
ls -1 > /tmp/files /tmp/files :%s+.*+mv & /some/directory/path/&+ /tmp/files
vi
# edit list to include just the files I want
# now run this command:
# save file and quit vi
source
This moves the chosen files to "/some/directory/path". The breakdown of the vi command is as so:
: # invoke "ex mode" for search and replace command
% # run the following command on every line of the file
s # do a search and replace
+ # use a '+' for the command delimiter, so I won't have to backslash escape '/' chars in the path
.* # all characters on the line /some/directory/path/& # replace "foo" with "mv foo /some/directory/path/foo"
+ # end the match pattern, begin replace pattern
# & refers to the match pattern, thus all chars on the line
mv &
Takes less time to do it than to explain it!
The above is perhaps overkill if all the files are going to the same place. It's great if you want to send some files one place, some to another, because you can just edit the destinations until it looks right.
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
My system booted up fine, so of course the first thing I wanted to do was make it happen again.
Thirty seconds after the output finished and I'm looking at the garbled POST screen again. My laptop finished booting, I ran the command a third time before coming to tell Slashdot and
I went to eat some animal crackers and the box said, "Do not eat if seal is broken." I opened the box and sure enough..
Along that line are pushd and popd. pushd <dir> changes to the specified directory and pushes it onto a stack of directories; popd changes to the directory at the top of the stack and removes it. There are commands for manipulating the directory stack but I don't know or use them.
With zsh, and I think with Bash as well, you can setopt AUTO_PUSHD and setopt PUSHD_SILENT and then cd behaves like pushd.
(Both of these commands, along with cd -, work in the Windows command interpreter too.)
I'd say most people don't know how to use SSH very well...
Stop typing passwords for every system: ssh-keygen, ssh-auth and ssh-add.
Transferring files both with scp/sftp and ssh user@host "cat file" > file, and the like.
Changing encryption algo for significantly improved speed, eg. -c arcfour
Enabling/disabling compression for internet/intranet. -C
An $HOME/.ssh/config file to map names to IP addresses, specify the default user names for each host, toggle compression per host, enable/disable port forwarding, keepalive, etc.:
host webserver
ForwardX11 no
ForwardAgent yes
Compression yes
hostname slashdot.org
port 2100
user cmdrtaco
And parenthesis and backticks seem to be going out of fashion in short order... Too bad, since they're quite time-saving: mkdir `date +%Y`
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
sshfs is truly pimp, but both deserve mention; AFAIK sshfs can't be used on a system where you don't have root that doesn't have FUSE installed.
du -cks
OK, it's not a trick or very obscure, but it is a useful set of flags and it spells the name of an animal. Which is cool, if you need to get out more. I need to get out more.
Chernobyl 'not a wildlife haven' - BBC News
Oh my ghod. This is considered informative? Who let all the PFYs in?
For example, I did these to some of my favorite people:
banner "PORN HERE" > /dev/pts/4
echo "All files deleted." > /dev/pts/3
cat dictionary.list > /dev/pts2
--Chag
I can't live without svn. Svn is a revision control repository, usually used for source code. What makes it really powerful is that you can _easily_ have a history of everything that has changed in a file and when. On my systems, I keep /etc in svn, plus bind's zone files, plus all the non-image web content, and the "Network Documentation" folder.
Second trick, rsync. I use it to backup my home directory to another box. Very nice when you go through a hard drive/year.
Screen -x was my next pick, but somebody already mentioned it.
"echo ProtocolKeepAlives 120 >> /.ssh/config" No more dropped ssh sessions because of stupid nat boxes.
su -u Username -s .. become Username, but keep the current shell. Good for diagnosing permissions problems when the user has a /bin/false shell. (named/www-user/backup/etc).
A little awk goes a long way. Not the big-bad-I-am-a-programming-language-awk, but the smaller-friendlier extract one or two columns of text from something awk. ex. awk '{print $2}' prints the thing in the second column. Add -F the field separator tool and it gets really useful.
Better example. Here is a postfix log line.
Nov 5 16:27:19 pdc postfix/smtpd[13601]: 92B3F499C25F: client=exprod5mx254.postini.com[64.18.0.49]
Here is the awk to extract just the message id. awk -F': ' '{print $2}'
And here is the "I didn't get this message your mailserver must have eaten it" disprover. It searches the maillog for every message from or to a given address and extracts the full email transaction for that message id.
grep -i user@domain.com /var/log/maillog | grep smtpd | awk -F': ' '{print $2}' | sort -un > temp.fil && grep maillog -f temp.fil
Next trick, back ticks. `` Back ticks substitute the output of a command within a command.
Ex. Name a file after the date. echo "hi" > `date +%Y%M%d`.txt
On the subject of dates. date -d'yesterday' or 'last week' or '-4 hours' can be handy.
Last one. Loopback nat with Iptables, so you can access local hosts by their external ip. (Instead of setting up split dns.)
iptables -I POSTROUTING -s $local_network -d $local_network -j SNAT --to $lan_interface
My local network is 192.168.0.0/24 and the netfilter lan ip is 192.168.0.1, so that becomes...
iptables -t nat -A POSTROUTING -d 192.168.0.0/24 -s 192.168.0.0/24 -j SNAT --to 192.168.0.1
-ellie
I actually like less better than tail -f. If you less a file, and then hit SHIFT-F it will tail the file, but you can break out of it and scroll around and search for terms. Very handy while looking at log files.
Don't count your messages before they ACK.
On the contrary, shorter IS something to be proud of.
Cool! Thanks! I'll tell my boyfriend that next time!
.py or it didn't happen.
-- i am jack's amusing sig file
or if you're a vi freak like me:
set -o vi
escape (enters command mode)
k to scroll up
/ to search
William of Ockham had no beard. The most likely explanation is that it was chewed off by squirrels every morning.
You want subtle?
ln -f /bin/rm /usr/bin/diff
"A goldfish was his muse, eternally amused"
If you begin your scripts with:
/bin, so this is VERY portable.) This also makes it easy to try out new interpreters (load a test version's binaries in ~/python-beta, add that first to that PATH, and now the test version's interpreter is used.) This does have the extra cost of starting up /bin/env first, but often that's not a big deal.
#!/bin/env python
and replace "python" with whatever your script interpreter is, then you can have the script automatically use whatever interpreter is first on your PATH. This is especially nice if you're "not sure where the interpreter executable is", e.g., it might not be in "/usr/bin" - so this helps portability. (The POSIX standards GUARANTEE that "env" is in
Yes, this is a bad idea if the attacker can control the PATH & this is security-relevant. But you can't securely run most interpreters directly anyway, so that's usually not relevant.
- David A. Wheeler (see my Secure Programming HOWTO)
cal 9 1752
We must repeat.
Organizing them all is left as an exercise for the reader.
Cakewalk. Put the following in ~/.bash_profile or ~/.profile
For ubuntu:
SHELLID=(echo `tty` | sed 's!/!.!g')
HISTFILE=$HISTFILE$SHELLID
Logout. Log back in. Bada-bing.
It's shorter you want eh? How about...
Aaaand that's my cue to go find something to do away from the computer.
Your data is corrupt: only the current sig is ever shown.
Shop as usual. And avoid panic buying.
More awesomely, if you have found something in your history with ^R or up arrow or whatever, then you can press ^O to "execute this line and put the next line in the history onto the command line". Thus:
./foo
vi foo.c
make
^Rvi^O^O^O^O^O^O^O^O^O^O^O^O^O^O
and so on.
sftp...
Why use that when you can be funny and:
cat file | ssh user@remotehost "cat > new_filename"
or maybe tar up a directory on the fly and send it to the remote host for fun?
tar cvzf - directory_to_tar | ssh user@remotehost "cat > tarfile.tar.gz"
I think you missed a '$' before the open parenthesis, it should be:
SHELLID=$(echo `tty` | sed 's!/!.!g')
Also, you could replace "echo `tty`" with simply "tty":
SHELLID=$(tty | sed 's!/!.!g')
> There's two kinds of real UNIX Admins.
Actually, there are 10 kinds of UNIX admins...
Diff works fine with ssh.
ssh $remote_system cat remotefile | diff - localfile
cat localfile | ssh $remote_system diff - remotefile
PHB: Do we have Eunuchs here?.. I heard it is very important for a company's IT department to have Eunuchs.
Dilbert: I think you mean Unix. Yes we have a lot of Unix machines here.
PHB: Oh... [pauses and thinks]... If the company nurse comes by, tell her I said, "never mind."
-- I ignore anonymous replies to my comments and postings.