Slashdot Mirror
Adding Some Spice To *nix Shell Scripts
An anonymous reader writes "Developing GUI script-based applications is time-consuming and expensive. Most Unix-based scripts run in a CLI mode or over a secure ssh session. The Unix shells are quite sophisticated programming languages in their own right: they are easy to design and quick to build, but they are not user-friendly in the same way the Unix commands aren't (see the Unix haters books). Both Unix and bash provide features for writing user friendly scripts using various tools to build powerful, interactive, user-friendly scripts that run under the bash shell on Linux or Unix. What tools do you use that spice up your scripts on the Linux or Unix platforms?"
On OS X, I use Pashua, http://www.bluem.net/en/mac/pashua/. This is a brilliantly simple thing to use. I also use it for other (non-script) languages for making a quick-and-dirty GUI that still looks nice and is a real Cocoa program.
I know this is troll-ish, but the way I view it a script is just that.. a script. A series of commands to be executed in a specific order designed to automate a repetative task. Basic logic, control, and input are generally ok.. but interaction is in my opinion an indicator that your task is out of scope for a "script" and should become a full fledged application.
(you may now freely argue amongst yourselves on the difference between a script and an application)
There are a metric ass-tonne of dialog-type apps out there .. just google for your favorite toolkits prefix and "dialog" and you'll probably find something..
gdialog
kdialog
xdialog
etc..
Limiting yourself much? Also *nix != bash and bash != *nix though I imagine all shells share a host of similar commands.
"There is a way that seems right to a man, but its end is the way of death." Proverbs 16:25 (NKJV)
None, I have given up bash scripting. The syntax and semantic are simply to wierd. And it can't handle filenames with space in them without some serious hack magic.
Maybe its time someone (re)-invent a total new shell, with a sane scripting language, commands with consistent names for the same arguments and in general something which don't feel like I live in 1980.
And I am a fulltime linux used, and a software developer, and I do use the shell as an interactive interface, but I newer script it, and I always have the feeling that I am using a 20 year old interface with so many issues that its insane. Kinda like the same feeling I get when I use sas(The static package. Nice features HORRIBLE interface)
I always request the budget for a small unit of scantily clad maidens from management in addition to the team beer budget.
One day.... one day....
Moved to http://soylentnews.org/. You are invited to join us too!
I work as a Linux netadmin and system developer, so I do a lot of shell programming in (ba)sh. Here's some of the niftier things you can do to a shell script:
- Make colored output with shell escape sequences. Works especially well with red type for error messages, makes them really stand out.
- Use inline functions a lot. The great thing about them is that you can pipe to them and use them in all kinds of places. For instance, to do mysql queries:
mysql_query() { /usr/bin/mysql --user=root --pass=topsecret database
}
echo 'SELECT * FROM accounts' | mysql_query
- "Here documents". For long MySQL sequences, something like the following (reusing the mysql_query function from above):
cat - EOF | mysql_query ...
SELECT bar
FROM foo
WHERE baz
EOF
This lets you easily format stdin for scripts and other programs. Also really useful for outputting HTML and stuff like that. Best thing is that variables are expanded inside the block.
- The || and && operators. Check if a file exists, remove if so, else complain: /tmp/somefile.txt ] && rm /tmp/somefile.txt || echo "Does not exist!"
[ -f
Also common in this form: /usr/bin/necessaryprogram ] || { echo "aaargh"; exit 1; }
[ -x
- Making a "multithreaded" shellscript is also one of my favourites. Say, you want to start five virtual machines at the same time. Write a function that starts a vm, and call it a few times in a loop, backgrounding each instance with &, and saving their PIDs. Then have a "wait loop" that waits for the PIDs to exit the system (or for a timeout to occur).
- Proper argument handling with getopt. Have your script take "real" arguments in any order, just like real binaries.
This just scrapes the surface of the surface, of course. I learn new stuff every day.
I use the usual: sed, [, wget, etc to automate downloads of pr0n.
I disagree with the premise that GUI interfaces are needed, desirable, or constitute "spicing up." Command line scripts are fine and dandy in my book.
CG Pin-Ups?
The shell is a poor clone of 1950's algol. Today, scripting in Ruby or Python yields scripts that can handle errors with advanced facilities like exceptions, and is more maintainable, and can connect to a number of different GUIs or the web.
Bruce Perens.
Combined with red and blue text the goggles make my facepalm ASCII art really pop!
You see, I use ASCII art in lieu of the dialog boxes for user feedback. It's more intuitive to show facepalm guy when I ask the user for a digit & they give me a letter. They understand right away that they're an idiot.
The CLI is powerful because it's a CLI, you do not need or want pretty dialog boxes. Help is whats available with man --help usefull errors messages and the contents of /var/log. It works over 9600 baud serial and works pretty well so you can ssh from your smartphone with 1 bar and fix something at 3am before the GUI would have time to come up to a login screen. A good CLI expects things to be piped into and out of it and can get any required information via the command line. The power of the CLI is that you can chain bits together run to do things or wrap scripts around other scripts and do useful work.
You point to a 20 year old book that mostly bitches about how slow/ugly X is, guess what things have come a long way, I run one laptop with native X and it looks good is responsive I export X all the time over ssh to my primary desktops. Take a step back and think why your trying to shoehorn GUI functions onto a CLI if you really need to do it look at some of the toolkits that can detect if there is a X server present and use that fallback to text gui and run entirely headless by pure command line but think long and hard about why you would want to do this.
No sir I dont like it.
Here are some random things I find useful, related to user interaction (mostly becuase it notifies the user):
Oven timer:
sleep $((20*60)); xmessage "Dinner is done"
Quick macro for automating some repetitive task in a program:
xdotool type "something"; xdotool key Return; xdotool mousemove $x $y; xdotool click 1; (and so on)
Copying a file to/from the clipboard (can also copy from /topipe, so the output of any command). Faster than opening a text editor:
xclip -in file
Notifying me when some specific thing changed on a website:
CHECKLINE="$(curl -s http://somewebsite.org/somepage.html | grep "currently undergoing maintenence")"
while true; do
sleep 120
[ -z "$CHECKLINE" ] && xmessage "somewebsite is open again" && exit
done
Or just checking for changes in general (I use this for notifying me when something changed when tracking something I ordered, so I know the minute the package is ready to get picked up at the post office):
while true; do
OLD_MD5=${MD5}
CONTENT=$(elinks -dump 1 -dump-charset iso-8859-1 "http://someurl.com/track?id=someid")
MD5=$(echo -n $CONTENT | md5sum -)
[ "${MD5}" != "${OLD_MD5}" ] && { :\n\n${CONTENT}")"
xmessage "$(printf "New action:
}
sleep 120
done
If you don't want to interrupt what you're doing with a pop-up you can pipe it to osd_cat instead to have the text appear over whatever program you're currently working with. Adding a few beep; beep; beep; beep; is also a good way to get your attention if you're not paying 100% attention to your computer all the time.
Posted by a Debian GNU/Linux user
Linux Shell Scripting with Bash
by Ken O. Burtch
Sams Publishing
One of only two "computer" books I've ever been able to just sit down and read rather than just using as reference (the other being Kathy Sierra's "Head First Java" -- which is amazing).
Ken does a fantastic job at putting "just the right" level of background, detail, context, and and depth for someone new to shell scripting to get started, then to use the book as a reference for all the traditional tools (sed, awk, etc..).
I've bought two copies, one for me and one I gave to someone else who wanted to learn how to do this stuff.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
I know this is like cursing in the church, but I use VB for most tasks others would use shell scripts for. Why? For one, the syntax is more predictable. With Bash you always have to worry about special characters and I can't stand that. (Same reason I dislike Tex.) Secondly, if you need user interaction, it has a really easy to use GUI builder. When VB4 came out it was like 1995 or something. It is now 2010 and in my opinion, for building simple dialogues (or even not so simple ones) VB is still among the best. That said, it has downsides, the most important one is that piping is missing by default. However, there are Win32/Wine functions you can call to alleviate this. Put them in a standard module with some wrappers and it's like they're part of the language.
Thanks, it's nothing I couldn't show a fella. Learnt a lot from my colleagues and from the O'Reilly 'Unix Power Tools' book. The Advanced Bash Shell-scripting Guide is pretty good (but chaotic) too.
The syntax filter here munged some of the examples, though. The here document example will not work as-is, because there should be two 'less-than' signs in front of the minus sign. The mysql_query function probably also won't work (can't bother to run a test), because the newline after the first bracket mysteriously disappeared. So best to loop up the concepts in some kind of reference manual.
Honestly it its just about adding a button so that its not necessary to remember the command line arguments/switches, i prefer tcl/tk. Lightweight, portable (and ported), and stable. And if you need a little more functionality, there are tons of libraries available.
The way I look at it is this: the "interaction" may actually be with another script. The whole abstraction that Unix-like OSes enforce, at least with file based IO, is that it is irrelevant what is on the other side of a file descriptor -- a disk, a pipe, a user, a socket, or something else entirely.
Of course, this all starts to break down with GUIs.
Palm trees and 8
For Unix shell scripting purposes (and I know the Slashdot crowd may scoff at this but), nothing compares to KSH. It has many features not found in Bash and most other shells, such as coprocesses, associative arrays, compound variables, floating point arithmetic, discipline functions, etc. It's also fully extensible and posix compliant. For GUI scripts, almost all commercial Unixes include dtksh, which provides access to much of the Xt and Motif APIs. A TK version of ksh also exists.
KSH just gets a bad rep because Unix vendors insist on only supplying an ancient version (ksh88), or its clone (pdksh) that lacks all of the functionality and behavior of the original. As a result most people have never used a modern version of the shell.
Of there's a right tool for the right job. Depending on the nature of the task one might also want to consider perl, python, or some other scripting technology.
As said previously, scripts are scripts and don't often need a GUI. But for grep's sake, make them consistent!!! The only spicing up _really_ needed are some standards:
... and the most annoying thing of all - make sure --help _always_ works, even if the script body itself can't - at least the user can then be told about what the prerequisites are.
o output errors to STDERR; normal output to STDOUT
o include (-h, --help) processing - and send it to STDOUT so the help can be piped to 'less'
o use getopt(1) or process-getopt(1) so that options on the CLI parse in a predictable and standard way
o keep it terse except for errors so that the user can easily see if it worked or not without scanning vast output
o provide a --verbose option to help with tracking down those errors
Head over to http://mywiki.wooledge.org/BashFAQ for much wisdom on how to write better bash scripts.
Here's a neat trick to access the output of commands as file handles:
diff <( echo 'hello') <( echo 'world')
Now that I've got your attention ;) I'll take this opportunity to plug my open source bash libraries:
bash-script-lib, a collection of scripts that let you augment your own scripts with advanced capabilities:
bash-sys-manage, a collection of scripts that lets you manage VPS instances by installing components and backing up and restoring discrete aspects of a server. E.g.:
install.sh system.apt system.locale system.users system.nginx nginx.config packages.utils.base packages.utils.build php.package php-fm.build apc.package memcache.package
backup.sh system.users system.config mysql.database
I will quickly write a shell script any time I have some simple task I want to automate. You cannot beat the convenience:
/some/directory/$1
cd
some_program --foo $2 --bar $3
rm -f *.temp
Wow, three lines, and it runs the program, then cleans up the temp files that program always litters in my directory. And I don't have to memorize the --foo and --bar options! Shell scripts rock!
The problem comes when you start to do nontrivial things. When you start processing lists of files, and the files can contain spaces, the amount of quoting drives me insane. At that point I rewrite in Python.
The spaces-in-file-names problem can bite even this trivial shell script! If any of the three arguments ($1, $2, $3) is specified as a string containing spaces, this script won't work, because the shell interpreter needs quotes at every step where it evaluates something. If you pass "my file.txt" as the second argument, the $2 won't evaluate to "my file.txt" in quotes, it just evaluates to the bare string. So to be fully safe, the above program needs to be:
/some/directory/"$1"
cd
some_program --foo "$2" --bar "$3"
rm -f *.temp
And woe is you if you forget the quotes.
Python loses in convenience for running a program... here's a Python equivalent of the above:
import os
import subprocess as sp
import sys
os.chdir("/some/directory/%s" % sys.argv[1])
lst_args = ["some_program", "--foo", sys.argv[2], "--bar", sys.argv[3]]
sp.check_call(lst_args)
lst_args = ["rm", "-f", "*.temp"]
sp.check_call(lst_args, shell=True) # run in a shell to get wildcard expansion
At first glance this looks horrible. It's much more than the three terse lines of the original. But it's easier to get right, and this is safer to run. If the user specifies something silly for the first arg, or doesn't provide it, this program will immediately stop after trying to change directories. The original would change to "/some/directory" and blindly run on, trying to run "some_program" there, and who knows what would happen? Likewise, if "some_program" fails, this script will stop immediately, and the deleting of the *.temp files will not occur (making it easier to debug what's going on). Finally, in this code we don't have to worry about quoting the arguments; we can just use the arguments and it just works. It is much harder to write a fail-safe shell script: you would have to explicitly test that $1 is provided, and you would have to check the result of running "some_program" to see if it failed or not.
The nontrivial scripts I write tend to have a lot of logic in the scripts themselves, and Python is much much more pleasant and effective for evaluating the logic. If I want to write a script that sweeps through a bunch of directories and deletes files that match certain criteria, it is so much easier to write the tests on the file in Python. If I write ten lines of "if" statements to look at a filename, that is ten lines where I didn't need to fuss with the double quotes. In Python, you can do things like
junk_extension = (".temp", ".tmp", ".junk")
if filename.endswith(junk_extension):
os.remove(filename)
Shell scripting cannot match this convenience. And note that if I use the native Python os.remove() I don't need to worry about quoting the filename; it can have spaces in it and os.remove() doesn't care.
Other people might prefer to use Perl or Ruby. Either of those, or Python, are much better than shell scripts for anything nontrivial.
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
I use sh and relatives (and vi) because they're ubiquitous, stable, small, light, and reasonably fast, consistent, capable, and fairly understandable. Every program in /etc is a shell script, and by default system utilities such as cron call on sh. Everything entered at a command line is interpreted by sh. sh is as much a part of UNIX systems as C. You might as well suggest GNU/Linux be rewritten in a better language than C.
And if you're going to suggest that, why not also reexamine the basic architecture of UNIX? If anyone produces an open, formally verified microkernel OS in Haskell that actually works, isn't dog slow, and has sufficient functionality and apps to be useful, I'll surely check it out. I'd love to see more consistency between how applications accept parameters from the command line and how programming languages handle parameters. The former tends to be named and unordered, while the latter is anonymous and ordered. Then there's the defacto standard for libraries, worked out in the days when memory and disk space was extremely limited. It doesn't support enough meta information, making it necessary for a compiler to read header files. It's made libraries many little worlds of their own. As long as a programmer sticks to C/C++, it is relatively easy to call C library functions, but step outside that and it becomes a huge pain. Therefore we have these monstrous collections of duplicate functionality and wrapper code such as CPAN, abominations such as SWIG, attempts to bridge things by providing some commonality and standardization such as CORBA, and separate worlds such as the gigantic collection of Java libraries.
Something like Perl or Java is heavy enough to be impractical on a slow computer with little RAM. Can take over 5 seconds just to load the language. I'm not familiar enough with Python or Ruby to know if they're as heavy. You can't always be sure they're there, whereas whatever was used in /etc/rc.d, and is run in a terminal, is guaranteed to be present. Don't know about a "pysh", but there is a "perlsh", for use in a terminal. Never seen perlsh used though, and it seems to demand a nasty hackish sort of interaction. Press Enter twice to execute commands, as one press of Enter is apparently used as a statement or formatting break. Maybe that's because those languages actually aren't too suitable for an interactive environment? As to connecting to the web, there's wget, wput, and curl.
It could be a lot worse. Bash is pretty nice compared to MS DOS batch language.
Intellectual Property is a monopolistic, selfish, and defective concept. It is "tyranny over the mind of man"
something like "for file in *" in bash will ignore special characters and run the loop once, for each actual file.
bash's for loop understands * as a special case. if you need something like "for file in $(find ...);" you'll get one loop per word again. Also, even when you get one loop per file, you still have to quote $file when you use it because bash parses arguments to the command after variable substitution, so something like touch $file when $file is foo bar becomes touch foo bar where foo and bar are separate arguments, rather than what most people would expect (that the value of $file would be passed to the command as a single argument)
Removing space from $IFS (the "standard" way is to make it tab and newline: IFS=$(echo -en "\n\b") ) fixes many of these quirks.
If I have been able to see further than others, it is because I bought a pair of binoculars.
The first rule of shell scripts is "you don't write programs in shell." /bin/sh. Your script probably contains bashisms and you don't even know it. What's more, bash has some great features that are only available if you use it explicitly in your shebang.
The second rule of shell scripts is "you DON'T WRITE PROGRAMS IN SHELL." Seriously. You want perl or some other high level language.
The third rule is to start your script with "#!/bin/bash", not
The fourth rule would be to read the bash faq at http://mywiki.wooledge.org/BashFAQ as it contains many tips and tricks that won't be obvious just by reading the man pages.
I would, don't get me wrong. Emacs is a lovely operating system. I just wish it had a decent text editor.
Bash scripting definitely has everything you'd need to write an "application" in, but many data constructs would be awkward to implement in Bash, so you'd use Python, Perl, or Ruby.
But what I can do in Bash I could also do in Ruby or Python very easily. What I could do in Ruby or Python would be very difficult to do in Bash.
Then you have Java and C++ which are clearly not scripting languages, but I could do everything there that I could do in Bash. On the other hand, most things interesting you can do with C++ would be next to impossible with Bash. You just can't beat the performance of compiled languages. On the other hand, development would be costlier and portability might be an issue.
Then again, you can tap a thumb tack in with a sledge hammer, but you run the risk of putting a nasty hole in your wall!
It all comes down to what is the right tool for the job. So many tools, though! So pick and choose wisely. But you knew that already.
Personally, I have given up on making a distinction between what is a "scripting language" and what's an application language. Javascript(!) is a "scripting language", but there is a high level of interactivity in the applications it's used for typically. Both in communicating with the server (AJAX) and with the user.
Many of these so-called "scripting languages" allows for object-oriented programming. Bash, of course, does not. But then it was never meant for that level of sophistication. But that's even more blur for you. But you knew that already too.
Ruby Neural Evolution of Augmenting Topologies
I have written some pretty sophisticated shell programs, including one thing for Debian that took a set of executables and a set of shared libraries and made versions of the libraries containing only the functions called for in the executables. It got the system to fit on a floppy when it otherwise would not have. It did a lot of list-processing in shell. Fortunately, someone eventually replaced that with a non-sh program.
At this point I would not write another large shell script of any kind, and am mostly doing the three-liners in Ruby as well.
The reason you would learn the shell would be to pass a certification test. You would be a better sysadmin or programmer if your main scripting language was Python or Ruby.
Bruce Perens.