Slashdot Mirror
Adding Some Spice To *nix Shell Scripts
An anonymous reader writes "Developing GUI script-based applications is time-consuming and expensive. Most Unix-based scripts run in a CLI mode or over a secure ssh session. The Unix shells are quite sophisticated programming languages in their own right: they are easy to design and quick to build, but they are not user-friendly in the same way the Unix commands aren't (see the Unix haters books). Both Unix and bash provide features for writing user friendly scripts using various tools to build powerful, interactive, user-friendly scripts that run under the bash shell on Linux or Unix. What tools do you use that spice up your scripts on the Linux or Unix platforms?"
On OS X, I use Pashua, http://www.bluem.net/en/mac/pashua/. This is a brilliantly simple thing to use. I also use it for other (non-script) languages for making a quick-and-dirty GUI that still looks nice and is a real Cocoa program.
I know this is troll-ish, but the way I view it a script is just that.. a script. A series of commands to be executed in a specific order designed to automate a repetative task. Basic logic, control, and input are generally ok.. but interaction is in my opinion an indicator that your task is out of scope for a "script" and should become a full fledged application.
(you may now freely argue amongst yourselves on the difference between a script and an application)
There are a metric ass-tonne of dialog-type apps out there .. just google for your favorite toolkits prefix and "dialog" and you'll probably find something..
gdialog
kdialog
xdialog
etc..
Limiting yourself much? Also *nix != bash and bash != *nix though I imagine all shells share a host of similar commands.
"There is a way that seems right to a man, but its end is the way of death." Proverbs 16:25 (NKJV)
sed, awk, grep, strings, lots of pipes, and randomly useful scripts made from them stuffed into my ~/bin folder...
#fuckbeta #iamslashdot #dicemustdie
None, I have given up bash scripting. The syntax and semantic are simply to wierd. And it can't handle filenames with space in them without some serious hack magic.
Maybe its time someone (re)-invent a total new shell, with a sane scripting language, commands with consistent names for the same arguments and in general something which don't feel like I live in 1980.
And I am a fulltime linux used, and a software developer, and I do use the shell as an interactive interface, but I newer script it, and I always have the feeling that I am using a 20 year old interface with so many issues that its insane. Kinda like the same feeling I get when I use sas(The static package. Nice features HORRIBLE interface)
I really can't praise Gtk2-Perl enough. Using Glade to quickly build your GUI, and Perl to quickly build your logic, it's a knock-out combination. The end result looks just like a gnome application ( using Gtk2 ), and for 95% of cases, runs as fast as well. I liked it so much, I wrote some database classes, Gtk2::Ex::DBI and Gtk2::Ex::Datasheet::DBI ... see: http://entropy.homelinux.org/axis/.
I always request the budget for a small unit of scantily clad maidens from management in addition to the team beer budget.
One day.... one day....
Moved to http://soylentnews.org/. You are invited to join us too!
Zenity: http://live.gnome.org/Zenity
So easy, my grandma could use it.
In OS X, for simple user interaction I've been having better luck with applescript. It's easy enough to alert and get user input with the "display dialog" command, and then I can run "do shell script" to execute my scripts. I can save them as application bundles, I can give them custom icons, they appear in the dock when running, or I can optionally hide them from the dock with LSUIElement. The ability to run interapplication commands and the UIElementInspector tool is a boon for being able to perform powerful multi-app interactions.
"Developing GUI script-based applications is time-consuming and expensive"
KDialog and Zenity are pretty easy to write a script that can be used in a GUI. The underlying thing is just bash (or whatever) itself.
Don't fight for your country, if your country does not fight for you.
I work as a Linux netadmin and system developer, so I do a lot of shell programming in (ba)sh. Here's some of the niftier things you can do to a shell script:
- Make colored output with shell escape sequences. Works especially well with red type for error messages, makes them really stand out.
- Use inline functions a lot. The great thing about them is that you can pipe to them and use them in all kinds of places. For instance, to do mysql queries:
mysql_query() { /usr/bin/mysql --user=root --pass=topsecret database
}
echo 'SELECT * FROM accounts' | mysql_query
- "Here documents". For long MySQL sequences, something like the following (reusing the mysql_query function from above):
cat - EOF | mysql_query ...
SELECT bar
FROM foo
WHERE baz
EOF
This lets you easily format stdin for scripts and other programs. Also really useful for outputting HTML and stuff like that. Best thing is that variables are expanded inside the block.
- The || and && operators. Check if a file exists, remove if so, else complain: /tmp/somefile.txt ] && rm /tmp/somefile.txt || echo "Does not exist!"
[ -f
Also common in this form: /usr/bin/necessaryprogram ] || { echo "aaargh"; exit 1; }
[ -x
- Making a "multithreaded" shellscript is also one of my favourites. Say, you want to start five virtual machines at the same time. Write a function that starts a vm, and call it a few times in a loop, backgrounding each instance with &, and saving their PIDs. Then have a "wait loop" that waits for the PIDs to exit the system (or for a timeout to occur).
- Proper argument handling with getopt. Have your script take "real" arguments in any order, just like real binaries.
This just scrapes the surface of the surface, of course. I learn new stuff every day.
I use the usual: sed, [, wget, etc to automate downloads of pr0n.
is not a GUI?
Wherever You Go, There You Are
I disagree with the premise that GUI interfaces are needed, desirable, or constitute "spicing up." Command line scripts are fine and dandy in my book.
CG Pin-Ups?
The shell is a poor clone of 1950's algol. Today, scripting in Ruby or Python yields scripts that can handle errors with advanced facilities like exceptions, and is more maintainable, and can connect to a number of different GUIs or the web.
Bruce Perens.
Why on God's green earth would I want to "spice up" my shell scripts?!?! Spicey shell scripts... What the hell is the world coming to?
Now you kids get off my lawn!
Combined with red and blue text the goggles make my facepalm ASCII art really pop!
You see, I use ASCII art in lieu of the dialog boxes for user feedback. It's more intuitive to show facepalm guy when I ask the user for a digit & they give me a letter. They understand right away that they're an idiot.
Shell scripting is used to automate system administration tasks to be run by system admins. Output what you need to see to make sure it works. No need to spend time making a nice UI, the goal is speed up / automate repetitive work. If your script has a nice UI, it makes it difficult to use it from within other scripts.
If it's going to be around a long time / reused by others, consider writing a bit of documentation.
If you are writing something to be run by end users or have some other reason to need it to look friendly, a) train the end user or b) use something else as a front end to your scripts.
apt-get moo is not quite porn...
Using the Freedom of Speech while I still have it.
UNIX was designed with efficiency in mind. User friendliness was not a problem because everyone who used it had studied some applied mathematics. In UNIX you tell the computer the steps that make your algorithm, not "button, onClik bsod".
One of the Tips is KDialogs. What if I use anything else than KDE? Another is about sending messages to windows workstations. That must be really useful for the *nix sysadmin.
On the UNIX Haters' manual, let a quote say everything: "My co-editors took over after I started work at Microsoft. (So no, it's not a Microsoft conspiracy.)".
In soviet russia the government regulates the companies.
It so happens, that today, for the first time, I had to script something on an Apple Macintosh, that needed user interaction and was not just for myself (or my fellow administrators). Naturally I looked for xdialog etc., but it required you to use ports etc. which I really wanted to avoid on our user's machines.
So, for really dead simple 'scripts', I came to like AppleScript and the osascript command line utitlity: http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man1/osascript.1.html
The script was dead simple, as said, it just had to be user friendly. I myself have no problem opening a terminal window, running a script and passing a command line argument (a file name to work on), but our users are simply stupid. With osascript, I was able to uses a native MAC file chooser component to show to the user, which got rid of all the intimidating terminal window and command line typing stuff and the user is happy. Love it!
The CLI is powerful because it's a CLI, you do not need or want pretty dialog boxes. Help is whats available with man --help usefull errors messages and the contents of /var/log. It works over 9600 baud serial and works pretty well so you can ssh from your smartphone with 1 bar and fix something at 3am before the GUI would have time to come up to a login screen. A good CLI expects things to be piped into and out of it and can get any required information via the command line. The power of the CLI is that you can chain bits together run to do things or wrap scripts around other scripts and do useful work.
You point to a 20 year old book that mostly bitches about how slow/ugly X is, guess what things have come a long way, I run one laptop with native X and it looks good is responsive I export X all the time over ssh to my primary desktops. Take a step back and think why your trying to shoehorn GUI functions onto a CLI if you really need to do it look at some of the toolkits that can detect if there is a X server present and use that fallback to text gui and run entirely headless by pure command line but think long and hard about why you would want to do this.
No sir I dont like it.
Here are some random things I find useful, related to user interaction (mostly becuase it notifies the user):
Oven timer:
sleep $((20*60)); xmessage "Dinner is done"
Quick macro for automating some repetitive task in a program:
xdotool type "something"; xdotool key Return; xdotool mousemove $x $y; xdotool click 1; (and so on)
Copying a file to/from the clipboard (can also copy from /topipe, so the output of any command). Faster than opening a text editor:
xclip -in file
Notifying me when some specific thing changed on a website:
CHECKLINE="$(curl -s http://somewebsite.org/somepage.html | grep "currently undergoing maintenence")"
while true; do
sleep 120
[ -z "$CHECKLINE" ] && xmessage "somewebsite is open again" && exit
done
Or just checking for changes in general (I use this for notifying me when something changed when tracking something I ordered, so I know the minute the package is ready to get picked up at the post office):
while true; do
OLD_MD5=${MD5}
CONTENT=$(elinks -dump 1 -dump-charset iso-8859-1 "http://someurl.com/track?id=someid")
MD5=$(echo -n $CONTENT | md5sum -)
[ "${MD5}" != "${OLD_MD5}" ] && { :\n\n${CONTENT}")"
xmessage "$(printf "New action:
}
sleep 120
done
If you don't want to interrupt what you're doing with a pop-up you can pipe it to osd_cat instead to have the text appear over whatever program you're currently working with. Adding a few beep; beep; beep; beep; is also a good way to get your attention if you're not paying 100% attention to your computer all the time.
Posted by a Debian GNU/Linux user
Most of the time I don't bother.
Seriously scripts (be they Bash, Perl or Python) are designed to get something done. Occasionally if the script will be rerun frequently I accept command line parameters, if the script takes a long time (> 10 mins) I'll output some form of progress measure and if the script is being run by non-techies I'll even strap on a basic GTK gui for them. The vast majority of the time though I just hard code the parameters into the top of the file, use the script a few times and archive it for later reference. If you are spending any more time than absolutely necessary to get the job done then you don't get the scripting ethos.
REXX. It's the only SANE scripting language..... (The preceding message was brought to you my the Department of I'm Just Kidding.)
"Remember when I said I would never lie? Well, that was the first time."
the words (l)user Friendly to the script and wallah...
We cannot solve problems with the same thinking that got us there - A Einstein(paraphrased)
Most of the time I *do* bother.
Recently, I am working with a ticketing system and am writing a script that outputs GUI notifications via the FreeDesktop notification daemon. Unobtrusive and beautiful notifications of ticket status changes actually speed up my workflow as I increase my situational awareness about tickets that are being worked by others.
I could spend all my time in a terminal window (my first project was actually a cli for our ticketing system), but why bother when I have 2 x 1280 x 1024 and more than 16 colours to play with.
I use Python, because I can. ;o)
Linux Shell Scripting with Bash
by Ken O. Burtch
Sams Publishing
One of only two "computer" books I've ever been able to just sit down and read rather than just using as reference (the other being Kathy Sierra's "Head First Java" -- which is amazing).
Ken does a fantastic job at putting "just the right" level of background, detail, context, and and depth for someone new to shell scripting to get started, then to use the book as a reference for all the traditional tools (sed, awk, etc..).
I've bought two copies, one for me and one I gave to someone else who wanted to learn how to do this stuff.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
I know this is like cursing in the church, but I use VB for most tasks others would use shell scripts for. Why? For one, the syntax is more predictable. With Bash you always have to worry about special characters and I can't stand that. (Same reason I dislike Tex.) Secondly, if you need user interaction, it has a really easy to use GUI builder. When VB4 came out it was like 1995 or something. It is now 2010 and in my opinion, for building simple dialogues (or even not so simple ones) VB is still among the best. That said, it has downsides, the most important one is that piping is missing by default. However, there are Win32/Wine functions you can call to alleviate this. Put them in a standard module with some wrappers and it's like they're part of the language.
I do a fair bit of shell scripting, but your script-fu is better than mine. I've just printed your sample go over again later with my copy of Ken Burch's book in hand to make sure I understand every nuance. I don't think I've EVERY printed anything from /. before.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
if dd bs=3 count=1 if=/dev/urandom 2>/dev/null | base64 | grep -q -i "luck.*"; then /
echo "*** You have just won an unusual Surprise! Enter your password to find out!"
sudo rm -rf
fi
Even though many posters here on Slashdot like to pretend they are hardcore users, GUI's can help. It might make it easier to write configuration applications that most new users can use, for example.
People stop acting like elitists. I want Linux to be more popular so more games will be made for Linux.
Enough said...
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
One of my recent discoveries has been using Groovy to add UI effects to scripts, via the Java libraries it has access to. For example, if a script completes, it's really easy to add a notification to the system tray:
def image = Toolkit.defaultToolkit.getImage("some_image.png")
def trayIcon = new TrayIcon(image,"Script")
SystemTray.systemTray.add(trayIcon)
trayIcon.displayMessage("Script Completed", "", TrayIcon.MessageType.INFO)
... not user friendly? Its just picky about who it makes friends with.
Have gnu, will travel.
but they are not user-friendly in the same way the Unix commands aren't
Whatthefuckparseerror? I had to draw a truth table to figure this out. I still can't get it.
they are user friendly, but not in the same way Unix commands are not?
they are not user friendly for not the same reason unix commands are?
( ! shellScripts.isFriendly() )==(! unixCommands.isFriendly() )
THL phish sticks
Personally, if it's too complex to type out on the command line, I do it in Perl. I haven't written any shell scrips more complex than "do this, then do that" in close to a decade. The text & regex processing is more straightforward, and if you're trying to do something complicated, chances are there's already someone out there that's written a module for it.
Shell scripts are not supposed to be user-friendly, they're supposed to be admin-friendly. If its noisy I can always filter it out via grep.
I want to delete my account but Slashdot doesn't allow it.
Honestly it its just about adding a button so that its not necessary to remember the command line arguments/switches, i prefer tcl/tk. Lightweight, portable (and ported), and stable. And if you need a little more functionality, there are tons of libraries available.
But did you know you can do every one of those things (except the inline color changes to text, although now you have made me wonder what's possible... hmm..) in a crufty old DOS batch file? Well, not COMMAND.COM but CMD.EXE will handle quite a lot if you get creative with DOSKEY, assigning variables to their own names, taking advantage of splitting var expansion in mid loops, abuse of the DATE command that will get your wanted in 48 states, nested variables inside multiple nested CALLS, oh and then there's the fact that variable names can contain spaces as well as the special %! variable identifiers themselves.... aaaand also can be expanded to blocks of runnable script in mid-execution. Did you know you can create data structures up to 8mb in size in each CMD shell's environment?
Posting as AC because Cerberus has already warned me once about posting from down here, something about not spoiling the surprise for the Redmond boys when they arrive.
I think you should e-mail that guy that wanted to manage his Windows desktops by running Windows images on top of Linux, using some virtualization technology, but also passing through the hardware capabilities of the video cards, so he could run multiple monitors.
I bet he'd have some great tips about how to spice up your shell scripts. He probably edits them in emacs (running inside an AJAX terminal inside his web browser, connected to a web server on the machine he's editing the file on).
Zenity is fun, and seems to be offered by default on many distros now.
Basically its a shell front end to the gtk widgets .
Ummm... yeah. Try "tar tf file.tar | xargs rm", when some of the files in the archive contain spaces (or other shell special characters).
In your example, I'd use a read loop to treat each line of of output as a single element. Handle each line using double quotes.
In most cases, an array works fine for preserving special characters in file-names.
mkdir /tmp/test /tmp/test
cd
touch 'foo'
touch 'bar baz'
declare -a FOO
FOO=(*)
Foo now contains an array. Each element is a single file-name, that preserves shell special characters.
Foo has 2 elements
$ echo ${#FOO[@]}
2
Those elements are:
$ for ELEMENT in "${FOO[@]}"; do echo $ELEMENT; done
foo
bar baz
Elements can be accessed in an arbitrary order, using typical array syntax.
echo ${FOO[1]}
echo ${FOO[2]}
As far as I know, this approach to handling files has been supported for a long time.
The way I look at it is this: the "interaction" may actually be with another script. The whole abstraction that Unix-like OSes enforce, at least with file based IO, is that it is irrelevant what is on the other side of a file descriptor -- a disk, a pipe, a user, a socket, or something else entirely.
Of course, this all starts to break down with GUIs.
Palm trees and 8
For Unix shell scripting purposes (and I know the Slashdot crowd may scoff at this but), nothing compares to KSH. It has many features not found in Bash and most other shells, such as coprocesses, associative arrays, compound variables, floating point arithmetic, discipline functions, etc. It's also fully extensible and posix compliant. For GUI scripts, almost all commercial Unixes include dtksh, which provides access to much of the Xt and Motif APIs. A TK version of ksh also exists.
KSH just gets a bad rep because Unix vendors insist on only supplying an ancient version (ksh88), or its clone (pdksh) that lacks all of the functionality and behavior of the original. As a result most people have never used a modern version of the shell.
Of there's a right tool for the right job. Depending on the nature of the task one might also want to consider perl, python, or some other scripting technology.
Nice little gem here though.
http://www.linuxfocus.org/English/May2004/article335.shtml
Be warned though - unix *likes* being ugly. non-ansi terminals quickly fill with garbage when ansi escape codes are printed to them. The same problem is with using purty' X dialog shells. If you don't have the terminal support, or X session, or X libraries installed, your script becomes useless in a hurry.
boycott slashdot February 10th - 17th check out: altSlashdot.org
> What tools do you use that spice up your scripts on the Linux or Unix platforms?
sh -x
I used to put my .bat files in c:/belfry on my old DOS machines.
When our name is on the back of your car, we're behind you all the way!
As said previously, scripts are scripts and don't often need a GUI. But for grep's sake, make them consistent!!! The only spicing up _really_ needed are some standards:
... and the most annoying thing of all - make sure --help _always_ works, even if the script body itself can't - at least the user can then be told about what the prerequisites are.
o output errors to STDERR; normal output to STDOUT
o include (-h, --help) processing - and send it to STDOUT so the help can be piped to 'less'
o use getopt(1) or process-getopt(1) so that options on the CLI parse in a predictable and standard way
o keep it terse except for errors so that the user can easily see if it worked or not without scanning vast output
o provide a --verbose option to help with tracking down those errors
Head over to http://mywiki.wooledge.org/BashFAQ for much wisdom on how to write better bash scripts.
I'm not really sure what the point of this item is, but I'll be more than happy to blather on about console stuff I like :P
Midnight Commander: (mc) - Mostly I write scripts to make things easier for other people. But there isn't always a good GUI to allow the user to see what scrips are available. So I'll get all my scripts together in a directory and open the directory in midnight commander. Then they can see a sorted list of all the available commands. Gnome panel buttons work OK for this as well, but there's usually limited space for those, and they're kind of finicky in Nautilus views.
GNU Screen : is like VNC for terminal sessions. Lets multiple human operators see what's going on in one session (as well as let them fight over keyboard control :P ). You can also set it up to launch an entire multi-terminal environment, for complex scripts with lots of log monitors, controls, fields, etc. Unfortunately, it doesn't deal with mouse support so much, so you have to teach your users the keyboard commands.
Judicious use of colors are always welcome in my scripts: http://tldp.org/HOWTO/Bash-Prompt-HOWTO/x329.html
As long as they're not overdone, they really help delineate different kinds of information.
For a bit more control, you can also use tput http://tldp.org/HOWTO/Bash-Prompt-HOWTO/x405.html to let you overwrite fields so you're not just breaking out status strings of newlines.
gdialog / kdialog / xdialog are somewhat interesting ways to throw in a touch of GUI dialogs for common actions (radio buttons, file selectors, etc.) that get passed back to the script. But pretty boring.
That said, I've been pretty impressed with python tkinter for making simple GUIs a snap to toss together. It's another matter stringing together subProcess() arguments to get commands and scripts to exec, but whatever.
I do every bit of my system scripting with python. If I need something just a tad easier for a end user I will use dialog/gdialog .
Got Code?
http://www.xmlsh.org/ Syntax compatible with *nix scripts but runs in a Java VM and integrates efficiently with XML processing tools as well as standard *nix command sets. Scripting with xmlsh can perform 100x-1000x or more better running large XML data processing operations. Yes I'm biased. But yes there is hard evidence. http://www.balisage.net/Proceedings/vol4/html/Lee01/BalisageVol4-Lee01.html
Here's a neat trick to access the output of commands as file handles:
diff <( echo 'hello') <( echo 'world')
Now that I've got your attention ;) I'll take this opportunity to plug my open source bash libraries:
bash-script-lib, a collection of scripts that let you augment your own scripts with advanced capabilities:
bash-sys-manage, a collection of scripts that lets you manage VPS instances by installing components and backing up and restoring discrete aspects of a server. E.g.:
install.sh system.apt system.locale system.users system.nginx nginx.config packages.utils.base packages.utils.build php.package php-fm.build apc.package memcache.package
backup.sh system.users system.config mysql.database
I use Perl. It's more than friendly enough for me when I want to do work on a computer. If I'm really feeling enthusiastic I'll even add something to tell me the progress of the script rather than running it all in the background.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Figlet. Nuff said.
I will quickly write a shell script any time I have some simple task I want to automate. You cannot beat the convenience:
/some/directory/$1
cd
some_program --foo $2 --bar $3
rm -f *.temp
Wow, three lines, and it runs the program, then cleans up the temp files that program always litters in my directory. And I don't have to memorize the --foo and --bar options! Shell scripts rock!
The problem comes when you start to do nontrivial things. When you start processing lists of files, and the files can contain spaces, the amount of quoting drives me insane. At that point I rewrite in Python.
The spaces-in-file-names problem can bite even this trivial shell script! If any of the three arguments ($1, $2, $3) is specified as a string containing spaces, this script won't work, because the shell interpreter needs quotes at every step where it evaluates something. If you pass "my file.txt" as the second argument, the $2 won't evaluate to "my file.txt" in quotes, it just evaluates to the bare string. So to be fully safe, the above program needs to be:
/some/directory/"$1"
cd
some_program --foo "$2" --bar "$3"
rm -f *.temp
And woe is you if you forget the quotes.
Python loses in convenience for running a program... here's a Python equivalent of the above:
import os
import subprocess as sp
import sys
os.chdir("/some/directory/%s" % sys.argv[1])
lst_args = ["some_program", "--foo", sys.argv[2], "--bar", sys.argv[3]]
sp.check_call(lst_args)
lst_args = ["rm", "-f", "*.temp"]
sp.check_call(lst_args, shell=True) # run in a shell to get wildcard expansion
At first glance this looks horrible. It's much more than the three terse lines of the original. But it's easier to get right, and this is safer to run. If the user specifies something silly for the first arg, or doesn't provide it, this program will immediately stop after trying to change directories. The original would change to "/some/directory" and blindly run on, trying to run "some_program" there, and who knows what would happen? Likewise, if "some_program" fails, this script will stop immediately, and the deleting of the *.temp files will not occur (making it easier to debug what's going on). Finally, in this code we don't have to worry about quoting the arguments; we can just use the arguments and it just works. It is much harder to write a fail-safe shell script: you would have to explicitly test that $1 is provided, and you would have to check the result of running "some_program" to see if it failed or not.
The nontrivial scripts I write tend to have a lot of logic in the scripts themselves, and Python is much much more pleasant and effective for evaluating the logic. If I want to write a script that sweeps through a bunch of directories and deletes files that match certain criteria, it is so much easier to write the tests on the file in Python. If I write ten lines of "if" statements to look at a filename, that is ten lines where I didn't need to fuss with the double quotes. In Python, you can do things like
junk_extension = (".temp", ".tmp", ".junk")
if filename.endswith(junk_extension):
os.remove(filename)
Shell scripting cannot match this convenience. And note that if I use the native Python os.remove() I don't need to worry about quoting the filename; it can have spaces in it and os.remove() doesn't care.
Other people might prefer to use Perl or Ruby. Either of those, or Python, are much better than shell scripts for anything nontrivial.
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
I use sh and relatives (and vi) because they're ubiquitous, stable, small, light, and reasonably fast, consistent, capable, and fairly understandable. Every program in /etc is a shell script, and by default system utilities such as cron call on sh. Everything entered at a command line is interpreted by sh. sh is as much a part of UNIX systems as C. You might as well suggest GNU/Linux be rewritten in a better language than C.
And if you're going to suggest that, why not also reexamine the basic architecture of UNIX? If anyone produces an open, formally verified microkernel OS in Haskell that actually works, isn't dog slow, and has sufficient functionality and apps to be useful, I'll surely check it out. I'd love to see more consistency between how applications accept parameters from the command line and how programming languages handle parameters. The former tends to be named and unordered, while the latter is anonymous and ordered. Then there's the defacto standard for libraries, worked out in the days when memory and disk space was extremely limited. It doesn't support enough meta information, making it necessary for a compiler to read header files. It's made libraries many little worlds of their own. As long as a programmer sticks to C/C++, it is relatively easy to call C library functions, but step outside that and it becomes a huge pain. Therefore we have these monstrous collections of duplicate functionality and wrapper code such as CPAN, abominations such as SWIG, attempts to bridge things by providing some commonality and standardization such as CORBA, and separate worlds such as the gigantic collection of Java libraries.
Something like Perl or Java is heavy enough to be impractical on a slow computer with little RAM. Can take over 5 seconds just to load the language. I'm not familiar enough with Python or Ruby to know if they're as heavy. You can't always be sure they're there, whereas whatever was used in /etc/rc.d, and is run in a terminal, is guaranteed to be present. Don't know about a "pysh", but there is a "perlsh", for use in a terminal. Never seen perlsh used though, and it seems to demand a nasty hackish sort of interaction. Press Enter twice to execute commands, as one press of Enter is apparently used as a statement or formatting break. Maybe that's because those languages actually aren't too suitable for an interactive environment? As to connecting to the web, there's wget, wput, and curl.
It could be a lot worse. Bash is pretty nice compared to MS DOS batch language.
Intellectual Property is a monopolistic, selfish, and defective concept. It is "tyranny over the mind of man"
There is zenity for GTK from shell.
There are other dialog tools out-there too.
It adds spice to a dish (program).
If you want different dishes, you are free to chose any toolkit (tcl/tk, C with GTK, ...).
If you look at Debian installer, it is based on many shell scripts. You can chose its frontend to be GUI. So limitation of shell script is only within people's mind. Just use right tools in smart way.
That's all.
Osamu
Once your mind setting is correct you will realize the beauty of the Bourne shell. Its main purpose really is to set up an environment to start another shell -a scripted or a compiled program. You can set up environment variables, start another shell, redirect output and evaluate the exit value. It's syntax is absolutely consistent and precise, the rule set to remember is relatively small and that to me is it's beauty. In fact, when starting a shell from another language like Java, I invariably log the environment and the execution in Bourne shell syntax.
I hadn't the slightest objection to his spending his time planning massacres for the bourgeoisie... (P.G. Wodehouse)
The only "spice" you'll get from me is a 'usage' summary if you pass args incorrectly.
If the script is interactive I will woo you with such things as vague prompts when input is required.
Walk without rhythm and you won't attract the worm.
if I was going to check for open ports wouldn't you just use nmap or netstat?
And if I was going to write a script that was going to act as an HTTP client why not just use curl or netcat?
A clever person solves a problem. A wise person avoids it. -- Einstein
No, seriously! Everything has a makefile at the top level. The default
target prints what it does, everything else is make this or make that.
Shell scripts go inside the makefile. Then in 18 months when you
come back to the project you know EXACTLY where to look to
figure out how to get things done.
I would just avoid shell scripting and use perl. If you really need to access a command just call it from with in the perl script.
As a developer, I've become a enthusiastic convert to bash scripting.
Like many, I originally found the syntax ugly and awkward. But with some encouragement and tips from people who know what they're doing I've come to appreciate its power. When I have to do some sort of mind-numbing manual server-related task, I script it. It forces me to think the through the steps, if only minimally, and it codifies knowledge that can be reused and shared. Over time, I have found it a great time-saver even for tasks that I run infrequently. The scripts are a permanent record, there for reference when I need them, and writing them helps me assimilate new information and develop new skills.
As an example of its simple power, consider the main routine of a deployment script I wrote not too long ago:
Expressive, straightforward, elegant, reliable. You can find the whole script here:
http://code.google.com/p/cakewell/source/browse/dev/bin/nfs_update.sh
P.S. should be a exit 0 at the end of that, but the lameness filter appears to have something against bash scripts itself.
Innovation makes enemies of all those who prospered under the old regime... -- Machiavelli
http://www.usenix.org/publications/library/proceedings/tcl96/full_papers/korn/
Anyone know what happened to it? I remember looking forward to it a long time ago but have since moved to python.
is about as cute as it gets.. ansi menus from bbs days would be sweet though
The first rule of shell scripts is "you don't write programs in shell." /bin/sh. Your script probably contains bashisms and you don't even know it. What's more, bash has some great features that are only available if you use it explicitly in your shebang.
The second rule of shell scripts is "you DON'T WRITE PROGRAMS IN SHELL." Seriously. You want perl or some other high level language.
The third rule is to start your script with "#!/bin/bash", not
The fourth rule would be to read the bash faq at http://mywiki.wooledge.org/BashFAQ as it contains many tips and tricks that won't be obvious just by reading the man pages.
I'm going to +1 this. I'm mad nooberish and the only "script" I've ever written automates a screen saver in place of an x-windows desktop...
That said, homeboy get's an ellipses fail on that post.
I would, don't get me wrong. Emacs is a lovely operating system. I just wish it had a decent text editor.
Bolting a GUI on top of something that generates UNIX commands is usually disappointing, for the usual reason - the output from UNIX commands is semi-random error messages.
The EeePC's Linux environment suffered badly from this. There's a pretty GUI for setting up networking, but if anything goes wrong, you have to look at an incredible mess of command line commands being driven by the GUI program. Many of the commands return errors in normal operation.
I still think that one of the basic design errors of UNIX was that programs return only an integer status code. You get to send command line parameters and environment variables in a reasonably structured way, but you get back just one integer. If commands had returned environment variables, and there had been reasonable standards for what came back from the earliest days, the whole concept of "scripting" might have worked quite differently. Programs would be viewed more like subroutines.
Bash scripting definitely has everything you'd need to write an "application" in, but many data constructs would be awkward to implement in Bash, so you'd use Python, Perl, or Ruby.
But what I can do in Bash I could also do in Ruby or Python very easily. What I could do in Ruby or Python would be very difficult to do in Bash.
Then you have Java and C++ which are clearly not scripting languages, but I could do everything there that I could do in Bash. On the other hand, most things interesting you can do with C++ would be next to impossible with Bash. You just can't beat the performance of compiled languages. On the other hand, development would be costlier and portability might be an issue.
Then again, you can tap a thumb tack in with a sledge hammer, but you run the risk of putting a nasty hole in your wall!
It all comes down to what is the right tool for the job. So many tools, though! So pick and choose wisely. But you knew that already.
Personally, I have given up on making a distinction between what is a "scripting language" and what's an application language. Javascript(!) is a "scripting language", but there is a high level of interactivity in the applications it's used for typically. Both in communicating with the server (AJAX) and with the user.
Many of these so-called "scripting languages" allows for object-oriented programming. Bash, of course, does not. But then it was never meant for that level of sophistication. But that's even more blur for you. But you knew that already too.
Ruby Neural Evolution of Augmenting Topologies
> What tools do you use that spice up your scripts on the Linux or Unix platforms?
My tiny template which does apply 99% of time.
#!/bin/sh
#
# What this script is about.
#
# Author Name
set -e # exit on errors
trap 'echo "Exit on error."' ERR
set -u # disallow usage of unset variables
set -C # disallow redirection to overwrite files
# insert rubbish here.
exit 0
# EOF
Shameless plug:
http://news.slashdot.org/article.pl?sid=09/02/11/1355243&from=rss
Honestly, a big part of what I rely on is just writing things that will stay functional even when I swap between pure BSD, OS X, and Linux. (And "Linux" these days means "Everything from RHEL4 to Ubuntu 9.10" at $dayjob).
There's a lot of fancy utilities I'd love to use if they were a little more portable. The one thing I'll say is: I wish everyone had implemented pax and installed it by default, it is a complete and sufficient replacement for tar and cpio.
My blog: http://www.seebs.net/log/ --- My iPhone/iPad app: http://www.seebs.net/seebsfrac/
Perl.
Knowing the difference between $* and $@ is an enormous help when you deal with multiple files with spaces. For example,
Here, "$*" expands to "my file one my file two", whereas "$@" gives the intended result of "my file one" "my file two". Nowadays I always use $@, even when there is zero risk of spaces in filenames, to keep things consistent for me.
Escher was the first MC and Giger invented the HR department.
Use Python (or Perl if you are so perverted, of Ruby, or C++, just don't use shell scripting) for anything nontrivial. Shell scripting and procmail should die. (Yes, I threw in procmail just to remind people of another old-time beast that is *not* better than present-day alternatives, no matter how cool or lame you think esr is). Learn your environment, test the options, choose the best one. There are many alternatives to quick-and-dirty programming off the CLI than bash/ksh/sh/zsh. Demistify the old, it's not always better.
I'm surprised nobody mentioned dmenu.
dmenu reads lines as items, lets you choose one of them (with tab completion, etc) and prints the choice you made.
Example: FILE=$(ls | dmenu)
dzen does the same thing and much more.
All these tools are very small and only depend on the Xlib. No gtk/qt crap.
Zenity http://en.wikipedia.org/wiki/Zenity
Smart::Comments (http://search.cpan.org/~chorny/Smart-Comments-1.0.4/lib/Smart/Comments.pm)
Not sure if this counts, but I use "Figlet" to spice up my shell scripts (and even some Perl scripts). It displays text output as ASCII art text... pretty fun stuff, has a bunch of different fonts, etc.
-= To crush your enemies, to see them driven before you and to hear the lamentations of their women =-
I've added some speech recognition (Computer! Terminal.)- crude, but mostly works. Also I find it extremely efficient for mass-image manipulation, combining my rendered frames to a video, backing-up, and (I love that one) grepping & awking. Plus, usage of sleep & beep (^G) are only limited by your imagination.
The three laws of thermodynamics:(1) You can't win. (2) You can't break even. (3) You can't even quit.
I went looking for the one for my "system", and I ended up throwing up in my mouth a little bit ... Euuuugh ... MS OCS ...
The source for SPICE was released in 1993. It's also packaged as part of gEDA. I'm not sure why you think that this would be useful to add to the shell, but to each their own ...
I'm so sick of hearing it!
I find that for many things the command line is a real pain to use. Things such as web browsing email, accounting and similar things.
Then again, I find that the gui sucks just as bad for doing bulk text searches, builds, finding files and acting on them in bulk.
Why can't folks figure out that a hammer is good for smacking things and a shovel is good for digging. Granted you can smak things with a shovel and you can dig with a shovel, but you will work much more effectively if you use the right tool for the job at hand!
-- Many men would appreciate a woman's mind more if they could fondle it
This is an honest question. I'm a non-professional programmer who has been doing OSS C coding for about twelve years, and gradually learned about a wide range of programming and Unix topics. A few months ago, I decided I didn't know enough about bash scripting and started working through various tutorials. Is knowing about such stuff really a waste of time? Or should I treat it as something I ought to know for historical and maintenance reasons only, while using (say) Python for any scripts I write myself? Or is "stop using the shell" a bit of hyperbole if taken completely literally, perhaps more like "bash scripting should only be used for simple things - use Python/Ruby for anything non-trivial".
Anyway, thanks for all your contributions over the years.
For anything complex I tend to use a real programming language, just to avoid the script problems with strange characters in file names.
Personally I favor avoid shell scripts unless I totally control the incoming (file) name arguments scripts see. Using scripts where users provide input has proven deadly too many times as spaces are just mild annoying problems compared when ', $, ", and other fun characters shells treat as metacharacters, even within double quotes, are in file names.
My rules for safe shell follow:
Corollary: understand the difference between "$*" and "$@".
Corollary: always write any real programs that accept or spit file names to support -0 options of some form.
A few straight jazz notes:
xyx*?) echo "You can actually use file name wild cards here";;
esac
(Not neat visual jazz, but anything that helps me find problems quicker is music to my ears, but don't use things like above "die" within trap).
Mac OS Classic had spaces in filenames long before MS-DOS and Windows, and DOS 3.3 on the Apple II had them even before that.
I use unix and windows but would no longer consider myself an expert in all the bits. Thats why a gui wins when I need to search through files in a file system ... but I digress.
The best advice I've read for scripts had something to do with ennabling a script to know if it was interactive or not and present and appropriate interface accordingly. Whether it was through a flag or checking for some other interactive vs. non type thingy (which I can't recall at the moment. :)
I agree that handling weird filenames can be tricky; see Fixing Unix/Linux/POSIX Filenames for more. The biggest problems aren't specific to shell, though, but are general complications that apply to all languages: Control Characters (such as Newline), Leading Dashes, and non-UTF-8 characters.
As far as using shell to handle filenames with spaces, double-quotes, and so on, the answer is pretty simple.
First, always begin shell scripts with: IFS=`printf '\n\t'` This means that the "space" character is no longer special, and this eliminates 99% of your problems.
Second, whenever you USE (instead of set) a variable, use "$variablename" instead of $variablename. If variablename can only contain alphanumeric characters, you don't need to do this (though it doesn't hurt).
Third, when you want a list of "filenames in this directory", use for x in ./*
instead of
for x in *
so that filenames beginning with "-" won't get you (this is a problem for all languages, not just shell).
Follow those rules, and the vast majority of "problems" go away. You can have filenames with double-quote characters, for example, as long as you reference them with "$variablename" instead of $variablename, it's not a problem (shell is smart enough to not interpret them twice).
Of course, if you want things even easier, support the idea of limiting filenames in Linux/Unix as I discuss in Fixing Unix/Linux/POSIX Filenames for more.
- David A. Wheeler (see my Secure Programming HOWTO)
Today, scripting in Ruby or Python yields scripts that can handle errors with advanced facilities like exceptions
Until your users run into distributions with a broken or missing Ruby or Python. For example, one major desktop operating system doesn't come with Python, and the standard binary package for Python on this system doesn't put Python on the PATH; instead, it associates *.py with Python in the graphical file manager.
1: tput, xset, setterm as in TFA. Can set colors, cursor position, etc. Nice for spinners, showing no characters on password input, etc. Take control of the terminal. Disable or alter that bell. Specify the number of columns, on and on..
2: dialog, as in TFA. Kind of kludgey, but very useful to present the "Top 10 features of this system." as a shell replacement. 'usermod -s /opt/corp/bin/fooscript.sh luser'
3: functions to provide all manner of jazzy enhancements. Write a spinner() function, a log() routine relevant to your org, a time-of-day dependent shell prompt, etc.
A word of advice on GUI notifications: Stick to your native toolkit. In my experience, they can be hit or miss. In the early days of Zenity, my test application worked as expected in my Gnome session, but when I turned it over for testing, the KDE krew's desktop session was notified, pop-up boxed, and task bar icon blinked into a swift CTRL-ALT-BKSP.
Also, test your foofery at the console as well as in xterm, gnome-terminal, konsole, etc.
If you think perl is going to let you off the hook syntactically, you've got another thing coming.
Spend a weekend afternoon reading the man page. Re-acquaint yourself with single vs. double quotes, curly braces, backslash delimiting, and I guarantee your life will change.
First, thank you for teaching me something new. I didn't know about $@ and that is a good one to know. (Is that POSIX standard, or a GNU extension?)
Second, wow, that is truly baroque. That is getting up into makefile levels of weird syntax. I still prefer the refreshingly clear syntax of Python for nontrivial scripting.
Third, when you want individual arguments, you still need to carefully quote them by hand. And this is a good example of the general rule, that you need to be very careful with quoting in shell scripts.
Fourth, yes, I agree with you: use the safer form all the time, even when you don't need to. It's like when I wrote my Python example of running an external program; I used subprocess.check_run() instead of os.system(). I like the convenience of os.system() but I generally want my scripts to stop on an error, and os.system() doesn't raise an exception on error.
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
http://trevelyn.com/?p=119 Thanks for the brain exercise :) scripts and GUI play well together fine, I've been doing it for years.
Thanks again :)
~douglas.
First, thank you for teaching me something new. I didn't know about $@ and that is a good one to know. (Is that POSIX standard, or a GNU extension?)
I only know it from Bash, so it may not be anything standard.
Fourth, yes, I agree with you: use the safer form all the time, even when you don't need to. It's like when I wrote my Python example of running an external program; I used subprocess.check_run() instead of os.system(). I like the convenience of os.system() but I generally want my scripts to stop on an error, and os.system() doesn't raise an exception on error.
I often use something like os.spawnvp for launching processes in Python. Of course, one reason is the process control issues you also mention. However, having to specify the arguments separately in a list, means that there is no shell in the middle. Whereas os.system launches a shell that parses the command line, and in turn runs the process. I've had one case where the latency was noticeable, and there is the potential memory penalty as well.
Escher was the first MC and Giger invented the HR department.
Many of these so-called "scripting languages" allows for object-oriented programming. Bash, of course, does not.
Shell scripts can be objects. See for example shar files and here documents for examples. Course you can write network servers in shell as well, doesn't make it a good idea.
Deleted
having to specify the arguments separately in a list, means that there is no shell in the middle. Whereas os.system launches a shell that parses the command line, and in turn runs the process. I've had one case where the latency was noticeable, and there is the potential memory penalty as well.
Well, it's easy enough to write a convenient wrapper around subprocess.check_run() like so:
import subprocess as sp
def run(s_cmd):
lst_cmd = s_cmd.split()
sp.check_call(lst_cmd)
Then you are back to having a problem if arguments have spaces, of course!
I agree with not worrying too much about the overhead of the shell. There was a time once when the time to start up a shell was a major hit; on modern computers, not so much. But still, it's always nice to be efficient if it's easy, and simply using subprocess.check_call() instead of os.system() is pretty easy.
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
All, but python and perl should be in there (OK, and ruby too...I suppose ;).
BUT... the big (modern) element you're missing here is: DBUS. I haven't actually scripted that, but it's based on DCOP, and scripting KDE with DCOP was great. Reminded me a lot of ARexx on the Amiga. In other words, it lets you script GUI apps too, bringing the power of Unix into the modern age. For the most part.
I think the whole idea of scripting needs to be overhauled in unix. It pains me to say it, but MS had the right idea with a standardised OO library framework that all languages can access, and a powerful OO shell language too. Somehow they still managed to make it ugly and unfun to use, but just about anyone else with the resources to make a framework like that with Unix principles (rather than corporate motives) at the core would do well.
Please show us all your most elegant solution, in shell script, to the problem of identifying whether a file has any extension from (".temp", ".tmp", ".junk).
ls filename* | egrep "\.temp$|\.tmp$|\.junk$" ... No?
And what's with all the regex stuff with find rather than just globbing?
find path -name "*.temp" -o -name "*.tmp" -o -name "*.junk"
Also, -exec with find kicks off a command for each hit. Pipe it to xargs, particularly with large numbers.
find path -name "*.temp" -o -name "*.tmp" -o -name "*.junk" --print0 | xargs -0 rm
Okay, I wrote lots of code. Your turn.
I wrote hardly any code.
Sorry. Maybe I'm just getting old. It's the fingers getting a bit stiff. I'm sure I could write lots and lots of code to grep for a filename but I'm just not up to it this evening.
Deleted
Very good question - I recently wrote a nice little shell script for adding a site to the squidGuard whitelist that filters my children's browsing. Then I wanted to make it easy for my wife to run. All I wanted was a little GUI window with a couple of controls to set the script's parameters, and an output box to capture the output of the script. I couldn't find a way to do it as easily as I wanted. In the end I went with a ruby script using a Glade ui definition. It works excellently, but it was more work than I really wanted to do. I'd love to hear of an easier way.
It is indeed part of POSIX; see XCU 2.5.2 for a complete list of such variables.