Fault Tolerant Shell

Paul Howe writes "Roaming around my school's computer science webserver I ran across what struck me as a great (and very prescient) idea: a fault tolerant scripting language. This makes a lot of sense when programming in environments that are almost fundamentally unstable i.e. distributed systems etc. I'm not sure how active this project is, but its clear that this is an idea whose time has come. Fault Tolerant Shell."

Comments

Python

[derphilipp]

Also, I would appreciate (not quite the same) a auto-completing python interpreter and editor (which can complete methods and objects from modules)... Such kind of stuff really increases productivity !

Re:Python

[Noryungi]

a auto-completing python interpreter and editor

Try the Wing IDE. It has most of the functions you wanted... But it's not free software.

Re:Python

[LarsWestergren]

You can download Python plugins to work together with the excellent Eclipse.

Re:Python

[fxj]

It already exists !
IPython

Re:Python

[pebs]

You can download Python plugins to work together with the excellent Eclipse.

Does this actually give you auto-completion? I heard that all it does is give you syntax highlighting.

Ok, just downloaded it for myself to check it out. It doesn't do any kind of auto-completion.

You're dealing with the problem too high up

[phaze3000]

IMO (as someone who works on clustered systems for a living) you're looking at this from the wrong point of view. A clustered shell is useful only if the system it is running on top of is inherently unstable.

The real benefit is in having a system which is sufficiently distributed that any program running on top of it can continue to do so despite any sort of underlying failure.

Re:You're dealing with the problem too high up

[unixbob]

Doesn't that depend on the definition of clustered though? Clustered systems can be things like beowulf clusters. But often a collection of standalone web servers behind a http load balancers is commonly referred to as a web cluster or array.

IMHO as someone who works in a complex web server / database server environment, there are many interdependancies brought by different software, different platforms and different applications. Whilst 100% uptime on all servers is a nice to have, it's a complex goal to achieve and requires not just expertise in the operating systems & web / database server software but an indepth understanding of the applications.

A system such as this fault tolerant shell is actually quite a neat idea. It allows for flexibility in system performance and availability, without requiring complex (and therefore possibly error prone or difficult to maintain) management jobs. An example would be server which replicates images using rsync. If one of the targets is busy serving web pages or running another application, ftsh would allow for that kind of unforeseen error to be catered for relatively easily.

Re:You're dealing with the problem too high up

[Moderation abuser]

"An example would be server which replicates images using rsync. If one of the targets is busy serving web pages or running another application, ftsh would allow for that kind of unforeseen error to be catered for relatively easily."

It depends how you organise your systems. If you push to them then yes you need something like ftsh. If you organise them so that they pull updates, pull scripts to execute and arrange those scripts so that they fail safe (as they all should anyway) then you'll have something which is a couple of orders of magnitude more reliable and scalable. It just needs a little more thought to begin with.

Re:You're dealing with the problem too high up

[unixbob]

Well that was just an example. If on the other hand the system the images were pulled from was very busy then the same is true. The problem is that you can't architect for a moving target and the flexbility that rapidly changing environments require is something which ftsh would be quite useful for.

Re:You're dealing with the problem too high up

[ChuyMatt]

Does admitting that many of the things he says are right scare you? Yes, if you watch the movie once, you realize that he is slanted in his view. NOT A BIT SURPRISING!

I was skeptical also, but he touched on almost every source that i found, just looking on the internet and not including the defamation sites. You, sir, seem like you are desperately looking for a reason not to believe ANYTHING that the movie says. Being someone who has been shot at and lost school friends to nutters with guns, i simply cannot reject the truths that are in that film.

Re:You're dealing with the problem too high up

[vidarh]

If you can set up a distributed system at a reasonable cost where any program can continue to run without carying about an underlying failure, you would be richer than Bill Gates.

Resources DO become unavailable in most systems. It simply doesn't pay to ensure everything is duplicated, and set up infrastructures that makes it transparent to the end user - there are almost always cheaper ways of meeting your business goals by looking at what level of fault tolerance you actually need.

For most people hours, sometimes even days, of outages can be tolerable for many of their systems, and minutes mostly not noticeable if the tools can handle it. The cost difference in providing a system where unavailabilities are treated as a normal, acceptable condition within some parameters, and one where failures are made transparent to the user can be astronomical.

To this date, I have NEVER seen a computer system that would come close to the transparency you are suggesting, simply beause for most "normal" uses it doesn't make economic sense.

Re:You're dealing with the problem too high up

[Anonymous Coward]

The real benefit is in having a system which is sufficiently distributed that any program running on top of it can continue to do so despite any sort of underlying failure.

Very good point. Look at internet protocol. The network can be pretty faulty at the hardware level, but once you get up to the session layer, all the holes are smoothed over. Applications don't have to worry about fragmenting packets, resending things that get lost, reordering things that are recieved out of order, it's all done at a lower level.

Re:You're dealing with the problem too high up

[Pieroxy]

If you can set up a blah blah blah, you would be richer than Bill Gates.

Will you ever learn? Technical superiority of a product has never, ever lead to a commercial success. It might help a bit, I concede, but it is never the main reason for a commercial success.

If someone gets the killer product you're talking about, here are the possible outcome:
1. He'll be ripped off by MS, IBM, etc...
2. He'll lock it and will not make a cent out of it because of bad management/marketing/sales or more generally, lack of a proper understanding of the business.
3. He'll build a company around it and will be rich as hell. Not likely to happen.

Bad Idea

[teklob]

It's a well meaning idea, but it would cause more problems than it would solve. It would just encourage sloppy code; people would rationalize "I don't need to fix errors because it doesn't matter", which is a very bad habit to get into when programming, ignoring errors, or even warnings

Re:Bad Idea

[Anonymous Coward]

Well, your idea is well-meaning, but it is a bad idea to cling too closely to the principle of fixing all bugs first. It is a fact of life that some bugs will remain however hard you try (even formal proofs won't notice specification errors), and in a critical production system you need to have some robustness against failures. It's no good if your system up and dies the moment it hits a real bug. IMHO a fault-tolerant shell will be a useful tool in some situations, even if some people end up misusing it.

Re:Bad Idea

[Tx]

I agree. Web browsers were designed to be fault tolerant, and just look at all the horrendously broken crap that passes for HTML out there. Dangerous stuff.

Re:Bad Idea

[Androclese]

Exactly, you nailed it on the head!

The only thing I can add at this point is an analogy:

Think of it along the lines of IE and HTML; if you don't want to close your tags, say your table td and tr tags, it's fine, the IE browser will do it for you.

Nevermind that it will break most any W3C compliant browser on the planet.

(insert deity here) help the person that gets used to this style of programming and then joins the real world.

Re:Bad Idea

[unixbob]

OTOH, if the tool makes scripts simpler then bugs are less likely to be introduced. Why do we all need to keep on reinventing our own libraries and methods for checking an NFS server hasn't timed out (to use the example on the ftsh site). why not use a tool that does that stuff for us so we can concentrate on writing more elegant and functional scripts that are easier to read and debug and quicker to write.

Re:Bad Idea

[Yakman]

hink of it along the lines of IE and HTML; if you don't want to close your tags, say your table td and tr tags, it's fine, the IE browser will do it for you.

According to the HTML 4.01 spec </td> and </tr> tags are optional. So you can code a standards compliant page without them, as long as you declare your doctype properly.

Re:Bad Idea

[cgenman]

It's a well meaning idea, but it would cause more problems than it would solve. It would just encourage sloppy code; people would rationalize "I don't need to fix errors because it doesn't matter", which is a very bad habit to get into when programming, ignoring errors, or even warnings

The same logic could be applied to any security system, from the automatic door lock on the front of your house to Airbags in your car. Spell checkers discourage people from learning to spell. Antibiotics prevent the growth of the immune system. Why have a lock on your trigger, if it will encourage you to leave it in a place where your kids can find it.

The fact of the matter is, if the code works, it's good code. This is a shell scripting language we're talking about here... Not exactly assembly. Programmers would be better off spending more time thinking about the higher structure of their applications and less time hunting down trivial mistakes.

Of course, I know that this isn't quite what the article is talking about, but it's the principle of the thing. Augmentation would be an improvement.

Re:Bad Idea

[Jerf]

Spell checkers discourage people from learning to spell.

Done correctly, spellcheckers can be the best spelling-learning tool there is.

"Correctly" here means the spell-checkers that give you red underlines when you've finished typing the word and it's wrong. Right-clicking lets you see suggestions, add it to your personal dict, etc.

"Incorrectly" is when you have to run the spell-checker manually at the "end" of typing. That's when people lean on it.

The reason, of course, is feedback; feedback is absolutely vital to learning and spell-checkers that highlight are the only thing I know of that cuts the feedback loop down to zero seconds. Compared to this, spelling tests in school where the teacher hands back the test three days from now are a complete waste of time. (This is one of many places where out of the box thinking with computers would greatly improve the education process but nobody has the guts to say, "We need to stop 'testing' spelling and start using proper spell-checkers, and come up with some way to encourage kids to use words they don't necessarily know how to spell instead of punishing them." The primary use of computers in education is to cut the feedback loop down to no time at all. But I digress...)

'gaim' is pretty close but it really ticks me off how it always spellchecks a word immediately, so if you're typing along and you're going to send the word "unfortunately", but you've only typed as far as "unfortun", it highlights it as a misspelled word. Bad program! Wait until I've left the word!

Re:Bad Idea

[Phurd Phlegm]

It's a well meaning idea, but it would cause more problems than it would solve. It would just encourage sloppy code; people would rationalize "I don't need to fix errors because it doesn't matter", which is a very bad habit to get into when programming, ignoring errors, or even warnings

Initially I thought that you had a valid point, but then I reflected what happens if we apply that reasoning to everyday life. We get cars where you ride on the front bumper with no seat belt. Anything else would encourage careless driving....

Re:Bad Idea

[JohnFluxx]

"Nevermind that it will break most any W3C compliant browser on the planet."

Same problem with english. Most people have a high degree of fault tolerance when parsing natural language, which means any old crap will still just about be readable.

(sorry - I had to after reading that statement heh)

Re:Bad Idea

[whiteranger99x]

You couldn't have said it in better words. I mean, people using IE are used to each and every one of it's quirks, even if it happens to be "fault tolerant" by allowing badly formed HTML/JS/CSS/XML/etc to render the pages. Consequently they may think that Mozilla/Opera/Firefox/whatever aren't as good because they may not tolerate badly designed HTML/JS/CSS/etc as quietly.

Re:Bad Idea

[ChaosDiscord]

It's a well meaning idea, but it would cause more problems than it would solve. It would just encourage sloppy code; people would rationalize "I don't need to fix errors because it doesn't matter", which is a very bad habit to get into when programming, ignoring errors, or even warnings

You've got it backwards.

Most shell scripting is quick and dirty; no one checks error codes (mostly because it's a nuisance). FTSH makes it easier to check error codes, in part because the default behavior is the bail on any error. In essence FTSH adds exception-like behavior to shell programming. FTSH makes it easier to write shell programs that fail gracefully. How can that possibly be a bad thing?

Re:Bad Idea

[AnonymousKev]

Amen brother! What you're saying is that "fault tolerant" is the same as "fault encouraging".

Re:Bad Idea

[khallow]

The same logic could be applied to any security system, from the automatic door lock on the front of your house to Airbags in your car. Spell checkers discourage people from learning to spell. Antibiotics prevent the growth of the immune system. Why have a lock on your trigger, if it will encourage you to leave it in a place where your kids can find it.

The appropriate concept from economics is "moral hazard". If you provide a service that helps reduce the occurance a particular problem, then people will alter their behavior in a way that increases the chance of that problem occuring. So it is conceivable that the problem can occur more often than it originally did, due to the change in user behavior. Having said that, I agree with the above poster. There's a lot more going on than just the adverse change in programmer behavior and to focus on that alone is a mistake.

Re:Bad Idea

[bangular]

hmmm... that's odd because I've come to know Mozilla as quite fault tolerant. Even supporting IE's non-standard tags...

Re:Bad Idea

[tyrione]

Thankfully 24 December 1999 is long since dead, now get off your butt and write in XHTML.

Then try and validate now.

Luckily XML demands two options of well-formedness and validation. Neither of which allows sloppy coding. It sure makes the client parsing the files run more smoothly knowing it doesn't have to account for such sloppy practices and waste cycles in the process.

Re:Bad Idea

[Anonymous Coward]

I think there should be a special "learning edition" of the spell checker that instead of underlining the error - it sends you a nice little shock through the keyboard. It also keeps track of the number of times you misspell a word so that it can administer higher and higher levels of electricity the more often you misspell the same word.

Oooh! An idea whose time has come!

[linuxbaby]

More ideas whose time has come, including:

• DRM Helmets
• Jack Kemp
• Yankee Go Home
• Collaborative Dispute Resolution
• Microchips for Your Pet Parrot! (see page 2 of Google results)

Wouldn't be much work in Tcl

[DavidNWelton]

... or probably Perl or Python, either.

It doesn't actually seem to grok the commands that are being run, so something like

proc try {times script} {
if { [catch [uplevel $script] err] } { cleanup ; retry }
}

is all that's needed (of course to do it right you'd need a bit more, but still...).

try {5 times} {
commands...
}

Although Tcl is a bit lower level, and would require you to do exec ls, you could of course wrap that too so that all commands in the $script block would just be 'exec'ed by default.

In any case, better to use a flexible tool that can be tweaked to do what you need then write highly specialized tools.

Re:Wouldn't be much work in Tcl

[patSPLAT]

Here's a Ruby one:

def college_try (limit, seq =0)
begin
yield
catch e
# forgot the syntax for getting the block
college_try( limit, seq + 1, block ) if (seq < limit)
end
end

college_try( 50 ) {
begin
do some work
catch e
do error clean up here
raise e
ensure
do cleanup that should always run here
end
}

Anyways, I agree with the notion that most popular scripting languages have advanced error handling that is up to the task.

Re:Wouldn't be much work in Tcl

[vidarh]

Bzzt. Try again. Where is the exponential backoff? Where is the ability to restrict each contained statement for a specific amount of time? Where is the ability to execute each command at specific intervals?

Your example only does a fraction of what ftsh does.

Re:Wouldn't be much work in Tcl

[patSPLAT]

> Your example only does a fraction
> of what ftsh does.

yawn, so we didn't post a 100-500 line library in our slashdot comment.

the point is, this stuff would be trivial to implement in language like ruby. plus, using a full scripting language you get lots of other useful features like regular expressions, classes, etc, etc

It's a good idea, but it's a library implemented as a language.

Re:Wouldn't be much work in Tcl

[DavidNWelton]

No kidding... that's why it's an example and not a full implementation. You can do all those things in Tcl (or Ruby, Perl, etc...). The idea is that instead of creating some one-off shell, you add a neat feature as an extension to an existing tool.

Re:Wouldn't be much work in Tcl

[Anonymous Coward]

perl module

Worst idea since spell checkers

[91degrees]

This will not improve people's skills. In fact, it willl make them more prone to mistakes, and more likely to get the result that they didn't expect. It's similat to computer spell checkers. Ever since people started relying on these, their spelling has gone way downhill simly because they don't bother thinking. Computer do all the spelling for them. They don;t need a spell checker. They need spelling lessons.

This si even worse. Computers will try to second guess what the user means, get get it wrong half tyhe time.

A qualified shell scripter will be not make these mistakes in the first place. Anyone who thinks they need this shell actually just need to learn to spell and to ytype accuratly.

Re:Worst idea since spell checkers

[Anonymous Coward]

Was that a joke post?

Re:Worst idea since spell checkers

[WeeBull]

Dude - you could have spell-checked your post!

Re:Worst idea since spell checkers

[FrostedWheat]

just need to learn to spell and to ytype accuratly. -- QED - Quite Easily Done

<Teal'c> Indeed </Teal'c>

Re:Worst idea since spell checkers

[bkhl]

Ever since people started relying on these, their spelling has gone way downhill

I find that hard to believe. Rather, I'd say my impression is that spelling has never been as standardised as it is today. What did you get that from?

Re:Worst idea since spell checkers

[Anonymous Coward]

MS Word spellcheck for your first paragraph:

This will not improve people's skills. In fact, it willl make them more prone to mistakes, and more likely to get the result that they didn't expect. It's similat to computer spell checkers. Ever since people started relying on these, their spelling has gone way downhill simly because they don't bother thinking. Computer do all the spelling for them. They don;t need a spell checker. They need spelling lessons.

Re:Worst idea since spell checkers

[the_duke_of_hazzard]

I think you mean our spelling's gone way downhill, and we need spelling lessons.

Re:Worst idea since spell checkers

[Air-conditioned cowh]

Personlly, I think my spelling has improved due to spell checkers. I allways try to learn from whatever corrections is makes. Maybe other folks do too.

Also, this isn't about covering up mistakes. I am sure good script programmer will _allways_ assume a command can fail. Using the example of the "cd" command in the article, should I really just assume it worked before removing files? Of course not. How ftsh helps is that the necessary error checking code is made more readable and brief. I still have to trap errors whether I use ftsh or bash, the difference is ftsh is easier to understand.

Simply making code less convoluted and more readable is not the same as sloppy programming.

Re:Worst idea since spell checkers

[ArseneLupin]

This si even worse.

Ha!

Anyone who thinks they need this shell actually just need to learn to spell and to ytype accuratly.

Somehow I get impression that these inaccuratl msiytypings are intentional. My bad.

Re:Worst idea since spell checkers

[SpaghettiPattern]

Ever since people started relying on these, their spelling has gone way downhill simly because they don't bother thinking.
Disagreed. Spell checkers show me the mistakes/typos I make and allow me to reflect on them. Since decent, general purpose spell checkers are around my spelling has been improving. Especially English which isn't my native language.

FYI: I made just one small mistake in this posting.

Re:Worst idea since spell checkers

[smallpaul]

Argh. You are the second or third to bring up this silly spell checker analogy.

The software is not called the "mistake-tolerant" shell. It is the fault tolerant shell. It handles faults like hard drive crashes, network outages, cosmic rays, and yes, probably software bugs as a side effect. Look at the feature set: they are much more geared towards hardware failure than software failure. How does retry or exponential backoff help if a software bug prevents a computation from correctly completing? Actually, those things really help to keep the system running while somebody switches from the database server with the system crash to the backup machine. These sorts of retry algorithms are built into protocols like SMTP and TCP for the same reasons.

Re:Worst idea since spell checkers

[91degrees]

MS clearly has a different way of spelling the words "computer" and "do".

Re:Worst idea since spell checkers

[bar-agent]

Hm. When did Teal'c replace Spock?

Re:Worst idea since spell checkers

[FrostedWheat]

Hm. When did Teal'c replace Spock?

You know, I was just thinking the same thing. Teal'c would sometimes even raise his eyebrow the same way as Spock.

Spooky!

(offtopic, but..) Python

[Anonymous Coward]

auto-completing python interpreter and editor

An auto completing Python interpreter and editor:

Pythonwin (Windows only).

When it expands a class or module, select the one you want with the up and down arrows (or just keep typing to narrow the selection down), then press tab to select it.

It's got the concept backwards

[Moderation abuser]

While, yes, you manage distributed systems from the center, you don't *push* updates, changes, modifications because, it doesn't scale. You end up having to write stuff like this fault tolerant shell which is frankly backwards thinking.

Instead, you automate everything and *pull* updates, changes, scripts etc. That way if a system is up, it just works, if it's down, it'll get updated next time it's up.

I won't go into details but I'll point you at http://infrastructures.org/

Re:It's got the concept backwards

[vidarh]

I guess you didn't READ the article, considering that the example given on the page was specifically an example of pulling a data file, trying multiple hosts in turn.

The thing is, if you run a distributed system with a thousand servers, and your patch distribution points drop of the network, you don't suddenly want a thousand servers hammering the networks endlessly. You want things like exponential backoff, timeouts after which the system will change behaviour (stop requesting updates until explicitly requested to start again, start triggering alarms etc.), and that is exactly the kind of scenario ftsh makes easy to do in scripts without having to write all the logic yourself for every bloody script.

Re:It's got the concept backwards

[Moderation abuser]

I did. Endlessly is good. The network overhead is negligible.

Check once every 1,2,4,8,16,32,64,whatever,mins *all the time anyway* whether it fails or succeeds and you *absolutely don't* want to have to explicitly tell 1000 machines to start again.

You simply generalise the update process, get rid of the special cases. In the case of patches, you know you're going to have to distribute them out to clients at some point anyway so have all the clients check once a day, every day. If the distribution server is down for a couple of days it's pretty much irrelevant.

My error detection code is trivial the network traffic is negligible unless the job's actually being done and I still haven't been given a good case for ftsh. I have a good case for a better randomising algorithm within a shell and a decent distributed cron (which is simple BTW), but not for a specifically fault tolerant shell.

You've got to stop thinking of these things as individual systems. The network is the machine.

Sounds like good way to do some serious damage

[heldlikesound]

on a loosely configured network, not saying this tool doesn't seem interesting, but it seems prone for use in DOS attacks...

Re:Sounds like good way to do some serious damage

[vidarh]

I could more damage with "wile true; do ... " in bash. It's not like an infinite loop is very hard. ftsh on the other hand defaults to exponential backoff, which isn't exactly what you want if you want to DOS someone.

This would be nice...

[Ritontor]

how many times have you hacked something together in perl that ended up being relied on for some pretty important stuff, only to find 6 months down the track that there's some condition (db connects fine, but fails halfway through script execution as an example) you didn't consider and the whole thing just collapses in a heap - a nasty to recover heap cause you didn't write much logging code either.

This would REALLY be useful when you're connecting to services external to yourself - network glitches cause more problems with my code than ANYTHING else, and it's a pain in the arse to write code to deal with it gracefully. i'd really really like to see a universal "try this for 5 minutes" wrapper, which, if it still failed, you'd only have one exit condition to worry about. hey, what the hell, maybe i'll spend a few days and write one myself.

Let's draw a line in the sand...

[humankind]

All the programmers who need the environment to compensate for their inadequacies, step on one side. All the programmers who want to learn from their mistakes and become better at their craft, get on the other side.

Most of us know where this line is located.

Re:Let's draw a line in the sand...

[Anonymous Coward]

java, c++, c#

-------------

asm, c ?

Re:Let's draw a line in the sand...

[LarsWestergren]

All the programmers who need the environment to compensate for their inadequacies, step on one side. All the programmers who want to learn from their mistakes and become better at their craft, get on the other side.

Most of us know where this line is located.

"In other news, at the local beach today a vicious fight broke out between geeks about where to draw a line. Sand was kicked, noses have been blooded, we have some unconfirmed reports of a wedgie. We will have more on this breaking news as it comes in."

Re:Let's draw a line in the sand...

[ebuck]

Too good to pass up...

Redmond city limits?

Re:Let's draw a line in the sand...

[Avihson]

Isn't that line located outside of the unemployment office?

Re:Let's draw a line in the sand...

[Androclese]

Most of us know where this line is located.

yup...

#!/usr/bin/perl -wT
use strict;

Re:Let's draw a line in the sand...

[Anonymous Coward]

I'm with you. Brothers, cast down thy debuggers! Join us in the use of printf()!

Re:Let's draw a line in the sand...

[Rakshasa Taisab]

If you think C++ belongs on that side of the line, then you've either never programmed in C++, or you've written some pretty buggy programs (and are ignorant of it). C++ is kinda like a powered chainsaw, effective and powerfull but if you don't know how to use it you'll end up losing a leg or two.

Re:Let's draw a line in the sand...

[yess]

Ok. Wait! One second... Which side is first, which one the second?

Re:Let's draw a line in the sand...

[Anonymous Coward]

All the programmers who use libraries, programming languages or ready-made chips and circuits, get the hell out of the learning-from-mistakes & craft-improving side.

Re:Let's draw a line in the sand...

[AbbyNormal]

Better Line:

?
-----------
India

(Stupid Slashdot filter rules)

Re:Let's draw a line in the sand...

[vidarh]

I assume you do all your calculations in your head without a pen, pencil, paper, abacus, computer. And if you DO use computers to compensate for your inadequacies I do assume you stick to command line interfaces and program by doing "cat >executable" and typing in the executable directly without resorting to such cheating as assemblers or *shudder* compilers...

Or perhaps you didn't RTFA.

Re:Let's draw a line in the sand...

[gearry]

Come on, geeks at the beach? Wouldn't the sun hurt our pale complexions?

Everybody knows all the real geeks are still in the desert trying to get their robots to go a few more miles.

Re:Let's draw a line in the sand...

[Chris_Jefferson]

There was a time not long ago when people talked about anyone who didn't use assembler as a joke programmer, because they needed someone else to set up their variables, look after the stack, etc. and were willing to take the speed hit because of it.

I enjoy programming and I do it often. However in business the purpose of a programming language should be to allow you to as quickly and easily write the program to do what you want to do and (almost more importantly) accept that you WILL have bugs in that program, but try to make it as difficult as possible to put bugs in, and as easy as possible to find and remove the ones you did put in.

Anyone who talks about writing "bug-free code" has never spent any real amount of time writing code professionally.

Re:Let's draw a line in the sand...

[johnnyb]

C++ is only usable when used in conjunction with a garbage collector.

Re:Let's draw a line in the sand...

[Phil Wilkins]

Most of us know where this line is located.

Down the middle of the playground?

Re:Let's draw a line in the sand...

[LarsWestergren]

Down the middle of the playground?

Bingo...

Re:Let's draw a line in the sand...

[Anonymous Coward]

At which point it's then called Java.

Re:Let's draw a line in the sand...

[johnnyb]

Actually there's a lot of things you can do with C++ that just aren't possible with Java. They are ugly with C++, and perhaps someday someone will come up with ideas for LISP macros to make them cleaner.

Read "Modern C++ Design" by Alexandrescu. It's great, and will open your eyes to several new possibilities.

Nice, which brings me too....

[MrIrwin]

The idea of being to timeout and exception handle in scripts is a great idea......assuming you want to use scripts. I think most people end up resorting to Perl, Python or whatever for anything more complex. But perhaps with this facility Scripts would be more useful? But...now I come to a related topic. I build factory wide systems, systems which have eg. Automatic warehouses and whatever in the middle. I do a lot of stuff with VB6 not because it is fault tolerant but because it is 'fix tolerant'. During the comminssioning phases I can leave a program running in the debugger and, if it freaks out, I can debug, fix, test by iterating forwards and **backwards** in the the function that caused the hitch, and then continue to run were I left off. Many minor problems get fixed on the fly without users even realizing anything was amiss. In every other respect (syntax, structure, error trapping etc) VB6 is a disaster and not really suited at all to these types of progects, so the fact that I use it is a measure of how important this feature is. Like the fault tolerant shell, it is a 'non-pure' extension insofar as purists say it should not be neccessary, but in pratice it is a godsend. Anybody know an alternative for VB6 in this respect?

Re:Nice, which brings me too....

[aurelien]

Any good Common Lisp implementation ?

Re:Nice, which brings me too....

[Anonymous Coward]

What the computer world needs is a COmmon Business Orientated Language or similar, something that everyone can use and no-one can understand

Re:Nice, which brings me too....

[Anonymous Coward]

Ever considered posting this as an AskSlashdot, Irv?

Building on their first example

[gazbo]

They are deleting a number of files on a number of different machines, then downloading an updated version. The implication is that the fault tolerance means a failure is not fatal.

So what happens if the files are crucial (let's use the toy example of kernel modules being updated): The modules get deleted, then the update fails because the remote host is down. Presumably the shell can't rollback the changes a la DBMS, as that would involve either hooks into the FS or every file util ever written.

Now I think it's a nice idea, but it could easily lead to such sloppy coding; if your shell automatically tries, backs off and cleans up, why would people bother doing it the 'correct' way and downloading the new files before removing the old ones?

Re:Building on their first example

[Anonymous Coward]

If you look a bit closer you'll notice that you can add a section of code that runs upon failure, which you could use to clean up.

Re:Building on their first example

[gazbo]

No, you miss my point; I accept that the system has uses, and allows for cleaning up. But look at the code sample on their front page: how exactly do you "clean up" from a `rm -f data`?

The answer is that you don't - you clean up from a `mv data data.tmp`. But with a system that purports to have transaction-like facilities people are going to just assume that the operation is, well, fault tolerant - as evidenced by their own front page code sample!

Re:Building on their first example

[vidarh]

If people make assumptions that are completely unjustified based on everything the manual says, then they get what they deserve. The purpose and semantics are quite clearly documented. If you base your usage of a software package based on a few lines long example with no clear purpose outside of being an example, then that is the problem, not the software package.

It's not about cleaning up. It's about fault tolerance. Deleting the data file in the case of the example is NOT a failure. Presumably the file is no longer useful (if not the script is buggy). Deleting the wrong data would be a failure. Not downloading the new data at some point in time would be a failure. THAT is what the script as written protects against.

The assumption that it's not ok to delete a data file before the new file is in place assumes that you know something about the uses of the data that the rest of us (presumably including the website author) does not. The deletion could very well be placed first in the script on purpose, to ensure that the data is deleted at the earliest possible time, regardless of the time it would take to download a new data set - possibly because the old data set is obsolete and clients should rather wait for a new dataset than rely on the old.

How long until...

[simon_clarkstone]

...people start pronouncing "ftsh" as "fetish". Actually, I've started already, just ask the girl sitting next to me. ;-)

Re:How long until...

[simpleguy]

Yes I confirm.

Signed the girl.

Python already has that

[xlurker]

here you go

• import rlcompleter, readline and
• here

autocompletion in the editor is availible in vim here

Re:Python already has that

[derphilipp]

I got 2 words for you: Thank you !

RTFA

[Anonymous Coward]

You obviously didn't read the article.

"It [ftsh] is especially useful in building distributed systems, where failures are common, making timeouts, retry, and alternation necessary techniques."

It doesn't protect you from typos in the script, it handles failures in the commands that are executed.

Re:RTFA

[91degrees]

Of course not. I'm trolling. Details like facts get in the way.

Re:RTFA

[ichimunki]

Best part is: you've been modded up to +4, Insightful. Makes a person wonder if Slashdot isn't actually a Diebold beta of some sort.

Re:RTFA

[Anonymous Coward]

Oh Itchy Monkey! Scratch'em den!

login

[Rutje]

"Password fairly correct. Root login granted."

Re:login

[Linker3000]

Yup, imagine 'ole clippy:

Seems you're trying to guess the administrator password. Do you want me to:

* Let you in to save time
* Give you a couple of letters as a clue
* Not let you in until you get it right

Re:login

[Anonymous Coward]

In Windows 95/98 (not SE) all you had to do to get to shared folders (which were available to anyone if you were online) was guess the first letter of the password. (So 26 "fairly close" attempts and you were guaranteed in)

Using wget as an example

[kasperd]

It seems like a bad example to me since wget already has a lot of retrying build in.

Re:Using wget as an example

[MrIrwin]

Which brings a 1M$ question.

Should a script always use apps which have built in net problems detection (and return with error), or should we be able to use generic commands which ignore eventual problems and may be potentially hang indefinetly?

IMHO, wget is a good solution if appropriate. But suppose I wanted to e.g. check a file for content based on a remote dictionary, and failing that a local one. This shell could automatically revert to the local copy if the remote (updated) dictionary did not respond in time.

Re:Using wget as an example

[kasperd]

wget is a good solution if appropriate.

With the right options wget is appropriate for almost anything. You'd be surprised to see the list of possible options you can specify for wget. You can specify timeout as well as number of retries, which means it would also be usable for the scenario you describe. Though it seems like wget doesn't have exponential backoff, only linear backoff, and there seems to be no options to change the backoff strategy. Which one of exponential and linear is best of course depends on your point of view.

DOS is pretty fault tolerant...

[toesate]

since it cannot really do a lot (of damage) in the first place!

Anyway, a shell is just a shell is just a shell...

Re:DOS is pretty fault tolerant...

[Anonymous Coward]

for %a in (c: d: e: f: g: h:) do format %a /autotest /u

Sure, DOS can't do any damage at all...

this can essentially already be done in /bin/bash

[xlurker]

(the concept of fault-tolerant coding encourages sloppy coding. and it makes it harder to see what's actually happening in the script. but that's not what they actually mean.)

what they seem to essentially want is

• a try statement and error catching and
• a fortran like syntax for testing and arithmetic

I think the authors were a bit misguided. Instead of creating a whole new shell how about just extending a good existing shell with a new try statement a described.

it can even be done without extending the shell:

( cd /tmp/blabla
&&
rm -rf tmpdir
&&
wget http://some.thing/wome/where
) || echo something went wrong

as for the new syntax of .eq. .ne. .lt. .gt. .to.
certainly looks like fortran-hugging to me , yuck

as for integer arithmetic, that can be done with by either using backticks or the $[ ] expansion

% echo $[ 12 * 12 + 10 ]
% 154

Once again, Babbage was thinking ahead...

[andersen]

"On two occasions I have been asked [by members of Parliament!], 'Pray, Mr.
Babbage, if you put into the machine wrong figures, will the right answers
come out?' I am not able rightly to apprehend the kind of confusion of ideas
that could provoke such a question."
-- Charles Babbage

Re:Once again, Babbage was thinking ahead...

[McDutchie]

If anything Babbage's reaction to the question illustrates that nerds have always been prone to elitism and contempt for the non-nerd mind. The MEP in question asked a very good question (especially considering the century it was asked in), and the fact that he did ask it shows that far from being "confused", he was on the right track to understaning a fundamental computing concept.

Re:Once again, Babbage was thinking ahead...

[Anonymous Coward]

He did NOT ask a very good question. In what conceivable way could putting in the wrong information result in a correct answer? If that were true, we would only need to enter the number 0 once! Or perhaps no numbers at all. You hardly need to be a computer nerd to understand that.

Re:Once again, Babbage was thinking ahead...

[Anonymous Coward]

The question asked to babbage is pretty dumb.

He just got done demonstrating a counting machine. It ACCURATELY added two numbers. Then someone basically insults him by asking if they input any two numbers will it output what they want?

He wanted the damn thing to be intelligent and a mind reader! That tells me that person lived in a very different world then i do now.

Re:Once again, Babbage was thinking ahead...

[geminidomino]

the fact that he did ask it shows that far from being "confused", he was on the right track to understaning a fundamental computing concept.

Which concept is that? GiGo? IMNSHO, anyone who even has to CONSIDER that question (thus assuming computers are somehow psychic) fully deserves all the nerdly contempt you can pile on them.

Re:Once again, Babbage was thinking ahead...

[McDutchie]

Which concept is that? GiGo? IMNSHO, anyone who even has to CONSIDER that question (thus assuming computers are somehow psychic) fully deserves all the nerdly contempt you can pile on them.

Well, thank you for proving my point. :)

Re:Once again, Babbage was thinking ahead...

[geminidomino]

Which point was that? That people are too tolerant of others who don't even bother with the effort of TRYING to understand something? I guess you've not worked tech support. These are the people who don't perform the simplest feats of logic (i.e. checking a power cord when something electrical won't power on), then proceed to explain "I don't know anything about all that [geek subject] stuff" as if it was an excuse.

Why is there a Windows compatible port?

[OC_Wanderer]

I'm sorry, but I can't understand why a Windows port (even if not native) is even attempted. Seems kind of useless in a totally GUI environment. Of course, maybe it's just me?

Re:Why is there a Windows compatible port?

[Anonymous Coward]

Seems kind of useless in a totally GUI environment. Of course, maybe it's just me?

Uh.. it's just you. You should, y'know, maybe try using Windows 2000 or XP sometime... Windows has a perfectly good command line. Point at the "Start" menu, click "Run" -> type "cmd", and away you go.

You can turn on command line completion (search for "TweakUI" or "Windows Powertoys", I can't be bothered to link to them). And even pipes work just fine (as they have since the DOS days). For example:

dir *.txt /s /b > textfiles.txt
type textfiles.txt | more

There's a whole build target type that specifies Win32 executables as command line programs - such programs play nicely with command shells and piped input & output. For example, all the Microsoft compiler tools are command line programs that are wrapped by the GUI. You can pull code from a Visual SourceSafe database and rebuild a project all from the command line if you want - such build mechanisms would be a prime target for a fault tolerant shell (although I think Scons has well solved the rebuild problem itself).

Other tools, such as Python and Perl all play nicely with the command line and could also be used with this shell on Windows.

Re:Why is there a Windows compatible port?

[OC_Wanderer]

I use XP Pro. I run Apache, PHP, and MySQL on it. Still I see no need to this kind of shell.

Re:Why is there a Windows compatible port?

[Tei]

Mostly because Windows lack of good command line admin tools historically. Actually has a few, but cmd is not bash, so you have to suplement these..

some people, (I myself too) use bash as a daily basic for Windows, this new stuff can be interesting and maybe usefull for the unsafe windows enviroment.

Re:Why is there a Windows compatible port?

[vidarh]

Perhaps because if you have access to run software remotely on a few hundred desktop computers that "has to" run Windows you would like to have the same script environment to use on Windows as on the other platforms you use? Perhaps because automated scripts aren't particularly good at using GUI's?

Re:Why is there a Windows compatible port?

[Junks Jerzey]

You can turn on command line completion (search for "TweakUI" or "Windows Powertoys", I can't be bothered to link to them).

It's on by default in Windows XP Service Pack 1.

I don't see why eveyone is complaining...

[quakeslut]

What do you lose by using something like this?

Well.. besides pipes of course ;)

Variable redirection looks just like file redirection, except a dash is put in front of the redirector. For example, For example, suppose that we want to capture the output of grep and then run it through sort:

grep needle /tmp/haystack -> needles
sort -< needles

This sort of operation takes the place of a pipeline, which ftsh does not have (yet).

In Monopolistic America

[fruity1983]

In monopolistic America, you tolerate faulty shell.

Missing the point

[SmallFurryCreature]

This is not about catching scripting errors. It does not fix your code. It is about catching errors in the enviroment that scripts are running in.

Shell scripts should be short and easy to write. I have seen plenty of them fail due to some resource or another being temporarily down. At first people are neat and then send an email to notify the admin. When this then results in a ton of emails everytime some dodo knocks out the DNS they turn it off and forget about it.

Every scripting language has their own special little niche. BASH for simple things, perl for heavy text manipulation, PHP for creating HTML output. This scripting language is pretty much like BASH but takes failure as given. The example shows clearly how it works. Instead of ending up with PERL like scripts to catch all the possible errors you add two lines and you got a wonderfull small script, wich is what shell scripts should be, that is none the less capable of recovering from an error. This script will simply retry when someone knocks out the DNS again.

This new language will not catch your errors. It will catch other peoples errors. Sure a really good programmer can do this himself. A really good programmer can also create his own libraries. Most find of us in admin jobs find it easier to use somebody elses code rather then constantly reinvent the wheel.

Re:Missing the point

[f0rt0r]

I think it still will promote bad programming/scripting practices. Many people ( including myself ) started with scripting before moving on to full-fledged programming. What they learned in scripting they carry forward with them into programming, and trust me, I learned to be very meticulous when it comes to interacting with things outside of my scripts control ( such as files ). Every I/O operation should be tested for success. Trying to open a file? Did it work? Ok, try writing to the file...did it work? Open a database connection...did it work? Let the user enter a number...did they enter a valid number? Error handling and input validation is something you just have to learn, like it or not. Something that holds your hand and lets you code while remaining oblivious to the realities of the scripting/programming environment is a bad thing IMHO.

On a side note for Perl, one thing I always hated were the examples that had something like "open( FH, "file/path" ) || die "Could not open file!" . $!; I mean, come one, you don't want your script to just quit if it encounters an error...how about putting in an example of error handling other than the script throwing up its hands and quitting! LOL.

Please excuse any grammatical/other typos above, I was on 4 hrs sleep when I wrote this. Thank You.

Re:Missing the point

[Moderation abuser]

This simply isn't required in a properly organised distributed architecture.

Write your scripts to fail safe, then don't perform ad-hoc updates, schedule them regularly.

Re:Missing the point

[Anonymous Coward]

this isn't about people being good programmers. this about getting work done whether you are a programmer or not. less lines of code is a good thing.

Re:Missing the point

[vidarh]

So what you are saying is that programming should be hard, and people should be expected to do it right, or it promotes bad practices.

Yet we are expected to excuse your grammatical and typos. Doesn't that just promote bad practices? Shouldn't we whack you over the head with a baseball bat just to make sure you won't post when you're not prepared to write flawless posts?

The more work you have to do to check errors, the more likely it is that however vigilant you might be, errors slip past. If you have to check the return values of a 100 commands, that is a 100 chances for forgetting to do the check or for doing the check the wrong way, or for handling the error incorrectly.

In this case, the shell offers a function that provides a more sensible default handling of errors: If you don't handle them, the shell won't continue executing by "accident" because you didn't catch an error, but will terminate. It also provides an optional feature that let you easily retry commands that are likely to fail sometimes and where the likely error handling would be to stop processing and retry without having to write the logic yourself.

Each time you have to write logic to handle exponential backoff and to retry according to specific patterns is one more chance of introducing errors.

No offense, but I would rather trust a SINGLE implementation that I can hammer the hell out of until I trust it and reuse again and again than trust you (or anyone else) to check the return code of every command and every function they call.

This shell does not remove the responsibility to for handling errors. It a) chooses a default behaviour that reduces the chance of catastrophic errors when an unhandled error occurs, and b) provides a mechanism for automatic recovery from a class of errors that occur frequently in a particular type of systems (distributed systems where network problems DO happen on a regular basis), and by that leave developers free to spend their time on more sensible things (I'd rather have my team doing testing than writing more code than they need to)

Re:Missing the point

[Anonymous Coward]

What do you propose a script that doesn't find its input file should do? The *RIGHT THING* (in a unix context) is to print an error message on stderr and return a non-zero exit code. Guess what that code does? (assuming you had writting it correctly, that is... it should be 'or', not '||')

Re:Missing the point

[Anonymous Coward]

So I suppose in your real programming, you don't use exception handling?

Re:Missing the point

[Anonymous Coward]

On a side note for Perl, one thing I always hated were the examples that had something like "open( FH, "file/path" ) || die "Could not open file!" . $!; I mean, come one, you don't want your script to just quit if it encounters an error...how about putting in an example of error handling other than the script throwing up its hands and quitting! LOL.

perldoc -f eval
cpan i /error/

Do make some attempt to understand what you're criticizing. LOL.

Re:Missing the point

[prockcore]

I mean, come one, you don't want your script to just quit if it encounters an error...how about putting in an example of error handling other than the script throwing up its hands and quitting!

Sometimes you do. Here's an example that totally bit us in the ass a few weeks ago. We have a quickeys script that uses a quark manager, any files routed to a specific person get automatically converted and saved to a server, then the files get marked to show that they've been converted.

Now under OSX and OS9, if the network dies, our quark manager will automatically reconnect to the network, but our mounts won't autoremount. So the script ends up saving the files to a random location, and marking them complete.

I would much rather have the script die when it couldn't find the directory, rather than save them in a random place and mark them complete.

Whenever the network dies, someone has to go through and find all those files by hand.

Re:Missing the point

[ChaosDiscord]

Something that holds your hand and lets you code while remaining oblivious to the realities of the scripting/programming environment is a bad thing IMHO.

Try reading (or re-reading if necessary) the docs. FTSH does not allow you to remain oblivious. What FTSH adds is powerful exception and recovery system for shell. Sure you can rely on the default behavior (no exceptions, bail on any error), in that case at least it's better than plain old Bourne shell (which by default cheerfully charges onward on an error). But if you want to do error handling FTSH provides a much more powerful recovery system ("Try this block again up to six times").

I mean, come one, you don't want your script to just quit if it encounters an error...

For most people "script" means a small, specialized program designed to do a single job in a relatively controlled environment. If the assumptions of the script aren't valid then there usually any any sort of error correction possible, all you can do is write a complain to standard error and exit with a non-zero return value (that is to say, die() in Perl). If your nightly web stat reporting scripts can't find the log files there isn't anything it can do other than bail out.

Re:Missing the point

[soxos]

One of my first programming mentors gave me this pervasive bit of advice.

Programming is Error-Handling

Re:Missing the point

[vague]

Read the last post at

http://c2.com/cgi/wiki?FailFast

It's not about ignoring errors, it's about the central idea that you'll never, _ever_, be able to write 100% perfect code, and if you could your code will be so full of error checking that it's both unreadable and, as a result, unmaintainable, masking logic bugs and similar. It's a better economy to come up with better ways to deal with failure than trying to prevent it altogether. And the final solution will be more stable.

This is an important realisation: Failure is inevitable, how you deal with it is what matters.

Re:Missing the point

[x3ro]

Hmm ... I don't think it will encourage bad code any more than Java's exceptions and try/catch statements. It's a good thing because encourages you as a script writer to consider the possibility of your statements failing.

Fault Tolerance at its best

[Anonymous Coward]

For a scripting language, just follow the VB approach which is already ultra efficient and completely fault-tolerant:

On Error Resume Next

Re:Fault Tolerance at its best

[digitalsurgeon]

yeah rock solid fault tolerance, hehe, it might even "Resume Next" if u show how delete the executable from your file system.

One of the few who get it apparently.

[SmallFurryCreature]

This is indeed little more then the wrapper that you describe. Yet most seem to comment on its non-claimed properties of fixing the programmers errors. Wich it really really doesn't. In fact it is worse since this one would happily keep trying to execute a command like "rm -Rf / home/me/tmp".

I have often had to write such wrappers myself. Sure even easier/better would have been if somebody added this to say BASH as an extension but perhaps that is not possible.

How often have you needed to write horrible bash code just to pull data from an unreliable source and ended up either with a script that worked totally blind "command && command && command &&" wich never reported if it failed for days on end or ended up with several pages just to catch all the damn network errors that could occur.

I will definitly be giving this little language a try in the near future. Just another tool for the smart sys-admin. (smart people write as little code as possible. Let others work for you)

Re:One of the few who get it apparently.

[Krach42]

Hm... what happens when the first command in your catch says

rm datafile

With no -f? This is a failure condition if the condition is false, thus it would throw the code into an infinite failure loop until the timeout.

Like you're saying though, if it's a programming error, it won't get fixed, and in someways, even worse, it adds new dimensions to think about.

Not good

[Molina the Bofh]

joshua:~#rm -Rf //tmp
Probable typing error detected. Parsed as rm -Rf / /tmp

Re:Not good

[gowen]

Thats pretty much the canonical example of why DWIM is a bad idea...

Re:Not good

[pclminion]

But... //tmp is equivalent to /tmp, there's no problem with that command :-)

For that matter you could say ////////tmp

Re:Not good

[SEWilco]

Obviously that is the canonical example of why DWIM needs to be run through DWIM so any erroneous interpretations will be corrected.

Why don't use screen?

[Yag]

Screen is a terminal which can survive connection problems. You can start your script, detach terminal, and then came back 10 minutes later and watch what its doing. I know, that's not "fault tolerant", but, most of the times, its enaugh.

Re:Why don't use screen?

[Anonymous Coward]

Because screen does not do what this ftsh does. Read The Fuckin' Article ...

A different kind a fault tolerance

[de Selby]

I've always wanted a shell that deletes into a 'garbage' folder, but in a native way so programs calling a delete function would also. I've also wanted a 'file versions' feature to bring safety to accidently overwriting. Then it would really be tolerant of user faults.

While we're at it: a config file library so every config file is the same format; exportable functions so gimp can export gmp.imageResize fileName 800 600 to the shell; and a codecs folder with libraries for image, video, document, and data compression.

Not every might see that last one's benefit, but I think if every app exported its format there (quicktime, realmedia) and let it be universally called, apps would be judged by interface, not filetype support.

Another idea: make every shortcut in X the config file. That way, a simple copy+edit makes two easily created+accessed differently configed programs. (I don't know about network-wide configs, though.)

Re:A different kind a fault tolerance

[Anonymous Coward]

I've always wanted a shell that deletes into a 'garbage' folder, but in a native way so programs calling a delete function would also.

Fairly easily done with a preloaded library that intercepts the unlink syscall. Basic idea:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <limits.h>

int unlink(const char *filename)
{
char buf[PATH_MAX];

snprintf(buf, sizeof buf, "%s/%s/%s",
getenv("HOME") ? getenv("HOME") : "/",
[make your new path name here],
filename
);

rename(filename, buf);

return 0;
}

To use: gcc -shared unlink.c -o unlink.so. Then do:

LD_PRELOAD=$PWD/unlink.so rm foobar

This will call the specially crafted unlink() and do whatever it says.

I've omitted a real trash-making algorithm (one that avoids clashes, etc) and I've done no error checking (make sure snprintf() returns a value < (sizeof buf), and that rename() returns 0), but this is just a rough example. Oh yeah, real code wouldn't use rename() because that won't work over different filesystems. But hey, it isn't meant to be used, just as a guide.

Re:A different kind a fault tolerance

[Anonymous Coward]

libtrash

Re:A different kind a fault tolerance

[Anonymous Coward]

possible to bypass if the app uses syscalls directly and not libc calls (freepascal apps for example)

the other way would be to use ptrace :)

A distributed shell ...

[Anonymous Coward]

... was mentioned a few months back in one of the magazines I pick up almost monthly (forget which one out of the several it was).

I think the shell was called dsh. I believe this is the project site: http://dsh.sourceforge.net/

Are the aims of this fault tolerant shell and dsh the same? I'm not a programmer, but I'm trying to teach myself *nix system administration.

Eventually I'm hoping to cluster some older x86 systems I'm going to get at auction together for a Beowulf cluster. It sounds to me like one if not both of these two shells might come in handy!

OK, wise guys...

[JAPrufrock]

I'm working with Grid and ftsh as we speak. I'm a physicist, not a professional coder. I write reasonable code, but I'm no purist. With that said...

ftsh has great utility in the realm it's written for. Obviously, it's not a basis for installing kernels or doing password authentication. In a Grid (not just distributed) environment, things break for all sorts of reasons all the time. You're dealing with a Friendly Admin on another system, one who may well be unaffiliated with your institution, project or field of study. He doesn't have any particular reason to consult with you about system changes.

Now you find yourself writing a grid diagnostic or submitter or job manager. One does not need strongly typed compiled languages for this. Shell scripts are almost always more efficient to write, and the speed difference is unimportant. Right now, most Grid submitters are being written in bash or Python or some such. Bash sucks for exception handling of the sort we're talking about. Python does better with its try: statements, but there's room for improvement. ftsh is a good choice for a sublayer to these scripts. One writes some of the machinery that actually interacts with the Grid nodes and supervisors in this easy, clear and flexible form.

Now there are a lot of specific points to answer:

One needs a Windows port to be able to make the Grid software we write in Linux available to the poor drones who are stuck with Win boxes.

This is not a code spellchecker or coding environment. At all.

This is not a crutch for inadequate programmers. This is a collection of methods to deal with a specific set of recalcitrant problems.

As I was pointing out before, this is, after all, an unstable system. One is using diverse resources on diverse platforms in many countries at many institutions. I appreciate the comment made by unixbob about operating in heterogeneous environments.

This isn't a substitute for wget. One uses wget as an example because it's clear.

The "pull" model breaks down immediately when there is no unified environment, as is described on infrastructures.org. When you're not the admin, and your software has to be wiped out the minute your job is done, "push" is the only way to do it. This is the case with most Grid computing right now (that I know about)

All the woe and doom about the sloppy coding and letting the environment correct your deficiencies is... ill-thought-out. That's what a compiler is, folks. Should we all be coding in machine language? :) Use the right tool for the job and save time.

I do agree, however, that one should indeed hone one's craft. Sloppy coding in projects of importance is inexcusable (M$). There is no reason to stick to strict exception handling, however, in the applications being discussed by ftsh's developers (the same folks who brought you Condor). When code becomes 3/4 exception handling, even when the specific exceptions don't matter, there's a problem, IMHO. :)

rm -rf $(TEMPFILE) /dev/null

[Domini]

This was an obscure typo bug I found this morning (after 3 months)

Argh.

Wish the shell would have added the (obvious) ' > ' :P

Re:rm -rf $(TEMPFILE) /dev/null

[Anonymous Coward]

-- Wish the shell would have added the (obvious) ' > ' :P

If that is your fix, I already see another bug :)

"set -e" will go a long way to helping you

[divec]

The article says:

#!/bin/sh

cd /work/foo
rm -rf bar
cp -r /fresh/data .

Suppose that the /work filesystem is temporarily unavailable, perhaps due to an NFS failure. The cd command will fail and print a message on the console. The shell will ignore this error result -- it is primarily designed as a user interface tool -- and proceed to execute the rm and cp in the directory it happened to be before.

That shell script can be improved a lot by using " set -e " to exit on failure, as follows:
#!/bin/sh

set -e # exit on failure

cd /work/foo
rm -rf bar
cp -r /fresh/data .

This means that, if any command in the script fails, the script will exit immediately, instead of carrying on blindly.

The script's exit status will be non-zero, indicating failure. If it was called by another script, and that had "set -e", then that too will exit immediately. This is a little bit like exceptions in some other languages.

Re:"set -e" will go a long way to helping you

[jaoswald]

How does this get an "interesting" mod?

Scripts that simply die when they hit an error are not "fault tolerant" or "robust." They are simply par for the course in the UNIX universe, like regular expressions, which work great if everything is just right, and fail terribly when things change.

"Fault tolerant" systems KEEP WORKING in the presence of unexpected events. Emitting an error message and then leaving the user back at the command prompt is exactly the opposite.

This feature is a "little bit" like exceptions in the way that death is a "little bit" like a bump on the head. Real exceptions let you detect errors and then DEAL WITH THEM. They don't just prevent you from deadly accidents.

Re:"set -e" will go a long way to helping you

[Anonymous Coward]

Well, when you are depending on something that ISN'T fault tolerant (in this example, NFS), it's hard to "KEEP WORKING in the presence of unexpected events" as you say.

How would your robust program react when it's supposed to delete a file and it can't get into the directory where it lives? The behavior without -e "KEEPS WORKING" but destroys data that we didn't want to lose, so it fits your definition but is obviously not desirable.

Re:"set -e" will go a long way to helping you

[ChaosDiscord]

That shell script can be improved a lot by using " set -e " to exit on failure, as follows:

Indeed that will help for the simple case. However it doesn't really scale when you have blocks smaller than an entire script which you want to abort on a failure (and that a higher level block can recover from). You can try nesting scripts, but that's a nuisance to code and debug. -e is also useless if you want to actually do error recovery (including the popular "loop until it works").

Re:"set -e" will go a long way to helping you

[jaoswald]

Where did my definition equate destroying data with working?

The key thing, of course, is the definition of "working" you use, which depends on your unique requirements. If you think that exit() is a useful solution to a problem, you have a pretty easy problem to solve. Most useful software, however, has to provide a service; simply giving up because of something outside the client's control fails to provide the required service.

The answer to your question is that if you have to operate over an unreliable link you create a *protocol* that allows for retransmission of data, confirmation, etc., to create a robust, reliable virtual link over that unreliable link. You might queue transactions locally until the link is available, for instance.

"Work or die" is not the foundation of such a protocol. It is a symptom of the half-assed approach which is typical of UNIX hackers. Other symptoms include "use errno to signal failure," "everything should be a text file," and "use regular expressions to parse text files."

Re:"set -e" will go a long way to helping you

[divec]

How does this get an "interesting" mod? [...] Scripts that simply die when they hit an error are not "fault tolerant" or "robust." [...] "Fault tolerant" systems KEEP WORKING in the presence of unexpected events. Emitting an error message and then leaving the user back at the command prompt is exactly the opposite.

An excellent point - I failed to mention how you handle the "exception". In your outer script:

if ! ./inner-script.sh ; then
# Handle the exception here
fi

Then the outer script doesn't abort, it just runs your "exception handler".

Re:"set -e" will go a long way to helping you

[jaoswald]

Except, you have no real information as to what failure mode you encountered. You get a simple "pass/fail" result, with no preservation of the state at the point of failure.

Unless you partition your system's operations into tiny portions, each of which can only fail in one way, you don't know how to handle what went wrong.

You also left out the part where the handler wishes to try the underlying task again after waiting some appropriate time. Your simple wrapper then becomes a more complicated loop.

That is, you are suggesting to build, by hand, using shell scripting primitives, the functionality that a fault-tolerant error handling system attempts to provide.

Re:"set -e" will go a long way to helping you

[Arjuna]

Untrue. You have the return code of the script which can indicate quite a number of conditions, or you can capture the standard output/error of the script if you don't need it for other things.

I've often thought it would be nice to have the option of allowing environment changes or some more structured information get back to the parent. XMLBash anyone?

Re:"set -e" will go a long way to helping you

[jaoswald]

Well, the parent poster didn't do that, now, did he?

* His code *didn't* check the possible return codes, and would have become much more complicated if he had. That, and the called script would have had to collect and branch on all the outcomes of its constituents in order to determine which return code was appropriate.

* Most UNIX commands have a very limited, often undocumented, protocol of return codes (yet another example of the prevailing UNIX philosophy of keeping things simple rather than robust.) So it is hard to tell what happened beyond "worked" or "didn't."

* Capturing stderr is an even less robust technique; all sorts of UNIX utilities spew random stuff to stderr or the console whenever they feel like it. One needs to capture every important failure mode in a text search, and add yet more code to detect and select among all the possibilties. And don't forget the important information that ends up in log files or on the console as well.

There's no doubt that given enough time and effort one can make a script that is as bulletproof as one might like. But that is a far cry from "oh, just add one magic line to your script, and most of your problems are solved" which started this discussion.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Been tried

[old_unicorn]

Fault tolerant languages, have been tried, and never found to be a food idea. Computer languages are very precise for a reason - What if it reads "der *.*" as "del *.*" not "dir *.*"???

Re:Been tried

[vidarh]

RTFA. You're lightyears away from what it's about.

Golden hammer

[sangdrax]

Ofcourse bash can do it as well using the proper constructions. That is not the point. Care should be taken not to view bash as a golden hammer when it comes to shell scripting. The same goes for 'ftsh', ofcourse. It won't try to replace bash for every script out there.

The author merely thinks it would be nice to have a shell in which such fault-tolerant constructions are natural by design. Just to save people headaches when writing simple scripts which are there to get some job done, not to waste time dealing with every single possible failure and time-out by hand.

Code not very tolerant of my machine!

[nick_urbanik]

I finished building the shell after I changed the code that uses a non-standard way of printing the usage message, show_help() in src/ftsh.c. In emacs, I replaced ^\(.*\)\\$ with "\1", and then went back and changed the lines that did not end in a backslash, removed the beginning and ending quotes.

Then it compiled (on Fedora Core 1).

Then it failed the functions test, because my computer does not have the file /etc/networks. For a fault tolerant shell, it does not seem very tolerant of my machine! After sudo touch /etc/networks, make succeeded.

Anyway, those were the only two problems, and now it's installed. Let's see if it's worth building into an RPM package.

If you want a fault tolerant scripting language

[WetCat]

Erlang (http://www.erlang.org) has it.
You can have multiple linked interpreters and
even fault-tolerant database!
It is a scripting language.
From the FAQ:
1.1. In a nutshell, what is Erlang?
Erlang is a general-purpose programming language and runtime environment. Erlang has built-in support for concurrency, distribution and fault tolerance. Erlang is used in several large telecommunication systems from Ericsson. The most popular implementation of Erlang is available as open source from the open source erlang site.

ACID Filesystems

[NZheretic]

For a system like this to be truly effective you would need an operating system which supported a truly transactional filesystem.

Remounting a filesystem with ACID on, a process sets a rollback point , executing a series of commands with the operating system keeping a record of the changes to the filesystem made by the process and its children. The process would inform the OS to either commit or rollback the changes.

This still raises questions on how to deal with with two or more competing "transactional" processes which rely on read information which another process chooses to rollback to an early state.

Re:ACID Filesystems

[Animats]

Exactly.

UCLA Locus, a "grid computing" UNIX variant from the 1980s, had this. Basic file semantics were atomic:

• When a file was opened for writing, the system created a new file, which shared all the pages of the original file but was set up for copy-on-write.
• When the file was closed normally, the new file replaced the old file as an atomic operation.
• If the program exited abnormally, the new file was discarded.
• A program could explicitly call "commit" to commit the changes to a file as an atomic operation, or call "revert" to take the file back to its original state.

Thus, if a program failed, its changes to files were automatically backed out. This is usually what you want to happen.

The cases when you don't want that are few. Either you have a log file, to which you append only, or you have a true random-access database. Those take some special handling. But other than simple text log files, only a modest number of programs need such special handling.

UCLA Locus was way ahead of its time. Linux needs file semantics like that.

mod parent up

[xlurker]

excellent comment

this works also:

( cmd1 && cmd2 && cmd3) || echo something went wrong

but "set -e" hits the nail on its head

additionally you could also "set -x" to see where the script exited exactly

Push doesn't scale.

[Moderation abuser]

The system you pull from is a distribution server, all it does is distribute files. If it's slow, it's slow for all the machines sucking data and you need a bigger infrastructure. If it's down, the client scripts fail safe and do nothing.

Even here, pull scales better than push, look at a web server as an example thousands of machines sucking web pages from a server is not uncommon. Try pushing those pages out to the same number of machines.

Push methodologies simply don't scale, I've been there, done that and it's a bad architecture for more than trivial numbers of machines and I'm not the only one to notice:

http://www.infrastructures.org/bootstrap/pushpul l. shtml

Re:First Real Post

[Anonymous Coward]

try for 30 minutes
cd /tmp
rm -f data
forany host in xxx yyy zzz
wget http://${host}/fresh.data data
end
end

easy enough in any standard shell, bash for exanple:

SECONDS=0
while [ $SECONDS -lt $((30*60)) ]; do
cd /tmp
rm -f data
for host in xxx yyy zzz
do
wget http://${host}/fresh.data data && break
done
done

Re:this can essentially already be done in /bin/ba

[vidarh]

First of all, your comment about fault tolerant coding encouraging sloppy coding is either misguided or just plain stupid. Not catching and handling error conditions correctly IS sloppy coding. Catching and handling error conditions correctly IS fault tolerant coding.

ftsh provides a mechanism aimed at making it easier to catch and handle error conditions correctly. How does that encourage sloppy coding?

As for your bash example, it only demonstrates the most basic capability of ftsh - exiting on error. It does not differentiate between conditions where the error is likely to be temporary - which happens all the time in distributed systems - and conditions where immediate termination is justified.

What's with all the negativity?

[AxelTorvalds]

I know that if you need a ton of fault tolerance in your shell scripts that you should probably be using a different language but every time I look at any complex systems, not just a signle app but a system, there is always shell script glue. More importantly, I've never seen a shell script that checked the return codes of everything at best they look at a few key components and report on their success of failure. Exceptions would be nice.

I think perl is where it is because so many people use it as "super script." To me that says, a) we recode all the Bourne and csh and bash in perl or b) we look at why people do shell scripting in perl or other languages and add that to the shell. I couldn't tell you which is right. It's a neat idea though and I'm glad they made it.

A real example I can think of, I had a test machine that had some kind of ext3 corruption and so it mounted up in read-only mode when it booted. I spent time diagnosing an application error in our application because nothing caught that; these are redhat type startup scripts. I noticed that our app couldn't write logs and began to debug the system. More interestingly, a dozen or so start-up scripts failed to start up critical components and their failure wasn't noticed. If you can't write to the filesystem, you can't create a socket(AF_UNIX) and all sort's of things go tits up then. If that's how you debug it's only going to get more difficult as you add more and more complexity, you have to detect the lower level failures and report them. Perversely, this wouldn't have been noticed had a different partition been read-only. Turns out that a drive was going bad. Had it been a different partition, it would have been noticed at catastrophic system failure time when the drive died.

I've done a fair amount of embedded work and there is always a test for new guys, you can tell the new guy (new college grad, whatever) because he skips half or more of the error checking in his code. You know printf returns a value? Funnier still, if you develop something like a consumer app in embedded space, you'll eventually see things like printf fail. We know it never should, but with 20,000+ users in different environments and what not, things like that can and do fail and usually point to a greater problem, like a dead drive or something. Instead of logging/alerting something to the critical and unusual printf failure, the app fails in a different way because this printf failed. Heaven forbid that it was sprintf that failed and then you shove bad data in to a database or configuration file and not just fail the system but corrupt the data too. Inspite of all of that, even veterans will forget error checking at times, it's a common bug and so having higher level tools to help assist, like exception in the shell can only be a good thing.

Re:this can essentially already be done in /bin/ba

[xlurker]

the habit of calling something stupid that you don't understand doesn't polish your image.

"fault-tolerant coding encouraging sloppy coding" was addressing a common misconception in this thread:

namely a shell that will recognize misspelt commands such as the spelling correction mechnism in tcsh

the rest of the comment was an example that demonstrated that the simple functionaly of ftsh could be done in bash *without* entending bash with a new statement operator. Not a solution. The solution would be to extend an existing shell and not write a complete new shell. If the ftsh happens to be a fork of an existing shell, well then good for him, but I didn't find any reference anywhere in the ftsh site, manual or paper. So basically the ftsh is a new shell that can do

• try-statements and error-catching
• but doesn't yet have the mileage of standard true-and-tried shells that is used by many and where the bugs a few (when's the last time you had your unix command line shell crash on you, eh?)

basically this is just a repost of the original post. hope you understand now. flame on.

Why there are no pipes

[The Pim]

What do you lose by using something like this?

Well.. besides pipes of course ;)

Funny you should mention this, because I was going to write something about pipes. Getting pipes right with good error semantics is hard. For all the "just use set +e in bash" weenies out there, try running

#!/bin/sh -e
cat nosuchfile | echo hello

Where's your error?

If you think about the unix process and pipe primitives, you will see the difficulty. To create a pipeline, you normally fork, create the pipe, fork again, and run the two ends of the pipeline in the two sub-processes. This is scalable to deeply nested pipelines, but has a cost: Only one of the sub-processes is a child of the shell, so only one exit status can be monitored. To work around this, you really need to build a mini-OS environment on top of unix.

This demonstrates that unix was fundamentally not designed with concern for error semantics (consider Erlang as a diametric example). And this, I'm sure, is why ftsh doesn't have pipes (yet).

Re:Why there are no pipes

[Mignon]

you really need to build a mini-OS environment on top of unix.

Insert Emacs joke here.

Re:Why there are no pipes

[pclminion]

Only one of the sub-processes is a child of the shell, so only one exit status can be monitored. To work around this, you really need to build a mini-OS environment on top of unix.

Am I the only one who doesn't see how the second sentence follows from the first?

Solution: fork each child process directly from the shell.

Re:Why there are no pipes

[The Pim]

Solution: fork each child process directly from the shell.

The catch (which I hinted at in the original post) is that pipelines may be nested. Say you have

(a | b) | (c | d)

And say you want to avoid wasting resources by having multiple shell processes that are just waiting for other processes (ie, you only want to fork 4 times total, which is what bash would do). Then you need to fork a, b, c, and d from the main shell. Which means you need to open all the pipes in the main shell, and do lots of duping and closing in the forked children before exec. Plus, creating all these pipes in the shell uses lots of fds, so you probably want to consider scheduling things to minimize the number of pipes open at once.

This is what I mean by a mini-OS. At least, that's what it felt like when I tried implementing the above. If you have some clever way to simplify things, I'd love to hear.

Re:Why there are no pipes

[Nevyn]

Funny you should mention this, because I was going to write something about pipes. Getting pipes right with good error semantics is hard.

You should seperate the syntax of pipes with the implementation of calling pipe(). It should be trivial for the shell to transform "foo | bar" into "foo -> __internal; bar -< __internal;", it also seems like it shouldn't be that hard to use the pipe() on the backend but they should at least get the syntax right.

I'm also not convinced that you wouldn't want a more high level view, so you can move the script around machines ... and possible add dependancies between machines (like Tivoli). And, unsupprisingly, the throw/catch type error detection is just as buggy as real error detection.

Re:Why there are no pipes

[The Pim]

You should seperate the syntax of pipes with the implementation of calling pipe().

Absolutely--that's what I had in mind with the term mini-OS. You're pointing out the logical consequence: that the mini-OS pipe operation needn't even call unix pipe(2) at all. It could use temporary files, named pipes, or some other IPC.

Regarding the use of internal variables, it wasn't clear to me whether they were stored in memory, or used named pipes. I suspect the former, in which case this isn't really an acceptible implementation of pipes.

And, unsupprisingly, the throw/catch type error detection is just as buggy as real error detection.

Yeah, fault handling is hard. That's why I mentioned Erlang, which was designed with error containment as a core feature.

process relocation

[Anonymous Coward]

I know this isn't exactly on-topic, but I always thought the idea of moving a running process from one machine to another would be pretty cool. Is there any OS that implements this? I mean the actual program state, not something like screen (which lets you detach and re-attach from another shell).

I think this is "what you mean"...

[Anonymous Coward]

Ever heard of DWIM? :-)

OT: Bowling for Columbine

[Anonymous Coward]

thanks for the link in your sig, very interesting :)

Re:OT: Bowling for Columbine

[HyperbolicParabaloid]

dito. I saw the sig, followed the link, and enjoyed what he had to say.
I still haven't seen Bowling for Columbine, but am more interested than ever now. I really enjoyed "Roger and Me".

Re:this can essentially already be done in /bin/ba

[vidarh]

Trying to justify your response by introducing context to your original post that didn't exist doesn't polish your image.

If your comment on fault-tolerant coding adressed shells that correct misspellings, that doesn't make it any more insightful. There is no reason to assume such a shell would encourage sloppy coding - the many examples in this article demonstrates very well why no such shell has been written: It could create dangerous situations from otherwise harmless mistakes - there is no indication that it would in any way make "sloppy coding" any safer, and hence no indication that it would encourage sloppy coding.

As for extending an existing shell, you specifically made the point that you didn't need to extend bash to provide the functionality of ftsh, while your example only show the most trivial functionality. You now "clarify" in a way that to me reads as backtracking. Fine. Yes you can handle errors in bash in a slightly more elegant way than the ftsh website indicates. I pointed out that this was only a small part of the functionality ftsh provides.

Your complaints about ftsh being written from scratch just reinforces the impression I got of your first message. Yes, writing a shell from scratch just to add this functionality might be a waste from your perspective. But we're talking about about 4000 lines of code here - hardly years of work - where providing the "try" functionality and error handling is the ONLY focus (gee, I actually bothered downloading and looking at the source).

It is not a bash replacement. It is not something you would want to use as an interactive shell. It is a shell with the specific purpose of providing more fault tolerant access to distributed, error prone resources. Which is also how it is presented on the website: As a tool to solve a specific problem in a generic, reusable way. Very much in the Unix tradition.

It's possible it would be significantly smaller if written as a bash script. But if it came at the cost of a syntactic kludge it would defeat much of the purpose: Making the appropriate error behavior default throughout the script.

Another Clippy Moment

[whiteranger99x]

*poof*
I see you're trying to vaporize your UNIX system. Would you like to:

• Pay SCO $600 for rights to your Linux system
• Download and Install a W4r3Z version of Windows XP (c'mon, everyone's doin it!)

Spell checkers and homonyms

[mdfst13]

The biggest problem with spell checkers is that they don't handle homonyms at all. Further, their handling of misspelled words can be weak as well: how many times have you seen allot where the person meant "a lot." The problem is that the person used the common misspelling alot, which the spell checker identifies as allot. T.f. the person changes to allot instead of correcting to a lot, totally changing the meaning of the sentence (it would actually have been easier to read the misspelled alot).

To, too, and two and for and four are common examples of words that go through a spell checker misspelled. Replacing the contraction 've with of is another example (e.g. should've is written should of). A spell checker will never catch that. People who rely on spell checkers (even if they feedback immediately) will find that they miss a lot of common misspellings.

Spell checkers are *not* a substitute for knowing how words are spelled. Spelling seems simple (just a dictionary lookup) but is actually heavily context dependent. Computer programs remain bad at that (and even a separate human being can be bad at that; to do this properly, one must have as much understanding of that particular topic as the original writer; clearly, the best person to resolve these issues is the original writer, who knows what she or he was trying to say).

Re:Spell checkers and homonyms

[Jerf]

Spell checkers are *not* a substitute for knowing how words are spelled.

Of course not. But using a spell checker means having time to learn about the homonyms, instead of endlessly playing catch up.

You still predicated your post on "relying" on spell checkers; I'm saying that people learn from good spell checkers. That people can't learn everything from a spell checker is hardly a reason to throw the baby out with the bath water and insist that people use inferior learning techniques anyhow!

A kid that can spell accurately is in a much better position to learn about homonyms then one for whom spelling is still a mysterious, abstract art that involves guessing at things.

Re:Spell checkers and homonyms

[Anonymous Coward]

I think you mean homophone.

Sorry, there just isn't a case for it.

[Moderation abuser]

Nope, pull doesn't break down in heterogeneously managed/owned or even platformed environments. It works best in these environments. The www is an ideal example of such, apt-get is another, seti@home is another, distributed.net, I could go on.

In Grid based computing environments, jobs queue until they can be started, and as it happens, they tend to be architected as pull based systems whether you see that as a user or not.

Again, I can't see a reason for ftsh in this case. I'm sure there's a niche for it somewhere but even their examples don't give any good reason for it's existence.

Their cd example for instance resolves to a silly script:

----- FFS!Bloody slashdot compressionfilter
#!/bin/sh

cd /work/foo
rm -rf bar
cp -r /fresh/data .
----------Bloody slashdot compressionfilter

*They* resolve this to:

-----
#!/bin/sh

for attempt in 1 2 3
cd /work/foo
if [ ! $? ]
then
echo "cd failed, trying again..."
sleep 5
else
break
fi
done

if [ ! $? ]
then
echo "couldn't cd, giving up..."
return 1
fi
-----Bloody slashdot compressionfilter

Just for the cd statement, it's just silly. It simply shows that they are either trying to justify the writing of another shell or they don't understand how to shell script.

It should resolve to something more like:

-----
#!/bin/sh

JOBDONE=1
until $JOBDONE
do
cd /work/foo && \
rm -rf bar && \
cp -r /fresh/data . && \
JOBDONE=0

sleep 30
done
-----

You simply don't have to test every statement the way they suggest you do. It's a ridiculous premise. Even the decades old bourne shell has an easy way to only execute commands if they depend on a previous success.

i get it, but i don't...

[mydigitalself]

given his example:
cd /work/foo
rm -rf bar
cp -r /fresh/data

would this not suffice:
cd /work/foo && rm -rf bar && cp -r /fresh/data

my undertanding of && was that it only executes in the previous command didn't throw some sort of error. i understand its not as powerful as what he's talking about, but there is some degree of fault tolerance there.

secondly, i don't know about you, but i would be very uncomfortable with something that tries a few thousand times or for a particular amount of time - it could really lock up threads or disk IO quite sunstantially if not well considered.

Mod up!

[Ayanami Rei]

The language even allows you to create file "variables" which are safely allocated and destroyed during the operation of a script, analogous to automatic varibles in C++/Java or what-have-you. The point of this language is that it is entirely focused on exception handling. This is _excellent_ programming practice.

mmmm... lisp in emacs?

[Ayanami Rei]

Haven't used it myself, but I hear it's a rather "complete" environment, chock full of these and more you-wouldn't-know-why-you'd-need-it-until-you-need -it features.

Oh, exception handling w.r.t. perl/python.

[Ayanami Rei]

I think what you might use ftsh for is as an upper layer that calls the perl/python scripts. The perl/python scripts would just wrap some logic in a try, then exit on exception. ftsh would see it, and then attempt to restart the task as required. The ftsh can be the resource management layer, while the perl/python scripts do the complicated grunt work that may fail.

The problem is...

[Ayanami Rei]

ftsh creates a new session for each process, with the intention of using signals to forcably kill it if it runs over a time limitation. Handling this properly for multiple processes in parallel and getting the resource clean-up right sounds a little tricky.

Already exists...

[Nimloth]

MS-DOS is already a fault tolerant shell, it can work with Windows.

Well...

[Qbertino]

It's similat to computer spell checkers. Ever since people started relying on these, their spelling has gone way downhill simly because they don't bother thinking. ...
Computer do all the spelling for them.
They don;t need a spell checker. ...
This si even worse.
what the user means, get get it wrong half tyhe time....
etc.

Well, I can't say whether or not you're using a spellchecker (i'd guess not) but I'd bet my right arm your spelling couldn't get any worse if you did.
I strongly suggest you try one.

Cooperation

[robnauta]

In order for this to work, it might need changes in the OS level.
Imagine you access a block/char device or an NFS mounted directory and the device driver never returns from the system call. Your script would hang, and a kill would produce a zombie process.
If you want fault tolerance, you'd have to have a timeout mechanism for all device drivers. But if you read from /dev/mt0 and the tape needs rewinding and it takes 6 minutes, you don't want to have your script aborted after 5 minutes.

Re:Cooperation

[Animats]

If you want fault tolerance, you'd have to have a timeout mechanism for all device drivers.

QNX has that. Because everything is done by message passing, including I/O and networking, and you can time out waits for messages, you can time out of anything. This is essential in the real-time world, where being stuck waiting for something is unacceptable.

The QNX syntax is Posix-compatible. You can call TimerTimeout before any system call or message pass, and if the timer runs out before the call finishes, the call returns with errno=ETIMEDOUT.

It's also possible to force an exit from any system call that blocks via a signal, if necessary.

Re:Cooperation

[whittrash]

I am not an expert in programming, but I work on fault tolerances in building processes. Tolerances come in two general categories, one is 'slop' the other is a factored tolerance where you predict where and how the processes will interact.

Slop, AKA 'quick and dirty', is the easiest and the cheapest way to design. It requires little forethought and marginal design ability. You begin by applying generic and standardized processes, you do not venture into any customization, you do not get fancy in any way. You specifically avoid all situations where two things have to line up and interact. This method is inneficient, not in terms of design resources, but in maintanance and operational resources (in computing, I imagine it would be memory, bandwidth et cetera). In architectural terms, this would be a shanty town. A million people can live in a shanty town, but it isn't pretty.

My preferred method is a precisely defined solution.
1. You start off by designing for the inherent demands of the overall system, which look at the long range extremes of usage, you decide how robust you want your system to be and game out 'all' likely failure points at intersections and interactions of the overall architecture. You benchmark past experience and develop overall standards for performance. The precise details of this overall architecture will determine day to day maintanance and operation requirements. You need to be aware of not only the the way elements within the system interact, but the physical properties or specifications of each element to be sure you know what you are doing and how they will perform. You need to be an expert.
2. As a co-continuous problem you design each particular system to a level of robustness based on its appropriate use. You need to assign safety factors, and acceptable failure levels so as not to waste resources while maintaining protection. In some cases failure can be catastrophic and people can die (the space shuttle systems, 911 call centers) or worse lose money (banking). The best example of a fault tolerant system I can think of besides the internet is the Interstate Highway System, which was originally designed as a transportation system that would remain operational in the event of a nuclear strike (its form isn't an accident, it was designed to specific criteria). In architectural terms, this kind of fault tolerance is a skyscraper city. Skyscrapers are durable and beautiful but are extremely expensive to design and construct. A skyscraper could stand for 100 years without maintanance of any kind, where a shanty will fall to pieces in 5 years.

Fault tolerance needs to be put in perspective. It is only useful based on how much it costs to fix and maintain a system and its 'aesthetic' value (it is always hard to sell an ugly or stupid design). A $1 million system for a $1 hundred dollar problem is not effective unless it is a critical application (a missile or a laser for eye surgery). It may well be cheaper to employ a person or ten to keep some system up, rather than make it 100% bullet proof (most web sites can fail for a few minutes an no one will care). It depends on the situation. As far as aesthetics of your work, I don't know anyone who wants to live in a shanytown, they have a tendency to get cholera, malaria et cetera and die, it isn't necessary to live like that. We invented trailer parks to solve that problem and it is now Americas #1 form of affordable housing.

Our friends at Darpa are researching this problem and have a good outline of these issues.

Re:First Real Post

[danielrendall]

Um... no, that doesn't do the same thing. The whole point of ftsh is that the 'try' block encloses a set of statements which must all be executed or it fails. If the 'cd /tmp' fails, bash will blindly run the 'rm -f data' anyway, whereas ftsh will stop and jump to the start of the try block to have another go.

Re:First Real Post

[Anonymous Coward]

set -e

The problem with DWIM

[simon_clarkstone]

If you look in the Jargon Dictionary under DWIM, you will find a much older example of this mis-smartness.

FTSH is an exception system for shell programming

[ChaosDiscord]

What's with all of the people claiming that FTSH will ruin the world because it makes it easier to be a sloppy programmer. Did you freaking read the documentation?

To massively oversimplify, FTSH adds exceptions to shell scripting. Is that really so horrible? Is of line-after-line of "if [$? -eq 0] then" really an improvement? Welcome to the 1980's, we've discovered that programming languages should try and minimize the amount of time you spent typing the same thing over and over again. Human beings are bad at repetitive behavior, avoid repetition if you can.

Similarlly FTSH provides looping constructs to simplify the common case of "Try until it works, or until some timer or counter runs out." Less programmer time wasted coding Yet Another Loop, less opportunities for a stupid slip-up while coding that loop.

If you're so bothered by the possibility of people ignoring return codes it should please you to know that FTSH forces you to appreciate that return codes are very uncertain things. Did diff return 1 because the files are different, or because the linker failed to find a required library? Ultimately all you can say is that diff failed.

Christ, did C++ and Java get this sort of reaming early on? "How horrible, exceptions mean that you don't have to check return codes at every single level."

A lot of this has been done before.

[Ayanami Rei]

A system library that makes unlink "safe", called entomb.

Versioning can be provided by some userspace-fs combinations (atfs comes to mind). I also understand reiser5 will have built-in versioning support.

GConf is supposed to be the new way to store configuration information. Although I see a lot of new programs using XML (for example, ogle) which is kinda nice. But there's nothing wrong with the shell/windows .ini file syntax, as far as I'm concerned.

Gimp does let you do stuff in the shell, using script-fu and a perl script. However, what you really want is "convert" which is a part of the ImageMagick toolset, which handles tons of file formats, converting between them, and doing simple transformations (crop, scale, add text, color balance, etc) from the command line.

mplayer is the other equivalent that is basically a command-line front end to ffmpeg and win32 codecs.

You need a user-space tool to effectively expose the features of codecs or a library set. This is what mplayer (and also xine) do, besides just play media files.

Until realplayer, quicktime, etc. all conform to the same API, you won't see an interface driven engine.

On windows, this tool is called "graphedit" and it is part of the DirectShow SDK.

Finally, your X "shortcut" idea is superflouous. You don't really make "shortcuts" in linux desktop environments. What you do is make icons that run actions. So, you edit your configuration file, save it as something else in your home directory, then copy the icon, modify it so that the -f or -config option (application dependant) points to your config file, and off you go!

Let's not try to re-invent the wheel too much. Unix environments give you a lot of tools already to do much of what you suggest.

Re:A lot of this has been done before.

[de Selby]

"A system library that makes unlink "safe", called entomb ... Versioning can be provided by some userspace-fs combinations (atfs comes to mind). I also understand reiser5 will have built-in versioning support."

It looks like about one out of every thousand people knows about these (except Reiser5). Is it your mission to try everything? How the hell...

I've posted these two ideas several times before or have seen them proposed several times before to the universal reactions: "good idea" or "bad idea". Never, "been done already".

I was thinking of doing it myself. 'Saves me from that!

And you'd think a unified GUI/CLI trashcan would be something the usibility guys would go for. I think it's a mystery why Linux doesn't install like that. But you prolly already know of a distribution that does this...

"GConf is supposed to be the new way to store configuration information. Although I see a lot of new programs using XML (for example, ogle) which is kinda nice. But there's nothing wrong with the shell/windows .ini file syntax, as far as I'm concerned."

I like the .ini file syntax. I sometimes like XML. I was just thinking that making consistancy painfully easy would be a good thing.

"Gimp does let you do stuff in the shell, using script-fu and a perl script. However, what you really want is "convert" which is a part of the ImageMagick toolset, which handles tons of file formats, converting between them, and doing simple transformations (crop, scale, add text, color balance, etc) from the command line."

Well, what I really want is one ever-present image library (the one that every app everywhere will use to load .jpeg, etc.) to export some functionality, gimp to export some, abiword to export some of its stuff, etc. (but not some bulky add-on package). If this is already implemented, super.

"Finally, your X "shortcut" idea is superflouous. You don't really make "shortcuts" in linux desktop environments. What you do is make icons that run actions. So, you edit your configuration file, save it as something else in your home directory, then copy the icon, modify it so that the -f or -config option (application dependant) points to your config file, and off you go!"

That's exactly what I was trying to avoid. I think a simple copy+edit of the icon you already see is much more user friendly and fast.

"Let's not try to re-invent the wheel too much. Unix environments give you a lot of tools already to do much of what you suggest."

Funny. I came up with some of these ideas (config files, exporting) specifically because I saw too much reinventing the wheel.

Another thing Unix gives you: A million programs (be it at instillation or on the internet), no friggin' idea where to start. Looking around at things on FreshMeat and SourceForge tells me some large percentage of app development is not to outdo an existing program, but is instead out of ignorance of the 40 other implementations (most of which started for the same reason).

Ah well.

Simple.

[Ayanami Rei]

Either the reader process blocks until it's resources have been released by an outstanding transaction, or the process provisionally accepts a non-guarantee of correctness (signaled by the read) that says "I'll send you a signal if they finally commit or rollback", but program logic would have to handle it.

We might assume that the inital read would give EBUSY or something, which would be the signal that the data is unstable. A non-transaction aware program might quit, or back off and retry. A transaction aware program might use a special ioctl with the same offset would allow successive reads to the area with the guarantee of a signal delivery on end-of-transaction.

Presumably the process would block at the absolute last second, waiting for the go/no go signal... or it would ignore the warning with a result that would be unreliable.

Non-transaction aware programs would either halt or loop endlessly because of the EBUSY condition on read.

MOSIX

[Ayanami Rei]

You are right, partially

[SmallFurryCreature]

I am a sysadmin and a lot of times I just need a short script to automate a task. These are not super critical things or even that complex. I could hire someone to type them in every so often. However they quickly become complex when I try to build in very basic error handling. In fact as basic as the examples shown on the website, oops didn't work, retry it for a while then if it still doesn't work, do something else like tell me.

This would allow me to use somebody elses code to save me work. Call me a lazy bastard. For sysadmins that is a compliment.

You are wrong when you see shell scripting as a step up. In many ways when you are a sysadmin shell scripting remains one of the basic needs.

What I disagree with is that this will teach people wrong programming routines. Look at the code out there. They don't need a retry shell script to do that. Bad code has been with us since we invented programming. Rejoice. It means someone is going to be paid to clean it up.

Is the FTSH acronym pronounced...

[GodLived]

As a side-bar, is the FTSH acronym pronounced... fetish?

All kidding aside, this sounds like a great idea.

As for the comments about encouraging sloppy code, it is clear those posters have never worked in demanding moving-target environments. The kinds of errors encountered cannot be solved easily in code - this extension would help.

As for the comments on "you can do this in Perl, Python, and ", this is true, but if I know Bash and want tolerance, why should I have to learn a new language to get it? Likely all I'm doing is copying files, forking off subprocesses, and the like.

For the comments on "why another shell," I would tend to agree that it would be best integrated into Bash - but then, you change the implementation of Bash, create incompatible situations, and have to retest volumes of existing scripts. It's best to have this as a separate shell with close look/feel semantics to Bash (or Csh).

dwim? (Do what I mean)

[Pyrrus]

Something perhaps like this?

OT: Regarding your signature...

[synaptik]

--
The Romans didn't find algebra very challenging, because X was always 10

Does that mean they were (wait for it...) existentialists?

Re:OT: Regarding your signature...

[eschwinge]

If I had mod points, this would be +1 right now. Maybe I've been working on error-checking and proofing stuff for way too long, but this totally cracked me up.

Of course

[Pan T. Hose]

how many times have you hacked something together in perl that ended up being relied on for some pretty important stuff, only to find 6 months down the track that there's some condition (db connects fine, but fails halfway through script execution as an example) you didn't consider and the whole thing just collapses in a heap - a nasty to recover heap cause you didn't write much logging code either.

The proper answer to this loaded and overly complex question is of course: "Countless." I did not need a "fault tolerant" shell, though, but real asynchronous I/O with good exception handling subsystem and events. That is why I look forward to Perl 6 and Parrot.

This would REALLY be useful when you're connecting to services external to yourself - network glitches cause more problems with my code than ANYTHING else, and it's a pain in the arse to write code to deal with it gracefully. i'd really really like to see a universal "try this for 5 minutes" wrapper, which, if it still failed, you'd only have one exit condition to worry about. hey, what the hell, maybe i'll spend a few days and write one myself.

Good luck then.

Askemos

[duplicatedAccount]

I spent the better part of three year implementing a fault tollerant programming environment and released it under GPL. Please visit Askemos to find it.

Upon a mem fault... Fork it... ;]

[Anonymous Coward]

a mem fault... map around it (linux has a kernel patch for this), stack audit (linux secure kernel mod), and Pre-Fork a secure shell instance for each script (ha ha ha, kill -9 it when done). Run your shell config scripts off a ROM media, and spawn the whole process inside a virtual machine unlike your actual hardware (PC in PPC etc.) resistant indeed... ;]

Re:First Real Post

[IpalindromeI]

Please explain how you are accurately updating $SECONDS once each second (especially while wget is running), as well as how you stop the wget process if it exceeds the 30 minute time limit. Thank you.

what?

[sydres]

/bin/sh! ain't good enough for you, huh? why not make one of them new fangled gui shells why back in the day assembly was state of the art now you want a fault tolerent shell interpreter. bah weakness.

One good turn just leads to annother bad turn

[MC_Cancer_Pants]

I may be wrong, but we're talking about integrated logic that will decide for itself when somthing went wrong and come up with the right way of dealing with it, right? I understand that this is deeper than an error-handler. I think that it sounds wonderfully useful. I don't, however, see it as an end-all solution. I think that it is a nice concept but what if you're making a request that will only be true 1 in 1 million times? Consider a conditional statement which is false 99.9% of the time, yet is extremely crucial to be checked in case it happens to be that .1% of the time. that means that you're adding redundancy to 99.9% of your requests. Alot of times errors occur because somthing is simply broken and requesting somthing again won't fix it, but slow down everything else.

I hope maybe I made some sense, I'm not an english major nor a philosopher, but it made sense in my mind.

reversible commands

[Anonymous Coward]

A long time ago, I had a similar idea: reversible commands.

Basicly, the regular commands are wrapped with some scripting (perl, python, whatever) that saves the reverse of whatever commands the shell interprets. If an error is found, it just applies each of the reversed commands in reverse order, and the system is back where you started.

This would be good for a lot of sysadmin scripts like creating users, moving files, etc. where the system should be kept in a known state.

There are many commands which can't be reversed, but with some clever help, most things can be.

gzip gunzip
mv mv
rm (mv foo .trash) mv .trash/foo .

never really spent much time on it, but maybe it's an idea related to what this topic is about.

sort of like transaction logs in a database too

I got ur troll RIGHT HERE.

[Idealius]

AND the Linux Zealots wonder why their OS can't be user friendly.. Someone comes up with an idea to make the shell harder to screw up for a beginner scripter and all the penguins lose it.

Sad, really.

MEP?

[Anonymous Coward]

I don't think MEPs (Members of the European Parliament) existed in Babbage's day.

Thinking about the shortcut as link idea.

[Ayanami Rei]

I think there's already a way to implement it using a combination of file type actions and MIME magic. I will post a quick tutorial on how to get it working for you.

Question: what software were you hoping to operate with multiple configuration files in particular? Also, what environment are you using? (KDE, Gnome, CDE, etc. etc.) That way I can tailor the example to your specific situation.

Steps for GNOME in Redhat/Mandrake/what-have-you..

[Ayanami Rei]

First, we need to make sure that your configuration file is always "tagged" so that we can make it behave specially no matter how many modifications or forks you make.
We'll use MIME types for this purpose.

The easiest way is the most direct: edit the file "/etc/gnome-vfs-mime-magic" with a text editor as root (this is the only time you need root access).

Somewhere near the bottom, add a line like this:
0 [TAB] string [TAB] \#somethingunique [TAB] mime/type

[TAB] indicates you can use as many tabs or spaces to line up the data like the other columns in the file, but it doesn't actually matter to the system.
The first part (zero, then "string") indicates that the type tagger should look at the beginning of the file for a string to match. The string that needs to be at the beginning of the file is \#somethingunique, which can be whatever you want, but the first letter should be a comment character for the configuration file format (usually a hash mark "#").

The backslash is necessary to "escape" the hash mark because this gnome-vfs-mime-magic file ALSO uses the hash mark for a comment character, and you need to tell it that you mean for it to look for the hash mark, and not comment the rest of the line out. :-)

The last part, "mime/type" is any MIME type of your choosing. (MIME types are things like text/html, image/jpeg, etc. etc. and are used to uniquely identify file formats).
It has to be unique for the application you want to automate. So if it was "Visual Boy Advance", I might choose personal/vba-conf. I'm going to suggest you use the prefix "personal" to let you know that YOU added it for your own use.

Okay, now that you've done that, find the configuration file you want to be able to run by just double clicking.

Add in a line (it has to be the FIRST line) that matches the #somethingunique you chose earlier. Now, whenever the GNOME file browser sees this file, it will detect it as "personal/vba-conf" or whatever instead of the default "text/plain" for text configuration files.

Okay, now open up "File Types and Programs" from your "foot" menu, under Preferences. Click "Add File Type...". Fill in the description, choose a goofy icon, and in the MIME type box, type personal/vba-conf or whatever the new MIME type you're using for these configuration files.

There doesn't seem to be an appropriate Category choice for what we're doing, so I just picked "Misc".

Go all the way down to the Actions section, and go to Program to Run. Type in the full path to the program that uses the configuartion file. Make sure to add any command line options you want, and make the last one be the one that lets you choose a _different_ configuration file. In the case of VisualBoyAdvance, that's the -c option. So I type in: /usr/local/bin/VisualBoyAdvance -c
in the box. The system will always add the path of the configuration file as a last option to the program, so that's why you need to end it with the -c or whatever.

If you need a terminal window open to interact with the program or to read error messages, make sure "Run in Terminal" is checked.

When you think you have it straight, hit "OK".

That's it.

Do a ps -ef , and grep for "nautilis". Kill that process, so it restarts the file manager for good measure.

Now whenever you double click the configruation file, it'll call your custom command line with that configuration file as an argument.

You can copy the configuration file, edit it, rename it, move it, whatever, and it'll still launch the program when you double click it.

Only thing you have to watch out for is to make sure you keep that first "magic" line in there. A fresh, virgin configuration file will not have this magic behavior until you add the magic line. Of course, any modified copies you make will.

What do you think of that?