Slashdot Mirror
Ultra-Stable Software Design in C++?
null_functor asks: "I need to create an ultra-stable, crash-free application in C++. Sadly, the programming language cannot be changed due to reasons of efficiency and availability of core libraries. The application can be naturally divided into several modules, such as GUI, core data structures, a persistent object storage mechanism, a distributed communication module and several core algorithms. Basically, it allows users to crunch a god-awful amount of data over several computing nodes. The application is meant to primarily run on Linux, but should be portable to Windows without much difficulty." While there's more to this, what strategies should a developer take to insure that the resulting program is as crash-free as possible?
"I'm thinking of decoupling the modules physically so that, even if one crashes/becomes unstable (say, the distributed communication module encounters a segmentation fault, has a memory leak or a deadlock), the others remain alive, detect the error, and silently re-start the offending 'module'. Sure, there is no guarantee that the bug won't resurface in the module's new incarnation, but (I'm guessing!) it at least reduces the number of absolute system failures.
How can I actually implement such a decoupling? What tools (System V IPC/custom socket-based message-queue system/DCE/CORBA? my knowledge of options is embarrassingly trivial :-( ) would you suggest should be used? Ideally, I'd want the function call abstraction to be available just like in, say, Java RMI.
And while we are at it, are there any software _design patterns_ that specifically tackle the stability issue?"
How can I actually implement such a decoupling? What tools (System V IPC/custom socket-based message-queue system/DCE/CORBA? my knowledge of options is embarrassingly trivial :-( ) would you suggest should be used? Ideally, I'd want the function call abstraction to be available just like in, say, Java RMI.
And while we are at it, are there any software _design patterns_ that specifically tackle the stability issue?"
I'd hate to say it, but you might want to SERIOUSLY consider managed code. You could build some of the parts in C++ if need to be, but doing it purely in C++ seems like a bad idea to me. You're asking for a silver bullet that just doesn't exist...but managed code is getting faster and can be pretty stable.
> Sadly, the programming language cannot be changed due to reasons of efficiency and availability of core libraries.
You can easily embed C/C++ in other languages. Take a look at Inline::CPP, for example. With code like:
use Inline CPP;
print "9 + 16 = ", add(9, 16), "\n";
print "9 - 16 = ", subtract(9, 16), "\n";
__END__
__CPP__
int add(int x, int y) {
return x + y;
}
int subtract(int x, int y) {
return x - y;
}
you can put the parts that need to be fast in C++, and the parts that need to be easy in Perl. (If you do the GUI in perl, you won't have to worry about portability or memory allocation. And the app will be fast, because the computation logic is written in C++.)
> The application can be naturally divided into several modules, such as GUI, core data structures, a persistent object storage mechanism, a distributed communication module and several core algorithms.
Yup. There's no need for the GUI to know how to do computations, remember. The more separate components you have, the more reliable your application (can) be. Make sure you have good specs for communication between components. Ideally, someone will be able to write one component without having the other one to "test" with. For testing, write unit tests that emulate the specs... and make sure your tests are correct!
My other car is first.
try not to de-reference any NULL pointers and you should be ok..
If you're willing to compromise performance to the point that you can use CORBA for IPC, then you should be more than willing to write it in the language of your choice, within reason. C, C#, C++, Java, all are far faster than your CORBA transport.
If you can provide more details about the specific requirements, you might get more informed responses. As it is, though, your stated goals really don't seem to add up.
Even as stated, I would write the core in a highly tuned fashion (although C++ might not be my best choice for this), then write the GUI in the language of your choice, quite frankly. Optimise the bottlenecks (ie: your core processing) for speed, optimise everything else for maintainability and ease of development.
You're special forces then? That's great! I just love your olympics!
THere is no silver bullet for what you describe other than sound development practices. The best results in this area are acheived by teams who are constantly refining their processes based on lessons learned in previous software iterations.
Bulletproof code isn't cheap, but it can be done.
1. Write the whole thing in Python.
2. Once it's bullet-proof, replace each function and object with C++ code.
3. Profit.
Follow NASA's advice... http://www.fastcompany.com/online/06/writestuff.ht ml
Make sure his name is something like "Bjarne" or "Knuth".
"Speaking the Truth in times of universal deceit is a revolutionary act." -- George Orwell
First, consider how complex you want to make the system. The decoupling is a good idea, I think. However, I don't think that having modules automatically restart one another is a good idea; it introduces a whole slew of other problems. At most I'd say use a watchdog process (principle of single responsibility).
Furthermore, you're crunching large amounts of data, so I'm guessing batch processing. If you can have the application not be a server, then you simplify things a lot. Make it a utility that takes data on standard input and runs whatever analysis you need, and duct tape it together with cron or a simple program that watches for new input files.
Also, I'd like to suggest that you consider whether other languages could be efficient for the task. For example, Java is pretty good numerically, and as far as your libraries go, see if you can use SWIG to generate JNI wrappers. Also, then you get Java RMI.
Next, get them down to one platform. It's *way* easier to develop software with tight constraints on a single platform (versus multiple platforms). Investigate QNX: a reliable operating system (though admittedly quirky) with a beautiful IPC API. In any case, make sure you get a well-tested library with message queues, etc. You don't want to be using raw sockets; you could but that's just another pain in the ass on top of everything else.
Last, figure out what the cost of a failure is. Getting that last few percent of reliability is very very expensive. Unless you're a pacemaker or respirator, the cost of failure is probably not as high as the cost of five nines of uptime.
When coding something that needs to be stable, you need to keep your ego aside and concentrate on the task at hand. Stick with tried and true methods don't go with any algorithm that you are not 100% comfortable with even if it makes the code less ugly. Be sure to follow good practices make many function/methods, and make each one as simple as possible, makes it easier to check each function for bugs when they are simple. Secondly document it like you never want to touch the code again (in code and out of code), you want to know what is going on at all time and the bigger it gets the larger chance you could get lost in your own code. When working in a team and you are in someone else's code document that you did the change.
Next take into account what causes most Crashes.
Bad/Overflow memory allocation.
Memory leaks.
Endless loops.
Bad calls to the hardware.
Bad calls to the OS.
Deadlock
If you are going to decouple modules keep in mind that you will need to do as much processing as possible with minimum message passing and allow for mirrors so if one system is down and other can take its place, without killing the network.
For IPC I tend to like TCP/IP Client server. But that is because it tends to offer a common platform independence and allows for expansion across the network. Or try other Server Methods such as a good SQL server Where you can put all the shared data in one spot and get it back. But not knowing the actual requirements it may just be a stupid idea.
I would suggest that you also ask in other places other then Slashdot. While there are many experts on this topic there are also equal if not greater amount of kids on there who think they know what they are talking about, or they have there ego in this technology/or method.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
State machines help make sure you cover (almost) all possibles cases your app may encounter.
Here's a great framework to start with:
http://www.quantum-leaps.com/products/qf.htm
And the book:
http://www.quantum-leaps.com/writings/book.htm
I think perhaps what you REALLY mean here by stability is Fault Tolerance. It's impossible to write code that has zero defects, outside of any trivial examples. Real Code Has Real Defects. Now, as you talk about modular design and being able to restart modules, you're talking about, not stability, but fault tolerance; the ability of the application to recognize and recover from faults. For instance, you can't necessarily guarantee that the module on machine A running task B won't die, hell the computer could accidently fry, but if your application was Fault Tolerant then the application would kick off another process somewhere else on computer C to rerun job B. Stable systems aren't built necessarily by trying to write defect-free code, but by recognizing that defects will occur and architecting the system in such a way that it can recover from them. Here you need to be concerned about things like transactions, data roll-back, consistency, techniques (active vs. passive, warm vs. cold). The key thing is before you even write a LINE of this C++ code, make sure that you have a complete, comprehensive ARCHITECTURE for your application that will gracefully handle faults.
be assertive
...that you are about to board.
I've spent over a decade refining how best to create stable, great software. And guess what? I still learn things every day. If you are really new to enterprise-grade software, the best thing you can do is search amazon and choose 3 to 5 great books about writing stable, bug-free enterprise code and just start reading and scheming. Give yourself lots of time. Be neurotic, type-A, attention to every detail, stay up at night wondering how your system could fail and what you can do to prevent it. Some immediate thoughts, however:
1. Good hardware. Obviously. Redundancy everything, self-diagnosing, etc. How can things go wrong? What will go wrong? How can I know when something is going wrong? How I can fix it quickly without impacting the system? Etc.
2. Enterprise grade (n-tier) architecture: You'll definitely want to do something where you have a database running on one or two (or more) machines, at least two business servers and at least two web servers. Redundancy is good. As you suggested, a setup like this lets you isolate problems (and provides for better security in general).
3. Test, test, test. From the very start, every day to the very end. Start coding by writing test suites for your code. Learn about unit testing, black box testing, user testing, regression testing, etc. And hire developers and QA whose sole job is test, test, test using great automated testing software.
4. Profile. Stress-load-test. Know how your system responds to all scenarios. Feel comfortable knowing the limits of your system. There should be no surprises.
5. Assert. Learn the magic of assert(). If your code isn't at least 25% asserts, you are not trying hard enough.
I told myself I was only going to write the first five thoughts that came to my mind, otherwise I could spend weeks trying to answer your question!
Use TPS reports. You'll thank me later.
Take the cheese to sickbay, the doctor should see it as soon as possible - B'Elanna Torres, "Learning Curve"
I wish I could mod the article +5 Funny.
70e808a22cb027cde4a6abddf6435d55
If your develop safety critical code, or anything that requires hi-rel you need to break down the application into functional testable units, with test fixtures to test each module. Then a integration test framework. You can't create a "verified" correct system with ad-hoc testing. Unless you're very good and you own the whole thing and then it's just you that knows it's right, Ya right.
JimD.
Executive summary of this post: Keep it simple. As simple as it can be while getting the job done. The more buzzwords you think about implementing, the more you need to reconsider whether you really need that whiz-bang feature.
You need to abstract your design into really independent layers, such that the backend processing can be done across linux, windows and even beos slaves simultaneously, and the frontend is viewable via a web interface, fed into excel or whatever. You can't look at this as one big project, but many independent (and more easily verifiable!!) applications cooperating with each other.
My impression from the description is that you want a system like folding@home for corporate customers - they have a whole heap of data they want analyzed (parallel workload across many clients) and a small subset of results they're interested in. Don't make things any more complicated than they have to be - the data sets could simply be files that are partitioned by a master, sent out when requested to client workhorse computers, getting there by http, nfs or whatever, processed, and the results returned into an incoming directory for a simple frontend to tabulate.
The biggest mistake you could make is having one gargantuan application in charge of everything. The race conditions will drive you mad, be they in data access, allocation, retrieval, dispatch or anything else you're trying to manage that the OS could do for you.
Just look at Froogle. Their millions upon millions of store/price listings are fed by people ftp'ing a feed of tab-separated text values.
Don't Hate, Gestate
Extreme programming is your worst enemy on this one. If you need a system that is truly reliable, you cannot take an approach that fundamentally bases its quality controls on a finite number of tests, unless you can test absolutely every possible set of inputs your program can ever receive (legitimately or otherwise).
Testing is good, of course, but for this sort of job, you must have a proper design, such that all components can be properly verified. (And of course, you must have a proper spec against which to verify.) The XP methodolgy is pretty much the antithesis of what's needed here.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
Extreme programming is your friend on this one. Doesn't matter what language you use, test and retest at every change. Testing is the only, only, only way to get extremely stable software outside of formal verification methods.
e nt
Exactly. Three words: Test Driven Development.
Since you're tied to C++, may I suggest CppUnitLite2 1.1...
It's incredible how much more productive you can be writing the tests first (contrary to what you might think initially). I hardly ever need a debugger anymore, and I know that the code I wrote does the right thing, and doesn't adversely affect something else.
Put the unit tests as a post-build step (or a dummy target in a makefile) and any defect will pop up instantly. If you find a bug not covered by your test suite, add a test that reproduces the problem, ensuring that it will never bite you again.
If you're not familiar with TDD, check out Wikipedia for an explanation and some useful external links: http://en.wikipedia.org/wiki/Test_driven_developm
gcc: no input sig
valgrind -v ./myapp [args]
It gives you massive amounts of great information about the memory usage of your program.
The other day I spent nearly 3 hours trying to decode what was happening from walking the backtrace in gdb. Couldn't for the life of me figure out what was happening. Valgrind figured out the problem on the first run and after that, I had a solution in a few minutes.
Highly recommended software, and installed by default on several distributions, AFAIK.
Enjoy!
std::disclaimer<std::legalese> sig=new std::disclaimer; sig->dump(); delete sig;
Sounds like the poor soul is in over his/her eyeballs.
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
Those are low-level programming-jock languages disguised as high-level languages. As long as the punks who program them will have pissing contests in code obfuscation, you can count on having buffer overflows and memory leaks.
The reasons? A unit test suite that implements several million test cases (mostly pseudo-random probes -- the actual test code is about 1/3 the size of the functional code). In fact, the "defects" that hit production were more "oversights"; stuff that didn't get accounted for and hence didn't get implemented.
Just as importantly; every dynamically allocated object just got assigned to a "smart pointer" (see Boost's boost::shared_ptr implementation).
Quite frankly, compared to any Java implementation I've seen, I can't say that "Garbage Collection" would give me anything I didn't get from smart pointers -- and I had sub-millisecond determinism, and objects that destructed precisely when the last reference to them was discarded. The only drawback: loops of self-referencing objects, which are very simple to avoid, and dead trivial if you use Boost's Weak Pointer implementation.
We didn't have access to Boost (which I Highly Recommend using, instead of our reference counted pointer) when we first started the project, so we implemented our own Smart Pointers and Unit Testing frameworks.
I've since worked on "Traditional" C++ applications, and it is literally "night and day" different; trying to do raw dynamic memory allocation without reference counting smart pointers is just insane (for anything beyond the most trivial algorithm). And developing with Unit Testing feels like being beaten with a bat, with a sack tied around your head...
-- -pjk Perry Kundert perry@kundert.ca http://kundert.2y.net
"I need to create an ultra-stable, crash-free application in C++. Sadly, the programming language cannot be changed...
From zero to flame war in under 20 words. Well done!
Yes, some of those do conflict. How to keep things simple AND have fault-tolerence, for example. That's where a good design comes in handy, because you can get a better feel for where you should make the trade-off between certainty of working, certainty of working later on and getting some sleep this side of 2008. It's all a matter of weighing the options and investing time in the place most likely to benefit.
(Because everything is a trade-off, anything listed above may not apply. But then, it may not need to. If you've tested a component thoroughly along all boundaries, a good sample of valid conditions and a good sample of erronious conditions, AND everything has been kept as simple as possible so that really wierd cases are unlikely to crop up, then you may decide you can simplify or eliminate fault-tolerent components. There is no point in catching errors that won't occur. In fact, that adds complexity and violates the Keep It Simple rule.)
Oh, and as this is a networked system, testing should include testing network I/O. Use packet generators if necessary, to see how the system handles erronious packets or massive packet floods. You don't want "perfect" responses (unless you can define what "perfect" means), you want reliable responses. If X occur
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
I've dealt with software that automatically restarts a dead process, and in my experience, it doesn't work so good. If you want ultra-stable software, you want to know what caused the crash and why.
For your situation, where I guess you're doing lots of time consuming computing, I'd think you should also set checkpoints, save intermediate results, or something, so if it does crash, you can restart in the middle instead of going back to 0. (A standard practice when I was analyzing large databases for corruption, a task that could take days)
Do you even lift?
These aren't the 'roids you're looking for.
Jesus christ people he is asking you how he should go about building an ultra-stable application in C++. He told you he *has* to build it in C++ because there are critical libraries and other components that aren't availible in C++. Telling him he shouldn't build it in C++ anyway just isn't helpfull.
I hate to break it to people but there *are* libraries, especially for types of scientific computing, that are only (reasonably) availible in C++ or sometimes FORTRAN. Not only would abandoning these libraries mean he would completely have to reinvent the wheel but also might cause serious compatibility problems not to mention a much greater ongoing maintenence responsibility (he can't just check his program to make sure things still work when someone fixes a library bug).
Moreover, the idea that because he is considering using CORBA, IPC or whatever else speed can't matter enough to require C/C++ is dead wrong. It is true that whatever *parts* of the process are done using these components may not require huge amounts of speed but this doesn't mean one of these components isn't doing something very processor heavy.
In particular what he says sounds like the situation in some areas of scientific computing. If one is writing a program to do some sort of simulation or similar math intensive operations speed can be *very* important in the critical parts of the code but (in some cases) transfering information to the GUI or other components need not be particularly speedy (increasing by an order of magnitude may make a small difference in overall runtime). Imagine a program that does some kind of weather, or nuclear detonation simulation. The cross-processor communication and the core simulation kernel need to be very fast but the GUI and data input components need not be particularly fast. Also it is my understanding that often the critical libraries in this area are often only availible (at least freely) with C/C++ or fortran bindings.
Anyway I think it is important to distingush several different goals, ultra-stability, minimal downtime, and minimal data/computation loss. For instance a climate simulation that may run on a supercomputer for months it is very important to have minimal data/computation loss (i.e. if something goes bad you don't lose months of very valuable supercomputer time) but you need not have ulta-stability or minimal downtime. As long as when any node crashes the simulation can easily be restarted without loss of data there is no problem. On the other hand if you are running a website like slashdot it is minimal downtime that is important it doesn't really matter if some of the web server processes are rebooted once in awhile. If, on the other hand, you are writing code to monitor a nuclear power plant it is ultra-stability that is important (though I can't at the moment think of something that requires distributed processing and ultra-stability but I'm probably just missing something).
So I think the answer depends on what sort of stability you want. If it is important that no individual *node* crashes (though the GUI/other non-core components can crash) then you should pursue the seperation you described above. I have to admit I'm not an expert here but the client-server model (like mysql, X etc.) seems to work well in this context. However, this depends alot on what sort of data you need to transfer. If you just need to send the core setup commands and get back mostly unstructured info (say a grid of tempratures or other simple datasets) then I would suggest sticking with one of the simpler abstractions and don't get lost in CORBA. On the other hand if you need to send back and forth real objects with significant structure then creating your own serialization system/bindings is just asking for bugs.
On the other hand if what you want is minimal data/computation loss, downtime, or any other property where it is the overall system you care about not a crash at any particular node then I suggest concentrating less on dividing any one node into comp
If you liked this thought maybe you would find my blog nice too:
Avoid the latest "big thing" for the core of your project. It's usually specialized, non-portable, etc. The standard template library for C++ (for example) is here to stay, with tested algorithms that are safer and faster than you can usually write (because they are optimized for the platform you compile on). For the GUI, on the other hand, you may be better off with a GUI-based language/tool. That's less likely to be portable, but that's the way GUIs work.
Next, spend some time upfront on your design, with things like use cases, sequence diagrams, and other visualization tools to help you understand just what you want to happen in best case situations as well as failures. The level of detail/formality required is a moving target, so update as needed. You should have a solid error detection/correction plan so that you can design each component to follow it. Also design for test and with logging - it will help you while debugging, while testing, and while fixing the bug the customer is seeing.
Make sure management will allow sufficient time for testing. A lot more lip service goes into support for testing than actual schedule and money. Your test plan should be as bulletproof as your design.
That's my 2 cents. And a random book recommendation: books like Scott Meyers' "Effective " provide info on effective/error reducing ways to use the language/libraries, but won't help you get started with the architecture.
Way back in 1993, thanks to a three month schedule delay in shipping the original Apple Power PC hardware, Graphing Calculator 1.0 had the luxury of four months of QA, during which a colleague and I added no features and did an exhaustive code review. Combine that with being the only substantial PowerPC native application, so everyone with prototype hardware played with it a lot, resulted in that product having a more thorough QA than anything I had ever worked on before or since. It also helped that we started with a mature ten year old code base which had been heavily tested while shipping for years. Combine that with a complete lack of any management or marketing pressure on features, allowed us to focus solely on stability for months.
As a result, for ten years Apple technical support would tell customers experiencing unexplained system problems to run the Graphing Calculator Demo mode overnight, and if it crashed, they classified that as a *hardware* failure. I like to think of that as the theoretical limit of software robustness.
Sadly, it was a unique and irreproducible combination of circumstance which allowed so much effort to be focused on quality. Releases after 1.0 were not nearly so robust.
He didn't complain about anything... you added that part. He asked for advice. Were more people to do so, instead of relying on their overconfidence (like you) to get jobs done, maybe we wouldn't have computer instability be such an issue.
I completely agree with the parent, but I want to add a few things.
Before you do your design, you should try understand all of the system's requirements. Just how much reliability is really needed? Are lives at stake? Is a lot of money at stake? Is a system failure just inconvenient? Remember that each extra "9" in reliability multiplies the cost of the project by 10, so make sure you understand just how reliable your system must be before you start.
Once you understand the system's requirements, make sure that your fault-tolerant design is testable. Then design your tests (before you write a single line of code). Don't make the mistake of leaving testing until after implementation. Make sure all of your interfaces are specified and that the subsystems are decoupled enough such that you can test each unit individually and thoroughly. What happens if a subsystem receives bad input? What happens if a subsystem takes longer than expected to respond? How long is too long? What happens if a subsystem returns bad output? You should design your system such that you can replace any subsystem with a misbehaving (test) subsystem and that your overall system responds appropriately. In the process of designing your test, you should expect to find many design defects (or design glosses).
You should also get someone to peer-review your design (and your tests) with the mindset of making your system fail. People with different backgrounds will have different experiences of what can go wrong; don't expect that you have thought of everything yourself.
By now you may be thinking "that's a lot of extra work". You're right, it is. But it's all necessary. You can scale back some of this depending on how much reliablility you actually need, which is why it's essential that you understand your requirements. You also don't have to do all the work yourself. In fact, you probably shouldn't. You should get someone else to work on the test side of things.
By the way, one essential subsystem to modularaize is the allocation of resources. You will find a lot of defects just by inserting a memory allocator that occassionally simulates out-of-resource conditions. There are tools to do this, but they don't seem to be portable to different operating systems.
first and foremost, use good coding techniques. This means use exception handling where appropriate, use standard containers over hand rolled data structures (prefer std::string over char arrays, this will help prevent almost all common string based buffer overflows alone), and follow good style guidelines.
As for a GUI programming, if you are strictly tied to c++, i would recommend QT (www.trolltech.com) they have a fabulous API (takes getting used to, but it makes sense once you do). Nice part about QT is that it is source portable to just about every major platform (X11, Win32, Mac).
It is possible to write reliable, fault tolerate code in c++ (realize please that perfect code is impossible in any language), it just has to be well thought out and done right.
proxy
I really think you had better qualify this. IMO, assertion failures do not *cause* problems; they are messengers, and the message is always this: "Your program is broken."
I don't think you want to *recover* from a broken state. I think you want to debug it -- find out what went wrong, fix the code, recompile, test, and re-deploy.
Because, if you get to the point where an assertion fails, it means the state of the program is corrupted, and therefore you can't trust any part of it; e.g., you can't trust error-recovery code to be well-behaved. The best you can do is bring everything to a halt and fix the bug.
There are rare exceptions (no pun intended) to this rule, but for the most part, if you write out a condition and say, "if this is false, then the program has a bug", then you have some explaining to do if you *don't* want to use an assertion.
The suggestions I have seen here so far seem to boil down to "Don't do it that way". Sometimes that's not possible. If it truly has to be C++, and it truly has to be as fast as possible and as bug free as possible, there are a few guidelines that can help:
1. Unless the GUI will be I/O bound, and that's unlikely, try to write it in a safer language that has better GUI support.
2. Make all your classes small and simple, and create test harnesses that are as complete as possible. Try to make the classes simple enough that they can be individually tested in such a way that all code paths are exercised.
3. Check your arguments. This includes checking for invalid combinations, and arguments that are invalid given the state of the object.
4. Don't use new or pointers directly. If there may be multiple references to an object, then reference count it and create handle classes that hold the references so all instantiation is controlled, and all destruction is implicit. Make these handles STL compatible, and never pass around pointers to them.
5. Try to design the application to fail fast and recover from failure. For example, maintain the state of work being done in discrete transactions that can be aborted if a failure is detected. This can be on disk or in memory depending on your performance needs. This could be combined with the ability to restart the app in a new process and have it pick up where the last one left off.
6. Have the app keep track of its memory usage, and be prepared to recover from memory leaks, possibly by restarting as in item 5.
7. If the compiler you're using supports structured exceptions, then use them. They can degrade performance a bit, but they can also enable you to recover from NULL pointer exceptions.
8. If you have multiple threads, then to avoid both the performance hit from context switches and the chance of deadlocks, don't let them access the same data directly. Instead, have them communicate through lock free queue structures. That way, all your main threads can pretty much spin freely. Spawn worker threads for any I/O or other operations that can block. A context switch can take as much time as thousands of instructions. You want to use as much of every time slice as possible.
9. Keep the number of main threads down to the number of CPU's or less. That way, except for the times when the CPU is being used by the OS or other processes, (should be relatively rare) each non blocked thread gets its own CPU.
10. Have an experienced QA team, that understands their job goes beyond unit testing.
Now here's a few that are always important, but for what you want to do, they become critical.
11. Have the design laid out at least roughly before you start.
12. If at all possible, don't let requirements change in midstream.
13. Overestimate the time it will take very generously. You will probably still be crunched.
pornking
Rather than investing too much time and effort in creating a complicated crash-free program, just make sure your application can recover from a crash, and then use a process management application that restarts the program on it's node when it is detected to not be running properly.
It's simple to write a 100% correct program that checks the health of your main application, and restart it when it isn't responding.
http://pcblues.com - Digits and Wood
I partially agree.
If your code is unstable in a way that memory leaks and segmentation faults are not only a "remote possibility" but a - even if only rarely - reoccuring event, then any safeguards you implement won't be overly sucessfull, unless you fix the code that causes the errors first. (Disclaimer: There is no perfect code. Even if there were no bugs in the code, the program has still the "remote possibility" to crash due to errors in the hardware / OS)
That said, garbage collection or not is a different discussion. Some say it is bad and breed lazy programmers, while others argue (I amongst them) that it is a terrific tool for designers, since it almost eliminates the occurance of memory leaks (unless you do some really bad programming) and it might even speed up your program
+++ MELON MELON MELON +++ Out of Cheese Error +++ redo from start +++
Using the good-old Ada programming language (for which a new standard should be issued this year), you can go closer to what you're looking for (though I'm not sure your goal is realistic).
Here's a pointer to the new standard : http://adaic.org/standards/05rm/html/RM-TTL.html
With Ada:
It probably won't solve every of your problems, but it might help.
For a free, quite strong Ada compiler, have a look at https://libre2.adacore.com/ (it's based on GCC).
Oh yes, Ada is a statically, strongly, strictly typed language (e.g. the compiler won't let you assign an integer to a float variable). My opinion is that it's a Good Thing for critical programs. Useless to restart a "type war" on this subject ;-)
Good luck.
In my experience decoupling and automatic restarting is a recipe for failure. You set yourself up for all sorts of race conditions. For instance, if a module is unresponsive for a while but not crashing, do you restart it? And if you do, what if the original module finishes its grand execution plan and comes back up after a minute?
No, I'd go for:
* A "monolithic" application with module separation provided by OO design. At least you know that either your whole application is there, or it isn't. No inconsistencies between modules because of individual module re-starts, and if the app breaks, restart the whole thing. Starting the app is the code path you've tested, restarting separate modules usually isn't (and even if it were, there's usually 2^27324 different situations to test, i.e., all possible combinations of modules failing in any sort of way).
* Use smart pointers exclusively, preferably Boost's shared_ptr. Use weak pointers (Boost provides an implementation for that as well) to automatically break reference cycles.
* For error handling, use exception handling exclusively. Incredibly many bugs are caused by ignored return codes.
* Use "auto" objects for all resources that you acquire and that need to be released at the end of a code section. Cleanup that doesn't happen when a code path encounters an exception can cause resource leakage, instability and hangups (locks, anyone?). In my programming practice, when I allocate a resource (memory, refcount, open/close a recordset, etc.), I always wrap it in an auto object immediately, so that I can forget about managing it through all the code paths that follow.
* Use the correctness features that the language provides: write const-correct code from the start.
* Use automated testing right from the start, both unit testing and integration testing. If you don't do this, you will be forever tied to whatever bad design decisions you make in the first months of the project. Automated testing allows you to always make large implementation changes, giving you confidence that it will not break existing behaviour.
What this guy really needs is the time-tested, tried-and-true Waterfall development process !
Thomas-
Question 1: what strategies should a developer take to insure that the resulting program is as crash-free as possible?
Answer:
a. Use OO techniques and maintain all objects in your system extremely simple; furthermore, maintain all methods in your system extremely short, well-contained, well-defined.
b. Don't use C++ arrays, ever. Especially not for strings. Use and abuse the STL. is just plain beautiful IMH?O.
c. Check extensively the behaviour of your constructors and destructors.
d. Make a object-lifecycle diagram of each class you program. In the diagram, relate it to the neighboring classes (parents, children, siblings, classes involved in design patterns with, classes aggregated, classes value-aggregated, classes where this is aggregated or value-aggregated)
e. Use, carefully, and always when possible, smart pointers. Remember std::auto_ptr is your best friend -- its limitations are a defining part of its strength. Remember boost::shared_ptr is also a good friend, but its cousin boost::intrusive_ptr is even more friendly -- but use one of those (and their other cousins scoped_{ptr,array}, shared_array, weak_ptr) only in the (rare) cases where auto_ptr does not apply.
f. As a corollary to (e) above, use boost. This is really an extension of (b), too.
Question 2: How can I actually implement such a decoupling?
Answer:
I would use a simple, socket-base, take-my-data, gimme-my-results scheme. It would be network-distributable, easy to detect if some service is or isn't alive via timeouts... If you want something more sofisticated/RMI-like, SOAP (with binary XML or compressed) may be an option. The simpler the better IMHO.
Question 3: are there any software _design patterns_ that specifically tackle the stability issue?
Answer:
All of them? IMHO, DPs can represent huge tool to increase the stability of a system. Take a look athere [WARNING: PDF] (and in the bibliography) for some ideas.
I know many of my posts were self-marketing lately, but if you need someone to work with you, I'll be happy to send you my resume... write me at hmassa (at) gmail.
It's better to be the foot on the boot than the face on the pavement. ~~ tkx Kadin2048
Funny you should that, I read this: D just 5 minutes ago.
In a desperate rush for some reading material for the toilet, I grabbed what must be a 5 year old C/C++ User's Journal from a storage room. The theme of that month's issue was MULTITHREADING.
I thumbed through it and came across an interesting article ``ALWAYS HANDLED ERROR CODES''. The idea being that a lot of errors can go undetected because programmers are lazy about checking return values. And why not, who bothers checking printf()'s return value, for instance?
Simple enough design. The object constructor sets the result, the destructor will abort() the application if the Checked variable is false. The overridden == and != operators evaluate the result, and also set the Checked variable.
In your functions, instead of return SUCCESS; you write return ErrorCode(SUCCESS);
Wondering if anybody does this. If I needed something ULTRA STABLE I guess I might...
You can avoid some of the pitfalls of C++'s need for manual memory management and other problems by simply avoiding them. For instance, never do memory management yourself. How? By using STL containers to do it all for you. Next, avoid fixed arrays. Again, let STL do it for you. And, above all else, never do anything where you don't restrict the length. Since you're using STL for arrays, you're good to go there, and you won't end up running off the end of a character array (because you don't use them!). So what you're left with is doing I/O properly. Always limit the the amount of data you read to the buffer size you have allocated.
I'm sure that there's tons I've left out, but this has worked reasonably well for me. The only problem is that STL can be slow. Sure, map may be O(log(n)), but the constants are huge. Unfortulately, for practical reasons, performance and security are often inversely proportional.
Where do people get this idea? I have ported quite a few applications, and usually the porting done by locating the libraries you need on the new platform, and fix a few oddities in the current platform (like closing sockets in z/OS or switching to unsafe multitasking (p-threads) on windows. Porting to linux is so trivial that I often do it just to get access to the superior tools available there, especially valgrind. GUI is the exception, of course, unless you use a x-platform kit from the beginning.
Which leads me to my recommendations, in no particular order
The above approach works for me. You mileage may vary.
Religion is regarded by the common people as true, by the wise as false, and by rulers as useful.
I am by no means a specialist in this field and of course I do not know whether your project actually allows this approach.
But if I were asked to do this, I would take a database (a stable release of MySql is what I would choose) and use it both as the persistent object storage and communication module. The GUI and the number-crunching module(s) would be set up to primarily communicate through the database, rather than directly with each other. A task state/queue table in the database would inform the modules what tasks have not been assigned yet, are running, are complete, have not returned in the expected time, or have failed. This would make it asynchronous and highly traceable; databases are (supposed to be) good at managing the interactions between multiple user processes and still maintaining data integrity. Admittedly this is not the best approach if you want your results real-time.
The central managament of the processes could be kept minimalistic and simple, and "therefore" robust: Some very simple communication with number-crushing processes to test whether they are alive (a TCP/IP socket read-write might do), re-opening a task that has not returned in an expected time period (if its process still returns later, the newly started process will have to detect that its work was already done, and discard its results instead of writing them back), and perhaps signalling critical task completion to users (by GUI message, e-mail, text message, ...). The central management would not have the startup responsibility for distributed number-crunching modules, that would remain with the local servers they are running on. Such a process can then "knock on the door" of the database, register its presence, and take the next available task, or wait until one is available.
The persistent form of the core data structures would be in database tables, but the modules would of course have their share of the data in memory as class representations of the data structures, defined to be initialized from the database tables and written back to them. These class representations of the data structures then could be in a common library shared by the different modules, but alternatively you might opt for different class representations for e.g. the GUI and the number-crunching modules if that is more efficient (it often is) and even write them in different languages if that is more convenient. I admit that that adds to the amount of code and therefore to the amount of bugs. On the other hand, you could write two "completely" independent implementations of the same task.
If your number-crunching is complex and long, then evaluate whether you can write back intermediate states to the database as a recovery point, or even split the calculations in completely independent modules, each one starting and ending with a given database state. The desirability of this depends, of course, on the balance between I/O and processing costs. If you have modules that are relatively simple and safe but need to work quickly through a large amount of data, you could consider database stored methods for these; not very distributed but it reduces the amount of I/O and they can easily be called by client processes.
The database does not care in what language the different modules are written, so you can then write every one in the language that is most appropriate. For example, there may be no reason at all to write (parts of) the GUI in C++ -- and that is something I would try to avoid. If performance allows it, I would use Java for the GUI, both for portability and simply to avoid the mess of writing user interfaces in C++; in my experience that does not tend to be the most stable solution.
For the C++ part I would start by structuring pretty strongly; write a large number of simple classes instead of a smaller number of complex ones, and test every class before you move on to the next level. The "salami approach" works well if you plan it well. It is perfectly possible to write very ro
I've been following this methodology (Python first, then C++ as/where needed) for a number of years. In all of that time, I've only had one application where I ended up needing to drop into C++ at all. In that case, a couple of pages of Python did translate into a couple of pages of C++, virtually line for line. Heavy use of STL allowed this, as there are a lot of data structures and algorithms there that map more-or-less directly to Python. The main problem was that the STL tended to be either buggy or to have razor-sharp edges upon which to cut oneself.
Python is generally a win because (1) you can write the same working functionality much faster than in C++, and (2) the specifications for apps tend to vary wildly over time, so a high-level language lets you go with the flow.
"Not an actor, but he plays one on TV."
Have you have flown on a commercial airline in thelast 30 years? If so, you trusted your life to software.
Thare is a standard called DO-178B Level A that applies to aircraft software upon which lives depend. There is a saying in the commercial avionics business: "Nobody has ever died from software failure on an airplane, yet." There have been some accidents where software played a role, but I won't quibble with that now.
The point is that safety critical software is developed routinely. It has been developed in asembly language. It has certainly been developed in Ada, C, and sub-sets of C++. It is expensive. Validation of avionics software and certification in an aircraft can easilly cost an order of magnitude more that just writing the software, and writing the software using required processes and producing required artifacts is not cheap either.
Bullshit. C++ written well is portable by default (between windows and linux). There are a few minor issues between linux and sgi.
I agree.
This is by nature one of the biggest strengths of C and C++, how someone could conclude that by using C++ adds some sort of complexity in cross platform development actually amazes me.
If it adds complexity, in comparison to what? I would like to see the poster above you explain what is actually easier to use for diverse application development that is actually better at cross platform.
And if they start with Java, la la, then they need to get a life and see what JAVA is built upon itself.
C and C++ is a great solution for cross plaform development, look at the nature of Linux, BSD, and even NT and then ask why they are as portable as they are. Do people think these OSes would be more portable in another language?
Take Care.
Ah, but test driven development flies in the face of the new government backed, SEI approved software development silver bullet called TSP (Team Software Process). And by following TSP you too can consider just how much better it is than test driven development while waiting for your co-workers to inspect your code for a few months.
1) Learn to use STL.
Do *all* memory management via STL vector/string.
2) Don't ever type "new[]/delete[]".
Just don't do it. Not. Ever. Use std::vector instead.
"Arrays are evil" - the C++ FAQ.
PS: You can still use malloc()/free() but only as a last resort in low-level classes which are designed for data storage.
3) Get a reference-counted pointer and use it.
Automatic memory management...'nuff said.
4) Attach an alarm bell to your "~" key.
If you're writing destructors for classes which don't control system resources (eg. files) then you're probably doing something wrong - see notes 1, 2 and 3.
No sig today...
>
> is just plain beautiful IMH?O.
I'm sorry, but I just can't agree. It might appeal to a mathematician who wants to see everything use functional notation and hates every language except lisp, but to a non-abstract-elite-ivory-tower-mathematician this is absurd. cin is not an array of integers and the use of the adapter obfuscates the fact that you are using a conversion from a char array to an int. The back_inserter also makes it harder to see where the data is going by losing "v" in it. Many would also frown at it for taking a non-const reference, although since it is a standard adaptor it is probably ok.
C++ programmers are often unnaturally attached to efficiency and have to be watchful for template bloat. Your copy generates 88 instructions, whereas an equivalent iterative solution is only 33 instructions long, most of them belonging to the inlined push_back. Not only is the generated machine code smaller, but the source code is smaller as well, and is far more readable, making the algorithm obvious at a glance to any procedural programmer, who make up the majority outside the hopelessly out-of-touch with reality academia.
Academics love integer and float arrays because that's what they usually work with. Scientific simulations produce data in that form and require processing programs that take something from a data file, crunch some numbers, and output something to cout. In the real world people work on user interfaces, databases, and other complicated things, where one normally works with arrays of objects rather than numbers. If you ever tried to apply a functional algorithm to a vector of objects, trying to manipulate some member variables or call a member function, you would know that the result is so hideous that it isn't even worth considering. There is a reason people prefer iterative solutions; they are how the real world works. Reality is algorithmic, not functional, and so are user specifications for the things they want done. Trying to cram them into an abstract mathematical functional model is insanity.
> Use, carefully, and always when possible, smart pointers.
> Remember std::auto_ptr is your best friend
Most of the time, no. While I would not deny the utility of auto_ptr in localized situations manipulating the object state during reallocation, its constant use indicates lack of understanding of object lifecycle in the program. It is fashionable in Java to create objects left and right, without consideration of who is supposed to own them. Hey, just let the garbage collector take care of it! Who cares how long the object lives? Obviously, such immature mentality produces plenty of memory leaks for which Java is so infamous. In a good design object ownership is strictly defined. Objects belong to collections that manage their lifecycle. There ought to be no "dangling" objects that just "hang there". If you don't know to which collection the object belongs, you have no business creating it. If you think your objects are "special", you haven't thought beyond their internal functionality or considered where it fits in your overall design.
> Question 2: How can I actually implement such a decoupling?
> I would use a simple, socket-base, take-my-data, gimme-my-results scheme
And thereby slowing your program to a crawl? There is a reason people use CORBA and the like: those frameworks optimize distributed object calls to avoid network hits, often being able to reduce the overhead to be equivalent to a virtual function call. Furthermore, networked applications have their own set of complexities and security considerations. You get to keep an open port somewhere, handle authentication (becase wherever there's an open port, there will be malicious connections), and extensive data validation (for the same reason). While these problems are applicable to dis
The idea is simply to define the "space" of legal inputs for each module and the correctness criterion for each input, and then generate random inputs based on the spec. This is far more effective than traditional hand-coded test data at both unit and system test levels, and as an added bonus the test spec doubles as a formal specification of the correct behavour that coders can actually work from. This is similar to the XP practice of "test-driven development".
Paul.
You are lost in a twisty maze of little standards, all different.
Not understanding something is one thing, but not understanding something so let's reject it as being "elite-ivory-tower" and "academic" is another. I've seen a lot of buggy C++ code being rewritten employing this style in obvious places - many defects were automatically addressed.
I disagree. Reality is reality. Algorithmic or Functional are just ways people look at it. Aren't "algorithmic" also abstract? Isn't "object-oriented" abstract as well?
By the way, using your vocabulary, I view the world as a mixture of "algorithmic" and "functional". No pure anything can describe the world, in my opinion.
Being functional or algorithmic has *NOTHING* to do with one being "more mathematical" and the other "less mathematical". I advise you, that your use of the common peoples' fear for mathematics in your arguments is not going to help.
Templates, being code generators, differ by nature to hand-tuned codes. So your code generates only 33 instructions vs the template's 88. Great - now tell me - which architecture? What compiler? What version of that compiler, and whose STL are you using, and which version of THAT?
And before you count the instructions, did you realize that this code is waiting for keyboard inputs, therefore what you're doing is unnecessary (and obviously premature) optimization?
How does the constant use of auto_ptr relates to the understanding (or the lack thereof) of object lifecycle? Sorry, but understanding object lifecycle the liberal use smart pointers are not mutually exclusive.
It is fashionable *among incompetent* Java developers to create objects left and right which make their programs memory hogs. It is also fastionable for *incompetent* C++ programs to forget deallocations leaking memories. What's your point? This mentality, immature or not, is not unique to managed languages.
Bulletproof code isn't cheap, but it can be done.
This is the most insightful comment I've seen so far. Particular tools can fix particular problems, but that's the easy part. The hard part is finding and noticing the problems, so that you know to look for (or make) the tools.
My teams have in-production bug rates well below one per developer-month. Here are the ten things I think are most important:
> Not understanding something is one thing, but not understanding something
> so let's reject it as being "elite-ivory-tower"
I did not say I did not understand it. I said I did not like it. I do not like it because it does not fit with the reality of computer operation, as discussed below.
> Reality is reality. Algorithmic or Functional are just ways people look at it.
On the contrary, you can see reality being algorithmic. Things happen one after another. To type "algorithmic", you depress a, l, g, etc. in order; you don't declare a set of letters, fill it with appropriate values and throw it at the computer. When you receive a specification for your program, it will say something like "get this from the user, then do this, then do that, then print out the result". No specification is ever written in functional notation outside the academic world.
More importantly, the computer itself works algorithmically. It does one thing, then another. No computer has ever worked functionally, and no computer ever will. All of them will decode and execute a sequence of instructions, and if you refuse to write your code likewise, you're only adding translation overhead.
Even in the hallowed halls of science overuse of the functional notation creates serious problems. The entire hodge-podge nonsense we call quantum mechanics stems from the attempt to describe a complicated system as a function. Instead of trying to get a set of time-value maps for the whole system, it would be more appropriate to look at the system's constituent parts and algorithmically simulate them through time. That way you wouldn't get any "spooky action at a distance", stuff being there and not there at the same time, and all other equally ridiculous denials of reality.
> I advise you, that your use of the common peoples' fear for mathematics
> in your arguments is not going to help.
I wasn't using that argument, but, now that you mention it, it is a reasonable one. Most programmers couldn't care less about higher mathematics, and, even if they were forced to study it in college, they likely have forgotten it all by now. Computer algorithms require minimal mathematical background. The most I ever used was a bit of calculus to write scan-conversion routines. So, whether from lack of practice, or from lack of interest, most programmers will prefer you didn't drag them into the world of useless mathematics. (and I use the word literally here)
> Templates, being code generators, differ by nature to hand-tuned codes.
> So your code generates only 33 instructions vs the template's 88. Great
> - now tell me - which architecture? What compiler?
That is quite irrelevant in this case. istream_iterator notation generates extra code for reasons that will not go away no matter how hard you try to optimize it. Yes, I might be able to write an istream_iterator that would have no overhead over my iterative version, but it will not be standard compliant. The istream iterator has to read on construction; it has to store the read value; it has to be constructed, since it must keep a reference to the source stream; it has to handle special cases, like the end-of-file, and the subsequent conversion to the end iterator value. However good you might be at optimization, you will not be able to discard these and still be compliant with the specification.
Also, which compiler or architecture you use will not make all that much difference in the size of the compiled code. I guarantee you that your functional copy will never generate smaller code than my iterative loop, no matter what compiler you use or what architecture you compiler for. There is a certain amount of work to be done, and my version does less work. It is as simple as that.
> And before you count the instructions, did you realize that this code is waiting
> for keyboard inputs, therefore what you're doing is unnecessary (and obviously
> premature) optimization?
First, you should note that I
You cannot write highly stable code in C++, due to design flaws in the language. For this reason, the FAA doesn't allow C++ for use in aircraft systems. You can improve the situation with the use of a garbage collector though, but if stability and safety is critical, then you should use ANSI C. See this: http://www.hpl.hp.com/personal/Hans_Boehm/gc/issue s.html
Oh well, what the hell...
> for_each(components.begin(), components.end(), _1.disable())
It is never that simple. The fact that you can't do what you've typed is one of the reasons I dislike it so much. What you really need is:
Things suddenly got uglier, didn't they? But wait, what if you need to call a function with an argument? Gotta use a bind2nd adaptor to wrap it, and then it becomes:
Wait 'till you try to explain to some maintaining programmer how to untangle that! Oh, and just for laughs, try to debug this thing. Put an assert in SetParameter, and you get a lovely callstack from gdb:
Now that's something to scare newbie programmers with! Oh, and forget about putting a breakpoint inside the loop; templated functions aren't targetable until executed.
> in some code I need to maintain then to encounter
> for(i = 0; i < components.count(); ++i) components[i].disable()
So why not just use an iterator loop? for_each does not have a monopoly on it:
(foreach is a macro I wrote because I use this construct so often)
> first form permits, for instance, components to be a linked list or even a hash.
> The second is implementation-dependent and if you change the underlying data
> structure, you'll have extra work to refactor.
If you use iterator loops, this wouldn't happen to you.
> I once worked, changing all instances of SomeObject* to auto_ptr
> eliminated altogether 35 bugs we had lurking in the BTS for a long, long time,
> with less than one day of work (strange, delayed, errors were suddently
> transformed in EARLY null-pointer dereferences
Why were you using SomeObject* in the first place? When I was advocating moderation in the use of auto_ptr, I wa