When Do You Kiss Backwards Compatibility Goodbye?

Arandir asks: "Backwards compatibility is great for users. But it sucks for developers. After a while your normally sensible and readable code becomes a nightmare spaghetti tangle of conditions, macros and multiple reinventions of the wheel. Eventually you have to kiss off backwards compatibility as more trouble than it's worth. The question is, when? Should my code conform to POSIX.1 or Single UNIX 2? Should I expect the user to have a ISO Standard C++ compiler? What about those users with SunOS-4.1.4, Slackware-3.2, and FreeBSD-2.2?" This question is really kind of difficult to answer in the general sense. The best advice one can give, of course, is "when you can get away with it". Not much help, that, but the lost of backwards compatibility, like most complex decisions, depends on a lot of factors. The key factor in most developers eyes, of course, is the bottom line. Have many of you been faced with this decision? What logic did you use to come to your decision and what suggestions do you have for others who might find themself in this predicament?

Comments

What are you asking us for?

[el_mex]

You should know the best who your users are.

You should know who they are, what equipment they have, who is making who a favor (ie: who has to adapt to whom), and specially you should know what they want (such as how much backward compatibility).

OO design

[Uncle_Chachi]

A properly designed OO system should alleviate all those backward's compatibility issues. And yes, in spite of all the /bots who complain about it, Java sure solves a lot of those OS/hardware compatibility problems...

Re:OO design

[Anonymous Coward]

"A properly designed OO system should alleviate all those backward's compatibility issues."

Hardly, since these issues have *nothing* to do with methodology as such. A well-designed structured system will be more amenable to being backwards compatible than a poor OO design.

If the design of the external interfaces (file formats, network protocols, etc) allows for future features, specifies 0/blanks for reserved fields, etc, then it's much easier to be backwards-compatible. This has nothing to do with OO.

And if you're talking Java or C++, each of those languages has ancestral versions which make more recent versions have interoperability fits (old vs new GUI event model, older name mangling/namespaces).

Re:OO design

[PrismaticBooger]

It takes a lot more than OO design to solve this problem. The fact is that initial designs are almost always naive, and lack the specific flexibilities (and inflexibilities) the Ideal Solution would require. So, very often, are second iterations. But the third time's usually the charm.

The need to depart from backward compatibility is the result of correcting design flaws. And design flaws happen, regardless of your programming methodology. Design flaws are less a product of programming methodology, and more a product of not completely knowing the problem.

Java accommodates design flaws a little bit better not by being object-oriented, but by relying so heavily on dynamic binding. While this can make recovering from Bad Design simpler, it only lets you create bandaids that do nothing to fix the original design problems. And all that dynamic binding costs Java in performance across the board.

Re:OO design

[robbyjo]

Java accommodates design flaws a little bit better not by being object-oriented, but by relying so heavily on dynamic binding.

FYI, dynamic binding is a crucial element in OO programming. Programmers wouldn't have been able to downcast if you don't have this. It is indeed that this creates a lot of slowdown, but researcher has done flow analysis research to eliminate many of this.

Concerning the design: Yes, you're right. Nobody starts with a "perfect" design. We will extend it, twist it and twirl it until it comes into a huge mess when we have to scrap the whole thing and rethink it.

But, design flaws can be minimized. It doesn't depend on programming methodologies, but rather to a design methodologies -- which is pretty much debated by now. Using models (e.g. UML models) to depict a huge project maybe worthwhile because we can get a quick overview of what we are upto. Most of the times we can locate the design mistakes pretty quickly.

Design flaws can also be minimized by documenting specification (which is pretty much a "meta-programming" approach). Sadly, nobody wants to do this. If you can get this done, there are a lot of tools (albeit still in research) that can automatically check your program whether it conforms to the spec or not.

Re:OO design

[Anonymous Brave Guy]

A properly designed OO system should alleviate all those backward's compatibility issues.

A properly designed and maintained OO system will alleviate some of the backward compatibility issues. The thing is, most OO systems are not well maintained, even if they were reaonsably well designed originally. Most developers, in my experience, are far too reluctant to refactor properly.

This generally boils down to one repeated mistake. When changed requirements suggest a change in the responsibilities of your classes and/or the relationships between them, too many developers try to change the implementation or interface of classes in isolation instead.

Unfortunately, in the long run, no amount of interface hackery can make up for an inappropriate distribution of responsibilities. It's just fudging the design, and the one thing you don't want to do with an OO design is mess it up with fudges.

The reason you get to this "it's got no chance, let's start from scratch" syndrome is that too many fudges were allowed to accumulate. Instead of refactoring as they go along, keeping a design with clear responsibilities and relationships, programmers try to shove things in where they don't really belong. Once or twice, you can get away with it, but it doesn't scale, and ultimately leads to an unmaintainable mess.

Of course, the problem is that adopting the "I'll just fix it here" mentality is, in the immediate timeframe, quick and cheap. In the long run, it's much more expensive -- a rewrite from the top will do horrible things to your budget -- but people rarely consider that far ahead.

Such is the curse of the OO paradigm, where the benefits are only to be had by those who think: most people don't.

Re:OO design

[Thatman311]

Yes it may lead to some bloat...but face it. Any backwards compatibility *could* be considered bloat because what you have is code in the binary that isn't being used when the most up-to-date user of the binary is using it.

In saying that...one of the main purposes of something like COM is to provide backwards compatibility. Like other people say...when you have millions who use your stuff you really don't know what they do with it. So you really don't know what reordering the order of those function calls will do to some older user of your code. Maybe it will cause a race condition and break them. Who knows....that is why maintaining backwards compability on projects that have millions of users is a huge undertakening.

Compatibility is crucial

[The Ultimate Badass]

Even if you are writing a minuscule app for your own use, that could not conceivably have any use for anyone else, you should always adhere to the following rules:

• Use autoconf and automake. I don't care if there's nothing to configure. Do it.
• Try and place as much code into libraries as possible. Modularization is good. Use libtool religiously
• Try and test your code on as many unixes as possible. For this reason, I have every NSD, and linux kernel major on my machine. It is unthinkable to assume that if someone who wants to use your program on a platform you didn't test for would be willing to fix your code and send you the patches. Open Source just doesn't work that way
• Forget the above rules and use java.

Re:Compatibility is crucial

[Anonymous Coward]

"... Forget the above rules and use java."

1.0, 1.1 or 1.2? IBM or Sun or Blackdown? AWT or Swing?

Re:Compatibility is crucial

[Skapare]

So where does one get a box full of all these unixes? I do agree with you on that part about broad testing. But I can't afford to buy all those boxes (especially not an IBM zSeries). And finding shell accounts (especially root ones) for various testing seems to not go beyond Linux, FreeBSD, and a handful of OpenBSD and Solaris. Even the IBM mainframe accounts are available to only a few people, and then for a limited time (have you ever known an open source project to take 3 months and stop development then because it's "done"?). I do have Linux, FreeBSD, and OpenBSD on Intel, and Linux, OpenBSD, and Solaris on Sparc. What else would you suggest?

As for Java ... I'm waiting until environments are built that can do what I do now in C. I'm hoping gcj will let me do at least some of these things in Java. But there are some things I doubt it can ever do. You can prove me wrong by rewriting LILO and init into Java.

Break it all at once

[moebius_4d]

The two things I would say are, when you really reach the point where all the old crap is really clogging up the veins, fix it all at once. Make a clean break. Then people can at least keep in mind what is happening, what works with 2.x and what is still only for 1.x.

The other thing is, try to design to keep this from happening. Expose APIs that don't need to change much instead of the actual functions or objects that you use. One more level of indirection won't kill your performance in almost every case, but it will give you a whole lot more room to re-engineer when you decide you have to.

All that applies to the case where you control the interface and you need to change it. When you're publishing source code and want to decide what tools you can expect the user to have to make use of it, that's a marketing decision and not a technical one. You're talking about how many people will be excluded from your audience if you use GTK or assume a conformant C++ compiler. Technically the newer tools and libs are generally better, that's pretty clear. I think it's going to be a judgement call on the part of the developers as to how much they care about a lot of people being able to use their code. If they are willing to wait for the world to catch up before being able to use their program, then they can use the latest and greatest. If not, then they have to aim at a realistic profile.

MS went overboard

[MSBob]

Microsoft is the most notable supporter of backwards compatibility. That's a large part of the reason why Win32 is such a bad API. There is just too much junk there that accumulated over they years that seemingly won't ever go away. The other problem backwards compatibility creates is code bloat. Because old interfaces and often implementations have to be kept in the binary the libraries grow bigger and bigger purely to keep the old junk around. Microsoft COM is the most guility party in that. The rule that interfaces can never, ever change results in nosense like DirectX 8 still supporting all interfaces of DirectX2. Fortunately this trend seems to have reversed and everyone is now keen on starting afresh. MS is rolling out their .NET framework. Apple released their OSX which has new native APIs. It's probably for the best to the consumer in the long run as well.

Luckily open source doesn't have ot suffer from the issue as much since source availability ensures that old software can often be tweaked or sometimes just recompiled to make it work with new versions of dependent libraries.

How long to maintain backwards compatibility is really the question of your business domain. An in house app can probably be changed significantly without impacting many people while a widget library (like QT for example) must maintain backwards compatibility for at least a couple of minor versions. The ability to simply recompile old code after a major change in the library is a welcome feature too.

Re:MS went overboard

[SurfsUp]

So you're claiming Microsoft is making a mistake by providing backwards compatibility for DOS, but "open source," whose primary operating systems are Linux and FreeBSD, both designed to be backwards-compatible with 1970s UNIX systems, are somehow not?

Well, speaking as an experienced Dos programmer, the Dos interface is really a piece of crap. Nothing is defined with any kind of generality or foresight. Look at the "critical error handler" interface for example, it's unbelievably awkward and unworkable. Look at what happens if you shell to Dos, start a tsr, then exit the shell. Boom. This is broken by design. Just try to write a program that recovers control when a child aborts, that's when you find out how messed up the resource handling is, and the exit handling. The child tries to exit back to Dos instead of its parent unless you hack into all sorts of undocumented places, which everybody eventually has to do for a big system. All that undocumented-but-essential internal interface dung has to be supported, right?

That said, no I don't think Microsoft made a mistake in supporting old Dos code, it's all part of maintaining the barrier to entry, the more messed up it is, the more expensive to replicate. Never mind that the whole sorry tale isn't good for anyone but Microsoft.

Well, now that we have a several real operating systems as alternatives, there's no longer any advantage to Microsoft in keeping Dos on life support. When Dos gets painful, people just walk now. But Microsoft still has to support it or everybody will start publishing articles about how Windows doesn't work, and they have to keep all those "enterprise" solutions that consist of Dos programs and BAT files running. Now Dos support is just an expensive liability Microsoft can't escape. Heh, but the rest of us can, and do.

The original Unix interface on the other hand, was quite well designed. 30 years later, for the most part the whole thing is intact and still in use, functioning reliably in enterprise-level multiprogramming environments. So, no, maintaining compatibility with that old-but-good interface is unquestionably not a mistake.

The short story is: the whole Dos interface was a mistake, and Microsoft now has to live with it. The unix interface was not a mistake and we're happy to carry it forward.

Re:Open Source to the rescue!

[Graymalkin]

Did that make sense to you before you pressed the submit button? It sure seems pretty silly now. You're suggesting updating an old version to new standards. That is what the entire concept of versioning is based around! You cap old versions and start anew for the entire purpose of keeping your applications up to date.

rules of thumb

[deranged unix nut]

Two rules of thumb:
1) Support whatever 90% of your users are using
2) Support the prior two versions

If you can't do the above, make a clean break and give it a new name or change the major version number and list the changes in the release notes.

If you have to make a clean break, if possible:
1) Provide a migration path
2) Provide an interop interface

And above all, listen to your users.

Re:rules of thumb

[Ronin Developer]

If your software is targeted for Linux users, then ask yourself what are the marketing stats of IE on the Linux platform? Or, are you targeting specifically Windows or Macintosh users (in which case IE is the predominent browser).

If you are targeting the world in general and want to make money or want the widest possible dissemination of your code, then I'd reluctandly say "Yes", give IE preferential consideration over the more obscure browsers. Or, stay conformant to the HTML/XML specs and support everybody that's conformant but be willing to accept the price of losing specific capabilities.

If you have code that works with Netscape and not IE or vice versa, then why not contract (or develop yourself) for a plugin or activeX control that will provide that capability? Or, bundle an existing control with your code?

But, that is getting away from the "backwards compatibility" issue. The previous poster mentioned the rules of thumb, of which I wholeheartedly agree. If your code has to change so much that you break backwards compatibility or produce difficult to maintain code, then release it as a new product or new major version number. And, support the last to versions of your product. Makes good sense to me.

Re:Apple's plan

[darkonc]

Apple was very conscious of the idea of backwards compatability. More important than being able to run new apps on old boxes was the fact that most of the code that ran on a 128K 8Mz mac is still able to run on a 2GB 800Mz powerPC. Your investment in older software is not generally obsoleted by the new OS.

Often, Backwards compatability problems can be avoided by careful design. Leave room for improvements. Designate certain structures as ignorable. Presume that the current incarnation of the code is not the final version.

Design for elegence. If the current code is relatively clean, then chances are that it will be easier to tack on an addition later on. Include stubs for improvements that you can forsee adding later on -- even if you can't percive the exact form of the improvement at the time. When you tack on the addition, try and do that elegantly too.

With languages, you can sometimes avoid backwards compatability problems by not using the latest and greatest features just because they're there. (it also allows you to avoid creeping featurism growing pains).

If using a new feature makes a big difference in the implementation of a solution, then use it, but at least document it. It keeps you more conscious of the break, and makes life easier on the people who have to rip out your code and re-implement it on the older system that you thought nobody was using.

Anecdote: A friend of mine recently found out that that the security system where he had a storage locker was run on by apple IIc. The box was doing a fine job of what it was designed to do 20 years ago. Just because it's old, doesn't mean it won't work.

Major version breakage

[rjh]

Commit yourself to a strict policy: nothing in a minor version will break anything since the last major version. If your code is at 1.9.99, it should be backwards-compat with 1.0.0. If your code is at 1.1.0 and backwards compatability breaks, move it to a 2.0 release.

Typically, users expect breakage--or, at the very least, problems related to upgrades--with major versions. With minor versions, they don't expect breakage.

Follow the Law of Least Surprise. If you break backwards compatability, up the major by one.

Insofar as when to break backwards compatability, that's a much harder question. The obvious answer is a little philosophic: not all engineering problems can be solved by saying ``screw backwards compatability'', and some engineering problems cannot be solved without saying it.

The trick is learning which is which.

Re:Our very means of communication...

[Anonymous Coward]

But honestly, where do we set that limit?

I set it at 1% of browsers using the site. Currently Netscape is about 8%, divided half over windows and half over unixes (see data). I support Netscape 4 not only because of legacy, but because I want to support unixes, and until recently the distributions came with Netscape 4 as their best browser.

Compatibility in formats and protocols

[Skapare]

A lot of the discussion seems to be related to issues of things like programming languages and operating systems (which are important). But what about keeping up with old formats and protocols? I think the issue is more one of what your project works with, than it is what language you choose (including the OS as part of the former).

• Should a new web server support older versions of HTTP to the extent that they are not proper subsets of new versions?
• Should my mail server support POP3, or can I make it do IMAP4 only?
• Should my web site require every browser have cookies, Javascript, and Java enabled, or should it at least function with browsers that doesn't have these, or don't have them enabled, or have them filtered at the firewall?
• Should I support ISO-8859 character sets, or can I just stick to Unicode (UTF-8 and maybe also UTF-16) to be universal?

I'm not so much looking for specific answers to the above questions, but rather, a general idea of how you think one should go about deciding those issues to come up with the best answers in some given situation.

Re:Ironically..

[am 2k]

when Microsoft abuse the term "innovation", when IP-nazis abuse the term "piracy"

and when Slashdot-posters abuse the term "nazi"...

Backward compatability with what?

[Anonymous Brave Guy]

This is a confused discussion. A lot of people are mixing up "backward compatability for users" with "not making significant changes to the code base". The two are largely unrelated, except that screwing up the latter will also mess up the former.

What your user sees is, and always must be, decided by your requirements spec, not programmer whim. The only people who can get away with doing otherwise are those developing for their own interest (hobbyists, people involved in open source development, etc).

To put it bluntly, blanket statements like "meet 90% of your users' BC needs" are garbage. In many markets (notably the bespoke application development market) if you drop 10% of your users in the brown stuff, your contract is over, and your reputation may be damaged beyond repair. Look at MS; years later and in the face of much better libraries, MFC still survives, because people are still using it (including MS) and they daren't break it.

This is far removed from rewriting things significantly as far as the code goes, which is where things like the standards mentioned come into things. I'm sorry, Cliff, but you don't do this "when you can get away with it" if you're any good.

Every time you rewrite any major piece of code, "just to tidy things up", you run the risk of introducing bugs. You need to be pretty sure that your rewrite is

• necessary to meet your requirements, or
• fixing more than it breaks and not breaking anything unacceptable

or, preferably, both.

If the rewrite is justified using these objective criteria, then you do it. When you do, you try to minimise the number of changes you make, and to keep the overall design clean. You retest everything that might conceivably have been broken, and you look very carefully at anything that didn't work -- it's quite possible that the people who originally wrote this code months or years ago made assumptions they forgot to document and you've broken them. Finally, if and only if your rewrite is performing acceptably and all the tests are done, you decide to keep it. If not, you throw it away and start rewriting again.

And for the record, yes, I spent most of last week rewriting a major section of our application, as a result of a code review with another team member. We kept the overall design, tweaked a few things within it, and rewrote most of the implementation. Now we need to retest it all, update all the docs, etc. This little exercise has cost our company thousands of pounds, but in this particular case it's justified by a needed performance increase and the significant reduction in bug count. But you can bet we thought very hard about it before we touched the keyboard.

Design intent is a cornerstone

[crath]

A key to providing backward compatibility is "design intent"; i.e., closely examine the backwards compatibility issue when you are first thinking about creating a piece of software. Internal data structures, external file formats, APIs, etc. are all influenced by the design constraints placed upon a project. If one of those constraints is backward compatibility then these structures will all be built differently than in the case where no backward compatibility is ever required.

FrameMaker is a great example of an application that appears to have been architected from day one to provide backward compatibility: every version of FrameMaker imports and exports Maker Interchane Files (.MIF files) and so it is trivial to move files between releases of the application. While I'm sure this causes the developers some headaches from time to time, I know from personal experience that a constant anchor point like .MIF files influences coding decisions.

Having done work on an ASCII interchange mechanism for a multiplatform application, I can be fairly certain that the FrameMaker decision isn't very difficult to implement: each release of the application has a pair of small functions, one to walk the internal data structure and emit the ASCII interchange format, and another that parses the ASCII interchange file and produces an internal data structure.

When we designed our application, the ASCII interchange functionality was deemed important; this influenced the internal data structures, which in turn influenced the binary data files. If we had tried to bolt backward compatibility on at a later date (i.e., in version 2.0) it could have been a lot of work; whereas, building it in from day one didn't cause any extra work.

Conscious design intent is the key to making backward compatibility a non-issue.

Answers.

[mindstrm]

Your web server should support all versions of HTTP. (You meant HTML?)

POP3 and IMAP4 are not 'new versions' of each other... neither is outdated. one is not a replacement for the other. Needs dictated solely by users.

Your web site should require no more funcionality than needed ot operate the way you want it to. That's just good programming. Don't use cookies or javascript or java if you don't need to.

You can stick to Unicode, because ISO-8859 maps into it properly.

There are better examples

[Anonymous Coward]

Microsoft is the most notable supporter of backwards compatibility.

Not really, IBM on the mainframes and Intel on the 80X86 are the biggest backward compatibility stories around (over 20 years for Intel over 30 for IBM I would guess).

Re:Ironically..

[invenustus]

and when Slashdot-posters abuse the term "nazi"...

Tom Tomorrow takes apart Nazi analogies. This cartoon expresses my feelings on the subject better than I ever could.

Re:And Linux got it right the first time?

[spauldo]

Linux is a different story. Being as it's not one chunk of software like windows, and with most of the libraries being cross-platform, it makes it both a blessing and a curse for this.

But all that aside, yes linux did get it right the first time, being that they went with POSIX. UNIX has been around long enough the general API's are pretty stable. There's been some bumps - glibc comes to mind, since libc5 didn't support internationalization to the extent required, and the a.out to ELF conversion, but these things happen to all software. The vast majority of software needs little to no change between linux versions.

Sun's done a real good job with this from what I can tell, since the same app running on solaris 2.3 on a sun4m machine can be run on an E10000 with solaris 8 without a recompile (although changing it would certainly help performance in many cases). They've had to deal with a lot of things, like the conversion from 32bit to 64bit, and have handled it pretty well.

Now as for software written for libraries like GTK and whatnot, well, you knew what you were getting into when you wrote it. If you want something whose API doesn't change, program for xlib or motif (thanks to lesstif and the free motif clause, just about everyone can run motif apps).

Many of us who were linux users from way back don't neccessarily want a nice, stable platform that never changes, never pushes new ground. If that takes a bit of compatability breaking, then so be it - the old libraries are still there if you want to use them. I remember having both libc5 and glibc on the same system, and I remember when you could only get certain key apps for libc5 even after most people changed to glibc (netscape, for instance). We try to limit such changes to the major version numbers, and the standard UNIX way of handling this makes it no problem to have multiple major versions of the same library on the system. It's our answer to the dll hell problem, and it works rather well.