When Do You Kiss Backwards Compatibility Goodbye?

Arandir asks: "Backwards compatibility is great for users. But it sucks for developers. After a while your normally sensible and readable code becomes a nightmare spaghetti tangle of conditions, macros and multiple reinventions of the wheel. Eventually you have to kiss off backwards compatibility as more trouble than it's worth. The question is, when? Should my code conform to POSIX.1 or Single UNIX 2? Should I expect the user to have a ISO Standard C++ compiler? What about those users with SunOS-4.1.4, Slackware-3.2, and FreeBSD-2.2?" This question is really kind of difficult to answer in the general sense. The best advice one can give, of course, is "when you can get away with it". Not much help, that, but the lost of backwards compatibility, like most complex decisions, depends on a lot of factors. The key factor in most developers eyes, of course, is the bottom line. Have many of you been faced with this decision? What logic did you use to come to your decision and what suggestions do you have for others who might find themself in this predicament?

Comments

Ironically..

[Axe]

Ironically I am doing it right now. Good part it is Saturday, and other developers do not know. Or they will lynch me..

Break it all at once

[moebius_4d]

The two things I would say are, when you really reach the point where all the old crap is really clogging up the veins, fix it all at once. Make a clean break. Then people can at least keep in mind what is happening, what works with 2.x and what is still only for 1.x.

The other thing is, try to design to keep this from happening. Expose APIs that don't need to change much instead of the actual functions or objects that you use. One more level of indirection won't kill your performance in almost every case, but it will give you a whole lot more room to re-engineer when you decide you have to.

All that applies to the case where you control the interface and you need to change it. When you're publishing source code and want to decide what tools you can expect the user to have to make use of it, that's a marketing decision and not a technical one. You're talking about how many people will be excluded from your audience if you use GTK or assume a conformant C++ compiler. Technically the newer tools and libs are generally better, that's pretty clear. I think it's going to be a judgement call on the part of the developers as to how much they care about a lot of people being able to use their code. If they are willing to wait for the world to catch up before being able to use their program, then they can use the latest and greatest. If not, then they have to aim at a realistic profile.

Possible Compromise

[robbyjo]

I think there are several approach you may choose.

If you think your program/library has been widely adopted by many people, it would be very very hard to scrap the old one and start anew. You will provoke the wrath of other developers that use your program/library. If this is the case, then the possible compromise can be:

1. Overhaul the inner working of your API, but retain the signature (and the semantics) and the developers can think that it is just like the old one. This perhaps does a limited reform.
2. The second choice would be to extend the API. There are a number of choice you can do here. If your extension is like adding several parameters to a method/function, you'd probably want to add a macro to deal with the compatibility thing. Another way is to leave the old API that way and make a new function. (BTW, Windows does this). And then tell your user to use the new API instead.
3. If your API is not used by a lot of people, you'd probably want to revamp the whole thing. You'd have to tell people why and how to convert their program into your new convention. Note that in doing this, you should carefully design your API so that you won't do another revamp (users will hate you if you do so again).

To do revamping, you'd probably want to look on how the Windows COM approach. I'd hate to say this, but this approach is generally good, but I don't know whether I can come up with a better solution.

Or alternatively do an OOP approach. OOP can help modularizing your code if it is done properly. (That's why KDE rocks)

That's my 2c, though.

rules of thumb

[deranged unix nut]

Two rules of thumb:
1) Support whatever 90% of your users are using
2) Support the prior two versions

If you can't do the above, make a clean break and give it a new name or change the major version number and list the changes in the release notes.

If you have to make a clean break, if possible:
1) Provide a migration path
2) Provide an interop interface

And above all, listen to your users.

Re:Apple's plan

[darkonc]

Apple was very conscious of the idea of backwards compatability. More important than being able to run new apps on old boxes was the fact that most of the code that ran on a 128K 8Mz mac is still able to run on a 2GB 800Mz powerPC. Your investment in older software is not generally obsoleted by the new OS.

Often, Backwards compatability problems can be avoided by careful design. Leave room for improvements. Designate certain structures as ignorable. Presume that the current incarnation of the code is not the final version.

Design for elegence. If the current code is relatively clean, then chances are that it will be easier to tack on an addition later on. Include stubs for improvements that you can forsee adding later on -- even if you can't percive the exact form of the improvement at the time. When you tack on the addition, try and do that elegantly too.

With languages, you can sometimes avoid backwards compatability problems by not using the latest and greatest features just because they're there. (it also allows you to avoid creeping featurism growing pains).

If using a new feature makes a big difference in the implementation of a solution, then use it, but at least document it. It keeps you more conscious of the break, and makes life easier on the people who have to rip out your code and re-implement it on the older system that you thought nobody was using.

Anecdote: A friend of mine recently found out that that the security system where he had a storage locker was run on by apple IIc. The box was doing a fine job of what it was designed to do 20 years ago. Just because it's old, doesn't mean it won't work.

Microsoft's solution...

[Anonymous Coward]

I like Microsoft's solution, which I'm sure was done somewhere else first. At any rate, here's the situation...

Microsoft face the issue that they want to maintain backwards compatibility with everything, so they can leverage off the existing popularity of their platform. This means they can't let new libraries break old applications.

They had a problem once with MFC, where they updated its memory allocation scheme to make it more optimal. Problem was, a whole lot of old apps happened to work because they accessed memory incorrectly, but the old memory allocation scheme didn't reveal their bugs (I think they were doing buffer overruns, but the old scheme allocated a bit of extra room). Anyway, Microsoft released an updated MFC DLL, and suddenly old applications started breaking. It wasn't really MS' fault, but it was a big event, and I think it was the last time they touched old code like that!

Anyway, this is where they have their "DLL hell" problem too. Different apps are written against different versions of a DLL, many times with workarounds for known bugs in those DLLs. A new DLL comes along (installed by another application) and suddenly working applications start breaking.

So here comes COM. I've encountered it with DirectX, and it works like this. When you request something like a DirectDraw object or a DirectDrawSurface object, you also tell the class factory what *version* of the interface you want. Then you're provided with that actual implementation of the library. If you write your game against DirectX 2, but the user has DirectX 5, well your request to the DirectX DLL will actually give you the version 2 implementation! Which is cool, because if you worked around old bugs, those bugs are still there; they're not fixed for you! :)

As far as I know, you're not getting an "emulation" of the old implementation either; I'm pretty sure they just include binaries for each version of the interface. They could easily dynamically load whichever one you request.

Of course it means bloat, but there's no other solution. If you want backwards compatibility, you can't fake it, because it's all the really subtle things (like bugs that people worked around) that will bite you in the butt. It's been Microsoft's nemesis but its strength through the years, just as it has been Intel's too. Both companies need to remain backwards compatible with a *LOT* of stuff out there, and so they're forced to have all this legacy architecture. It would be nice to start from scratch, but then they'd be levelling the playing field, and that's no fun :)

Of course, this backwards compatibility drama affects everyone, not just Windows people. Just the other day someone installed a new PAM RPM on my Mandrake Linux. It installed fine, being a newer version, but some subtle change in behaviour meant I could no longer log into my own box. That's no fun either :)

- Brendan

Re:Microsoft's solution...

[doug363]

Much of Microsoft's problems don't just stem from multiple versions and fixing bugs (although I agree, these are significant), but badly thought out design in the first place.

Also, when there is an opportunity to make a clean break (i.e. the Win32 API), they only make half the changes they should or could. Have you seen the number of functions which end in "Ex" in the Win32 API? Ever noticed how the Win9x series is full of 16-bit user interface code? Or how only half of the NT kernel objects are supported under Win9x? In DirectX, it took between 4 and 8 major versions of DirectX to create something which was really worthwhile (depending on your opinion), and major changes weren't implemented when they should have been.

In regards to using COM, this helps versioning, but does not help the bad design problem. Even with full versioning, COM interfaces still have reserved fields in them, which is unnecessary if you're bundling all the previous versions.

Besides, not all compatibility problems can be solved by COM etc. Microsoft are getting better at providing slicker interfaces, but I don't feel that the underlying design is improving as it should be. For example, using Automation objects (which is a disgusting kludge for VB "programmers", to put it nicely) in Office apps is still a pain. (In particular, Outlook 2000's object model and MAPI implementation is inconsistent and buggy as hell.)

So yes, versioning does help alleviate backwards compatibility problems where they can't be avoided, but nothing is a substitute for good design.

A few thoughts

[Telek]

I know that we had to make the decision to redesign our flagship product, and it was a tough one (since this product was the core of about 5 other products in house as well). The problem was:

Code was ugly and hideous

Someone forked the tree a while back, and now we had to support 2 seperate source trees (this one was really annoying, because if fix a bug and change some behaviour in one side, since both sides had to be able to talk to each other, you'd have to introduce a corrosponding "fix" in the other side.

The core architecture was woefully outdated and ineficient

Speed was an issue, and the current architecture was limiting, and the code was optimized about as well as it could be (this was also part of the uglyness problem)

we were spending about 70% of our time fixing bugs in the old code, it took this much time because they were little and stupid and with the code in the state that it was in it took forever to trace things down.

Well, we had been wanting (desperately) to redesign from the ground up for a while, but the powers-that-be wouldn't give us the time, until one customer asked for a feature, marketing promised that they'd get it, and we said "Know what? We can't do that. Not with the current infrastructure." So the powers-that-be said "do what needs to be done!" and we said "yipee!".

Moral?

How much extra time is spent during debugging code that is due to the current state of the code?

How does the core architecture compare to what you will need in the future? Will it support everything?

How efficient is the old codebase, and how efficient does it need to be?

Can you get the time required to do so? This is a one-step-backwards before two-steps-ahead thing.

Do you trust the people that you are working with enough to be able to competantly and efficiently create the new architecture? This is a serious question, because I have worked with some people that are good if you tell them exactly what to do, but I wouldn't trust them to recreate everything.

Will you be required to keep the old codebase going while you are in the process of converting? If old customers need bug fixes, you might be forced to keep the old sourcebase around for a while.

Can you make the new design backwards compatible? If not, can you provide a wrapper of sorts, or some easy way to convert your customers from your old version to the new one?

If you are going to be redesigning the user interface or the API at all, then you must also think about the impact on your customers.

Just some food for thought.