When Do You Kiss Backwards Compatibility Goodbye?

Arandir asks: "Backwards compatibility is great for users. But it sucks for developers. After a while your normally sensible and readable code becomes a nightmare spaghetti tangle of conditions, macros and multiple reinventions of the wheel. Eventually you have to kiss off backwards compatibility as more trouble than it's worth. The question is, when? Should my code conform to POSIX.1 or Single UNIX 2? Should I expect the user to have a ISO Standard C++ compiler? What about those users with SunOS-4.1.4, Slackware-3.2, and FreeBSD-2.2?" This question is really kind of difficult to answer in the general sense. The best advice one can give, of course, is "when you can get away with it". Not much help, that, but the lost of backwards compatibility, like most complex decisions, depends on a lot of factors. The key factor in most developers eyes, of course, is the bottom line. Have many of you been faced with this decision? What logic did you use to come to your decision and what suggestions do you have for others who might find themself in this predicament?

Comments

Open Source to the rescue!

[stox]

At least if you are trying to support an Open Source solution, you have a chance of going back and fixing the old application to be conformant with new API's, etc. If you're running an old binary, you have no choice but to provide the old API's.

Hail Open Source, Death to binary only!

Possible Compromise

[robbyjo]

I think there are several approach you may choose.

If you think your program/library has been widely adopted by many people, it would be very very hard to scrap the old one and start anew. You will provoke the wrath of other developers that use your program/library. If this is the case, then the possible compromise can be:

1. Overhaul the inner working of your API, but retain the signature (and the semantics) and the developers can think that it is just like the old one. This perhaps does a limited reform.
2. The second choice would be to extend the API. There are a number of choice you can do here. If your extension is like adding several parameters to a method/function, you'd probably want to add a macro to deal with the compatibility thing. Another way is to leave the old API that way and make a new function. (BTW, Windows does this). And then tell your user to use the new API instead.
3. If your API is not used by a lot of people, you'd probably want to revamp the whole thing. You'd have to tell people why and how to convert their program into your new convention. Note that in doing this, you should carefully design your API so that you won't do another revamp (users will hate you if you do so again).

To do revamping, you'd probably want to look on how the Windows COM approach. I'd hate to say this, but this approach is generally good, but I don't know whether I can come up with a better solution.

Or alternatively do an OOP approach. OOP can help modularizing your code if it is done properly. (That's why KDE rocks)

That's my 2c, though.

Microsoft's solution...

[Anonymous Coward]

I like Microsoft's solution, which I'm sure was done somewhere else first. At any rate, here's the situation...

Microsoft face the issue that they want to maintain backwards compatibility with everything, so they can leverage off the existing popularity of their platform. This means they can't let new libraries break old applications.

They had a problem once with MFC, where they updated its memory allocation scheme to make it more optimal. Problem was, a whole lot of old apps happened to work because they accessed memory incorrectly, but the old memory allocation scheme didn't reveal their bugs (I think they were doing buffer overruns, but the old scheme allocated a bit of extra room). Anyway, Microsoft released an updated MFC DLL, and suddenly old applications started breaking. It wasn't really MS' fault, but it was a big event, and I think it was the last time they touched old code like that!

Anyway, this is where they have their "DLL hell" problem too. Different apps are written against different versions of a DLL, many times with workarounds for known bugs in those DLLs. A new DLL comes along (installed by another application) and suddenly working applications start breaking.

So here comes COM. I've encountered it with DirectX, and it works like this. When you request something like a DirectDraw object or a DirectDrawSurface object, you also tell the class factory what *version* of the interface you want. Then you're provided with that actual implementation of the library. If you write your game against DirectX 2, but the user has DirectX 5, well your request to the DirectX DLL will actually give you the version 2 implementation! Which is cool, because if you worked around old bugs, those bugs are still there; they're not fixed for you! :)

As far as I know, you're not getting an "emulation" of the old implementation either; I'm pretty sure they just include binaries for each version of the interface. They could easily dynamically load whichever one you request.

Of course it means bloat, but there's no other solution. If you want backwards compatibility, you can't fake it, because it's all the really subtle things (like bugs that people worked around) that will bite you in the butt. It's been Microsoft's nemesis but its strength through the years, just as it has been Intel's too. Both companies need to remain backwards compatible with a *LOT* of stuff out there, and so they're forced to have all this legacy architecture. It would be nice to start from scratch, but then they'd be levelling the playing field, and that's no fun :)

Of course, this backwards compatibility drama affects everyone, not just Windows people. Just the other day someone installed a new PAM RPM on my Mandrake Linux. It installed fine, being a newer version, but some subtle change in behaviour meant I could no longer log into my own box. That's no fun either :)

- Brendan

Re:Microsoft's solution...

[doug363]

Much of Microsoft's problems don't just stem from multiple versions and fixing bugs (although I agree, these are significant), but badly thought out design in the first place.

Also, when there is an opportunity to make a clean break (i.e. the Win32 API), they only make half the changes they should or could. Have you seen the number of functions which end in "Ex" in the Win32 API? Ever noticed how the Win9x series is full of 16-bit user interface code? Or how only half of the NT kernel objects are supported under Win9x? In DirectX, it took between 4 and 8 major versions of DirectX to create something which was really worthwhile (depending on your opinion), and major changes weren't implemented when they should have been.

In regards to using COM, this helps versioning, but does not help the bad design problem. Even with full versioning, COM interfaces still have reserved fields in them, which is unnecessary if you're bundling all the previous versions.

Besides, not all compatibility problems can be solved by COM etc. Microsoft are getting better at providing slicker interfaces, but I don't feel that the underlying design is improving as it should be. For example, using Automation objects (which is a disgusting kludge for VB "programmers", to put it nicely) in Office apps is still a pain. (In particular, Outlook 2000's object model and MAPI implementation is inconsistent and buggy as hell.)

So yes, versioning does help alleviate backwards compatibility problems where they can't be avoided, but nothing is a substitute for good design.

A few thoughts

[Telek]

I know that we had to make the decision to redesign our flagship product, and it was a tough one (since this product was the core of about 5 other products in house as well). The problem was:

Code was ugly and hideous

Someone forked the tree a while back, and now we had to support 2 seperate source trees (this one was really annoying, because if fix a bug and change some behaviour in one side, since both sides had to be able to talk to each other, you'd have to introduce a corrosponding "fix" in the other side.

The core architecture was woefully outdated and ineficient

Speed was an issue, and the current architecture was limiting, and the code was optimized about as well as it could be (this was also part of the uglyness problem)

we were spending about 70% of our time fixing bugs in the old code, it took this much time because they were little and stupid and with the code in the state that it was in it took forever to trace things down.

Well, we had been wanting (desperately) to redesign from the ground up for a while, but the powers-that-be wouldn't give us the time, until one customer asked for a feature, marketing promised that they'd get it, and we said "Know what? We can't do that. Not with the current infrastructure." So the powers-that-be said "do what needs to be done!" and we said "yipee!".

Moral?

How much extra time is spent during debugging code that is due to the current state of the code?

How does the core architecture compare to what you will need in the future? Will it support everything?

How efficient is the old codebase, and how efficient does it need to be?

Can you get the time required to do so? This is a one-step-backwards before two-steps-ahead thing.

Do you trust the people that you are working with enough to be able to competantly and efficiently create the new architecture? This is a serious question, because I have worked with some people that are good if you tell them exactly what to do, but I wouldn't trust them to recreate everything.

Will you be required to keep the old codebase going while you are in the process of converting? If old customers need bug fixes, you might be forced to keep the old sourcebase around for a while.

Can you make the new design backwards compatible? If not, can you provide a wrapper of sorts, or some easy way to convert your customers from your old version to the new one?

If you are going to be redesigning the user interface or the API at all, then you must also think about the impact on your customers.

Just some food for thought.

Re:The best time...

[deranged unix nut]

Why does every topic discussion have to turn into a anti-(insert favorite company to hate) bash?
[this really should be off-topic, but this is the type of user complaint that you will get if you don't carefully coordinate with your users before breaking old functionality]

1) Software developers writing software for XP can write it so that it is compatible with 9x/me or even Win3.1 if they want.

2) The topic question was backwards compatibility. Very few applications (if any) are 100% forward compatible. If you save your Word 97/2k/xp doc in a Word 6.0 compatible format, Word 6.0 can read it just fine. Where is the logic in expecting Word 6 to read the native file format that contains new features in Word 2k?

3) How does it become Microsoft's problem when 3rd party application programmers don't design their software to run on multiple versions of the OS? (It is possible, in fact a large percentage of the win9x software runs on w2k without being *designed* for w2k.)

Think about it for a second, how do you make the application that you released last year so that it will be 100% compatible with the file format and the new features that you will dream up next year??? Even if the file format was compatible (which would probably make it a kludge), there is no way that the functionality would be compatible.

And, umm, you can always save in the previous version's file format. :)

How is this any different from the new perl script to configure IPchains that won't work on kernels prior to IPchains? Or any other added feature for that matter...this is a flaw in your expectation, not evil marketing.

Do you expect your Netscape 1.0 browser to be able to handle CSS??

Migration Tools

[ImaLamer]

Lets say you are developing an app that works with a certain set of data and it is stored one way.

With your new release (to 2.0, or whatever) that is a whole new work - you bundle tools to convert the old data.

MS does this with office, and people sometimes think its a pain, but I do not. If I'm getting a new version that (sometimes) works better, I don't mind running it through a filter - but make sure you don't make the mistake MS has made. Make sure this filter works.

If you're talking a new OS or other such projects - create data (read: settings) BACKUP tools. If you want the people to all migrate to this new OS, or large system application, give them a tool to backup those old settings and put the code right into you're new version to grab those old settings from the backup.

Thats all we want!

Re:OO design

[jon_c]

OO design is a pretty vague term, as it means different things to different people. However one concept that is proven to work for backwards compatibility is component design as found in COM and other component based frameworks.

For those who are not familiar with the concept you don't deal with objects per say, instead you deal with interfaces to implementations of objects.

For example

ICat *pCat = NULL;
pCat = MakeCat(IID_ICAT);
pCat->Meow();
delete pCat

COM does it a little differently but the basics are there. You request an implementation to an interface, not the object itself. The way this works for different versions is that instead of IID_ICAT you can have IID_ICAT2 and ICat2 interfaces without having to break your old ICat clients. The implementation could even share much of the old code.

For example:

ICat2 *pCat = NULL;
pCat = MakeCat(IID_ICAT2);
pCat->MeowLoudly();
delete pCat

Admittedly it's not the most elegant design, but it works in the sense that you're not breaking old clients and still have room to support new interfaces.

-Jon

Sometimes stay off the cutting edge

[vanzilar1]

Sometimes the answer is not to target the newest implementation but the one that is used on all platforms in use. For Java this means sometimes going back to the 1.0 API! You lose a lot
of cool functions but it is the only way. Then just make sure it works best on the newest platforms or browsers depending if the program runs on an OS or a web browser. This isn't always an easy thing to find out. We made a web application using Java 1.1 and it turns out that Netscape on the Mac supported it without any of the changes from 1.0 to 1.1 so it didn't work there. But recently I've heard that Netscape may finally be supporting 1.1 for real on the mac and 1.3 with OS X. Cool!!! If only we wrote the program in Java 1.0 there would be no problems (except that it would probably be really clunky).