Why Open Source Doesn't Interoperate

bergie writes "There is an interesting article on Advogato on why it is so difficult for Open Source projects to interoperate or support common standards. Often cultural differences between projects, egoes, and many other issues stand in the way. The article outlines some practical ways for improving the situation, based on experiences from OSCOM efforts to get support WebDAV, SlideML and other standards into Open Source CMSs. Examples of successful interop projects include freedesktop.org, the cooperative effort between GNOME and KDE."

Comments

Re:Open Source

[pe1rxq]

Actually ximian has done some really nice things with openoffice to improve interoperability....
Although they mostly focus on gnome it should also interoperate better with kde since with the freedesktop.org stuff things like cut and paste are becoming less of a problem...

XFree86 is fine for desktops, just don't expect it to be a top gaming environment yet.

Jeroen

One factor is obviously...

[Anonymous Coward]

...that open source authors prefer solutions they like over "standard" solutions.

Industry standards, particularly those created by committees, are often abominations that people only use if they have to. In my experience, the extent to which people like things like CORBA and XML often seems to be inversely proportional to their level of technical sophistication.

RFCs have much more respect in open source circles than committee-created standards.

Re:One factor is obviously...

[Dashmon]

...that open source authors prefer solutions they like over "standard" solutions.

To add to the reply on this, I think that this particular "problem" is in fact quite and advantage to OS development. Fabricated standards are often not as good as they can be, and are only revised after a long, long time. As far as I know, W3C isn't that quick on updating their standards, for example. An OS developer that implements a solution "he likes" does this for a reason, and can show that there's a problem with an old standard, and a solution. Evolution of standards is pushed that, far quicker than the committees could, simply because those committess don't know what problems there are with a standard when developing an application x, simply because they don't write it - they don't write a whole lot, I believe.

Re:One factor is obviously...

[datadictator]

Programmers work in the real world, standards committees work from inside the time-capsules where they have been carefully protected from all outside influences these past 7 thousand years.

Programmers improve upon standards.
Open Source guys no less than the EULA-writers. The big difference is:
When an open-source guy extends a standard, the changes he makes becomes available for all to copy and interact with.
Even if he never writes so much as a man page for the change, you always have the source. Besides most times OSS guys extend a standard, the retain compatibility with the orriginal.

When that company that was named to describe it's founders private parts does it, nobody outside it's secret vault will ever know how they changed it or why - at least not without a long painfull road of reverse engineering.

Hence, when open-source guys extend standards, they are innovating, giving the technology new uses etc.
When closed source guys do it, they (USUALLY I know there are exceptions and more power to them) are merely trying to decomoditize the standard for the sake of market control.

Re:Because free software is not planned

[akadruid]

Unfortunatly a lot of code writing is done with inadequate planning, but this is an inherant problem. Without a crystal ball you cannot predict every twist and turn you will face.
This is not just a problem with free, Free or open source software, but also with planned, structured development of commerical model software such as Windows.
After all if we could predict the future why would ever need new versions? :)
However I don't think that a neatly integrated environment is impossible, just difficult.
Besides, getting around integration issues is part of the fun!

Re:OpenSource creates de-facto standards

[akadruid]

Yes, but VNC is a much more niche market though. And it is a lot simpler standard.
All of the other projects are based directly on a historical product - VNC, whereas with these projects they can be developed alongside or before other projects, and have much wider scope.

And there's a good reson why you can't change your posts - can you imagine the havoc when someone decides to change their +5 Interesting post to read 'First Post!'.

Blindered developers

[wowbagger]

One of the biggest problems I see with Free Software development is the problem of the blindered developer.

This is the guy who doesn't bother to raise his head from the computer to look at how his project works in any environment other than *his* system. You know, the guy who requires you to have libfoo.so.5.1.2.pl6-thursday-0741am-fred-mutant1 installed just to compile his code, and by $deity no other version of the library will work.

A concrete example: The developer of the GATOS project (a driver for the TV tuner/video capture (but not video out) functions of ATI All-in-Wonder cards) requires you to use HIS kernel module and HIS radeon driver. As a result, you may EITHER use his code XOR the DRI accelerated 3d code, but not both.

True, he does (to an extent) track the DRI development, but rather than working with DRI and XFree and coming up with a way his drivers can play nice with the standard builds (e.g. having hooks in the standard driver and having the standard driver load his modules if present) he is off on his own little branch.

He also uses libraries and packages that are not part of the standard installs of common distros - as a result just getting his code working is a real slog. So many people don't do it, and his project does not get as much support as it might.

Now, I am not picking on him - developing stuff like that is hard, since it is very poorly documented. And with DRI making changes, XFree making changes, and him making changes, you WILL have times when things don't play well together. But rather than that being a transient state of affairs it is the normal state the GATOS project w.r.t. DRI.

Unfortunately, it take time and work to stop, get a fresh install of RedHat/SUSE/Gentoo/... and see what it takes to get your code to build and install. It takes work to make sure that you really NEED the latest version of libfoo, rather than just any version. Especially when your code interoperates tightly with other people's projects it is difficult to plan interfaces that won't change frequently. If you can accept help from others this isn't so bad, but many project "leaders" have the attitude of "HOW DARE YOU IMPUNE MY PROJECT! IT IS PERFECT UNTO ITSELF! I CANNOT HELP IT IF YOU ARE NOT 31337 ENOUGH TO HAVE THE LATEST STUFF! L@M3Rz! IT IS UNDER DEVELOPMENT!"

But that is the difference between a hack and a software engineer - just "getting something to work" and "getting something to work well, under as many circumstances as possible, as smoothly as possible."

Because it's hard?

[IamTheRealMike]

I'm trying to create a packaging metadata spec at the moment to allow standardised LSB RPMs to be installed on all compliant distros using native dep resolvers etc - in short, making a standard that the relevant people can work to and are interested in is fantastically hard work. It's not just a case of writing something down and say "Agree to it!", everything has to be discussed, specified and of course there is always a slight worry in that you might be heading in the wrong direction etc.

Lack of apparent interest from vendors is also somewhat discouraging. There are quite a few specs up on freedesktop.org that are only implemented by GNOME, with KDE "pending". Then for instance Mosfet comes along and claims the thumbnail spec is stupid for reasons x, y and z and proceeds to invent his own (the so-called "professional" thumbnail spec) and ask for it to replace the existing one! Not exactly encouraging.

Re:Because free software is not planned

[You're All Wrong]

Your subject line says it all. Well, nearly...

I'd say the two biggest sins of the open-sourcers are
a) over-generalisation (it'll be able to do everything)
and
b) over-specialisation (it does one task, but can't do similar ones)

I'm finding it hard to think of examples, but I guess GNU grep's an OK example of something that's just about right.
Expanded to do enough things like context greps (e.g. give me 4 lines before the line containing "Name:" and 1 line after), and a few other features (e.g. '-c' so that you don't need to '|wc-l') that add to its functionality, so it isn't over-specialised. Likewise, it's not sed, awk or perl, they realised that just keeping it simple and lightweight was the way to maximise its usefulness.

YAW.

Interoperation adds overhead and complexity

[budGibson]

Interoperation means passing data between two different programs over some common bridge. This means typically writing some sort of connector code. In the best case, someone is able to bundle that connector cod into a library.

Consider coding to a web service interface such as SOAP vs. just keeping your application within one programming language and using its built-in constructs for passing data. With the web service, you either have to marshal into and out of SOAP envelopes on your own or use a library. However, not all libraries themselves interoperate. Hence, you have to test for compatibility by running against a suite of other implementations, all of which are also supposedly standard. It's the browser wars all over again. If you don't bother with interconnectivity, the job is over more quickly.

To get an interoperability standard that you can just code to seems to take the developer community years of effort. Yes, there is value to interoperability, and that is why people do actually undertake things like web services projects and spend years trying to develop standards. But for a first project, or even a mature, successfully functioning product, interoperability is not likely to be a first instinct.

Re:Interoperation adds overhead and complexity

[snatchitup]

I'm definitely hearing you on Web Services.

It's a great idea, that may get over-used IMHO.

We're starting to do it to tie existing applications together to be used in-house. Why? They're all J2EE based, or moving in that direction. In fact, a while ago, the push was to get other languages to be able to talk directly to EJB's. Since then, we've added yet another layer, hence, I'm still employed.

Closed source software

[Simon Lyngshede]

You might as well ask why doesn't closed source software interoperate.

In a perfect world everyone would write software that works together. One shouldn't only look at getting open source software to support a common standard, we should try getting everyone to support common standards.

It's not my expirience that getting open source software is all that hard. Getting non open source stuff to work with anything is hard. Closed standards is want gets in the way of interoperability. With open source you always have the option of adding the features you need. If two open source applications can benefit from working together, it will be implemented by someone who needs it. The original authors might not always see the need for a given feature.

I don't see it as a problem in the open source community. It's fare worse is commercial closed source software.

Re:Because free software is not planned

[g4dget]

To make software interoperate, developers need to create interfaces before writing their software.

Interoperability relies on standards that come from standards bodies. As long as the software conforms to those standards and works correctly, it interoperates. Interoperability with standards requires less planning on the part of the programmer because the standards have already been written by other people.

Often no planning is done and developers start writing their code without a clear vision of what they want to write.

Good. More software projects should start off with that kind of openness and willingness to adapt to changing circumstances.

Even Linus is against the creation of standard interfaces internal to the Linux kernel. That decision inhibits the creation of a truly modular system.

That part I fully agree with. But it has nothing much to do with "interoperability" because it is an issue within a single project.

Also, Linus's stance is unreasonable: it is quite possible to define interfaces and yet to maintain the same flexibility that exists in Linux. The problems of the Linux kernel with modularity are cultural problems; technically and managerially, they are solvable if people wanted to solve them.

The reason is simpler than some are making it out

[Overt Coward]

I've seen some posts that tap-dance around the real reason that OSS in particular doesn't interoperate well, but they all seem to miss the mark slightly. It's not really about ego, or the NIH syndome, or laziness, or poor design (if any).

The real reason, I believe, has to do with the fundemantal drive behind an open-source project -- find an itch, then scratch it. OSS projects (in general) start because someone sees a specific need or want for software that performs a specific purpose. By its very nature, that project is looking at the world in a bottom-up fashion -- and that inevitably pushes interoperability off until "later".

Integration or interoperability is typicaly an "add-on" to an already successful project -- no one really starts thinking about "well, I'd love to be able to do X from program Y" until both projects X and Y have developed strong user bases. It's sort of a natural selection in software -- the "best" projects survive and eventually breed (interoperate) with other projects to evolve higher-order software.

Of course, that's just my opinion. I've been known to be wrong... though of course, those who discover that have been known to disappear...

Someone explain bash

[swordgeek]

I've read a lot of comments here, with some interesting points, excuses, or disagreements on the premise that OSS doesn't support standards.

Bash supposedly conforms to Posix standards if you invoke it with the --posix flag. (Why it shouldn't default to posix-compliant I don't know) However, bash is not compatible with /bin/sh. WHY???!!!!

Will someone tell me why bash is the ONLY /bin/sh-like shell that I can't `echo "hi!"` in????

In case anyone things I'm just ranting (I probably am ranting, but that's not all there is here :-), consider my question closely. sh, ksh, zsh, ash, and every other shell that uses sh syntax (i.e. not the csh variants) deals with the above statement in the same way. Bash doesn't.

Why would OSS deliberately develop a shell (the default universal Linux shell no less) that breaks such a fundamental and long-standing de facto standard?