Slashdot Mirror
Linux Applications And "glibc Hell"?
cybrthng asks: "How do you stop glibc hell on Linux? I thought I'd long lost the ever familiar DLL hell on Windows, but with Linux it breaks the applications so bad its not funny. Will Linux only be able to truely survive with applications available in source form? For instance take Oracle Applications, it is nearly impossible to install it on RedHat 7.0 or any glibc 2.2 based distro since the applications were built against 2.1.x. When you install this software it tries to relink itself with the correct libraries and fails miserably. You can however force it to use glibc-compat, but that isn't a solution for a production system. Do vendors have to recompile there applications for every kernel, every library and every distro? How can I use Linux when the core libraries don't seem to be forwards or backwards compatible across different distributions?"
Ok, I don't think you understand at all what DLL Hell is on Windows. Your assumptions as flawed at any rate.
:-)
The problem with DLL Hell under Windows has nothing to do with a lack of versioning. DLLs do have versioning, and Microsoft has tried to always be careful in leaving old interfaces compatible so as not to break upward compatibility. In cases where they clearly have to break compatibility they do the same thing as Unix... they create new files with new names.
VSVBVM60.DLL was a replacement for VSVBVM50.DLL which left the 5.0 functionality intact. However there have been like half a dozen versions of VSVBVM60.DLL released which fix bugs internally without breaking external interfaces.
With COM you don't even care about the filename since all interactions with the component are through the ClassID. You can create a new file, with a new name that registers the same ClassIDs as the old component and redirects them internally as appropriate.
There are differences between COM and classic DLLs and I'm going to talk about classic DLLs primarily because that's where the DLL Hell problem has occured primarily.
Now one problem with classic DLLs is that only one copy of a DLL is ever loaded into memory, and this has created many problems. This isn't completely true under Windows 2000 which has additional new features I'll mention later.
Where DLL essentially comes into play is from REALLY STUPID BRAINDEAD INSTALL PROGRAMS! Not to mention Microsoft's lack of attention to this problem existing and unwillingness to do anything about it, even if it was just education.
As an example, let's say you have version 4.1 of generic.DLL installed on your machine by application X.
Application X works fine.
Now you install application Y. This install program installs a copy of generic.DLL version 4.0.
Now application Y's install program is broken, it either does two stupid things, it installs this old copy of generic.DLL over the top of the newer copy already in windows\system. Or it places it in the application directory.
In the latter case, if you run app X before app Y everything works fine. But if you run app Y, when the system searches for the DLL it finds one in the app path first and loads that. All subsequent requests for this shared DLL are passed the pointer to the one already in memory.
In the former case, app X is never going to work if it relies on functionality which existed only in the newer copy of the DLL that it had installed. app Y broke it permanently by stupidly overwriting a newer DLL.
Similarly with COM DLLs one could overwrite a newer version of a control with an older version which breaks assumed functionality.
The solution is actually really quite obvious in 95% of all cases. Most of these DLL problems result from shared DLLs as part of the OS or distributed with the MS Development tools. The solution therefore is to only allow Microsoft to distribute and update these via Service Packs. This is the path they have finally gone down in Win2k.
That is to say these MS shared system DLLs should never be deployed with an application. If your app needs a certain version then you should specify that you need Service Pack 4 or whatever.
We'll see if this happens with say Office XP.
You then only have an issue with shared third party controls. Now this can be solved by using Win2k's ability to localize DLLs to the application and thus load multiple different versions.
Anyway, that's basically a definition of the problem.
BTW, as to your signature line... Linux/BSD is not my main server/workstation and I'm pretty certain I'm far less ignorant than yourself on topics regarding Windows as a result.
I really don't see the point. What Hell is happening on glibc 2.2? As far as I see it is the first glibc that smoothly installs over older versions without cramming the whole system. And not only.
One development/testing system is working here since July 2000. It suffered more than 30 glibc upgrades, ranging from late 2.1 version, running through a whole series of pre-2.2 and right now working on 2.2.1.
During these upgrades, apps suffered some serious crashes during two-three pre-2.2 versions. Not more. Some applications, based on older 2.1 and even 2.0, have kept working until now. For example, Netscape and Quake2. Besides, I didn't note serious problems with 2.1-based apps.
Due to the purpose of this machine, I managed to see how most of these apps are rebuilt up to 2.2 glibc. Here, some incongruences did appear but I cannot say they are a "Hell". Most cases are the result of a few differences in variables. This can be a serious hassle for an average user but it does not hamper his use of a Linux box just by upgrading to 2.2.
Most of the packages I used came from Mandrake Cooker project.
First of all, a free system is not aimed primarily at making binary aplications work, but at making free aplications, which comes with source, work.
Of course binary compatibility is nuice - it means you, or your software vendor, doesn't have to recompile everything now and then. But it comes at a high price - unexpandability. You can not add a field to a datastructure, since that makes the struct bigger, and breaks compatibility. In source, adding a field is never a problem, and compatibility amounts to preserving old fields that someone might expect, and put values that they won't dislike, into these fields.
Of course, you can do uggly tricks like a hash-table of the extra fields for all objects of one type, that you index with the pointer to the original object. This is for example supported in glib. But it's terribly uggly, and is to beg for problems (like mem. management problems).
I agree however that glibc have had some problems - it hasn't allways been 100% source-compatible...
And - try to search for 100% binary compatibility between say Windows 95 and Windows NT 4.0. Have fuN!
--The knowledge that you are an idiot, is what distinguishes you from one.
You are 100% correct. People that have the money to run Oracle (and we are talking about LOTS of money here) go to Oracle and find out what it will run on and go with that. Oracle says RH6.2, then RH6.2 it is. You feel you must be using RH7? Great. Put RH7 on another machine and go play there.
The people who came up with glibc-compat did so because they anticipated the difficulties associated with upgrading the systems as a whole to newer libraries. There's nothing wrong with installing the older libraries and they're not something that should be avoided for "production" systems.
To be sure, it would be nice if Oracle got with the program and updated their tools to run on a more recent glibc, but until that happens, you have an alternative to sitting around, scratching your head, and saying "Gee, it doesn't work." That makes more sense than the "you should statically link all major applications" crud that others have posted.
I hate to say this, and hope this isn't considered flamebait, but Linux definitely needs more quality control with PRODUCTION releases of libraries and making sure that if it worked in glibc 2.1, it HAD BETTER work in glibc2.2 too.
If that is impossible, because the library behaved in a buggy (or nonstandard or unsupported or inefficient) way, and fixing it would break something that depended on that feature, that is different than just having something no longer work.
However, old apps need to work with new libraries. Here is a solution. Add a libc call called expect_version(). Your program calls it with the version it expects. The library will behave in a way compatible with that version. If that call is not made, have the library behave at a default level of compatibility . Have binary only software be built to issue this call with the version of all the libraries it is designed to run with.
Just because it CAN be done, doesn't mean it should!
Comment removed based on user account deletion
Static linking is one solution, but it seems a little bit heavy handed. Disk space is one problem, but indeed not major one. Another problem is that the library cannot be shared. This means that two programs using the same library will have to load it in their memory space. This means more memory consumption and more loading time.
Another nice solution could be something like bundles under Mac OS X/Darwin. First the library system knows the version number of each library, and can load the one the application needs - this alone would solve the problem described here. Secondly the library can be installed inside the application's framework, so you have the benefit of static linking without having to build a monolithic program.
This means that you can solve such problems easily. Need a specific library? Move it into the bundle. Can use the normal library? Move it out of the bundle. Simple. The DLL-hell problem comes, IMHO from the rather simplistic dynamic libray handling codesystem.
To have an idea about bundles, have a look at the article in on Ars Technica.
Oracle was originally built for specific operating systems, and in the non Windows arena, specific versions of UNIX. It's not at all surprising that you'd need to run a specific version of Linux from a particular vender in order to use it. Sad but true fact. It really can't be helped at this point so focus on running your organization, not resisting some obvious limitations of the current architecture. (Oracle doesn't work on Debian or Slackware either - my shop tried, and as much as we hated doing it, we were forced to run it on RedHat.)
On another issue... Some people say, "companies should static link libraries to their programs!" Well, this is only taking a bad situation and making it worse. If this is done, binary only releases of software will suffer with flaws in existing versions of whatever system libs they're linked against. Then you have to wait for said company to release a new version whenever the bugs in a system library are fixed. Eventually, we'll manage to do what Windows does, and that is have readily backwards compatable libs that actually work properly.
For now, conform and produce working results.
...or rather, it is only relevant for C++ libraries, the C ABI has been stable for a long time.
So has the glibc ABI actually, except that it is not 100% bug compatible. I.e., applications that relies on bugs in the library in order to work, may break when new the library is updated.
GLibC doesn't do this. Everything's crammed in. And that is bound to make for problems.
IMHO, what GLibC needs to be is a skeleton library with a well-defined API but no innards. The innards would be in seperate, self-contained libraries, elsewhere on the system.
This would mean that upgrading the innards should not impact any application, because the application just sees the exoskeleton, and the API would be well-defined. The addition of new innard components would then -extend- the API, but all existing code would be guaranteed to still work, without relinking against some compat library.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Read the whole message that you responded to and he explains the problem *and* the fix that is possible on Unix type systems. I can have five apps of various vintages that each require a different set of libraries that they linked against.
/usr/$sysname-glibc20-linux/lib/ld-linux.so.2 \ --library-path /usr/$sysname-glibc20-linux/lib \ $netscape $defs $cl_opt "$@" ( real world example )
Like so:
exec
With names like:
libORBit.a
libORBit.la*
libORBit.so@
libORBit.so.0@
libORBit.so.0.5.1*
libORBit.so.0.5.6*
We can keep different versions of files for use, not like the different versions of mfc42.dll that all have the same name. If another version of a library is completely backwards compatible, then a simple symbolic link gives the complete name that the run-time linker is looking for.
Kindness is the language which the deaf can hear and the blind can see. - Mark Twain
http://sources.redhat.com/glibc/glibc-faq.html
tell your vendor to link it static (using .a libraries instead of .so).
also remind them that a "Linux" version is
meaningless, they should say "Linux/x86" or
"Linux/Alpha" or whatever.
I hate it when a vendor supplies a "Linux" version
that won't work on my hardware, and I can't tell until *after* I've downloaded it.
Getting tired of getting a copy of Oracle for Solaris 2.3, iPlanet for SunOS 4.1.3 and Veritas for Solaris 7 and finding that none of them support my Solaris 8 system. Dammit, what is Sun doing wrong!?
;-) for those who did not guess....
You'd think that you would actually have to pick an OS revision based on the least-common denominator of the supported platforms for your application needs!
Someone needs to go write a Python-based OS and then never change anything. That'll solve it.
...but nobody reads the documentation anymore, so they bring problems on themselves.
Using private interfaces, using static libraries, not using versioning even when shared libraries are in use... I'm not surprised Oracle had problems.
You cannot apply a technological solution to a sociological problem. (Edwards' Law)
For instance take Oracle Applications, it is nearly impossible to install it on RedHat 7.0 or any glibc 2.2 based distro since the applications were built against 2.1.x. When you install this software it tries to relink itself with the correct libraries and fails miserably.
If there are substantial glibc 2.1-> 2.2 problems it is really poor coding on the part of the vendors. The use of private (but available) glibc functions was made impossible in the changeover.
There are a few models that will work in this case. First, the older version of glibc can be included with Oracle, and set LD_LIBRARY_PATH or LD_PRELOAD to load those libraries first. Then there is no problem.
Talk to your vendor. Ultimately, if you want to pay to use their software, they have a responsibility to ensure you can use it with some ease.
When using real software like Oracle under linux, you find out what the requirements are for the application you're going to run, and install a compatible setup. You don't just run out to the ftp site, burn a copy of the latest distro of Mandrake, and expect every application you install onto the new system to work flawlessly. Maybe you can run something like apache on every machine everywhere, but Big Important things like Oracle generally have pretty specific system requirements, even under other unicies.
Java, for example, couples libraries and user code much less tightly, yet uses statically type checked interfaces. Java's type checking is actually unnecessarily strict: classes are considered incompatible on dynamic linking even though only some aspects of their implementation changed. ML implementations could easily do the same thing.
Also, the fact that languages like C++ and Java tie inheritance hierarchies to static type checking is an unnecessary and idiosyncratic restriction. You can have perfectly statically type-safe systems that do not have these kinds of inheritance constraints: as long as the compiler and/or linker determines that the aspects of the interfaces you are relying on are type-compatible, it can make the two ends fit together safely, no matter what other changes or additions have happened to the classes. The "signature" extension for GNU C++ did this at compile time, and something similar could be done by the dynamic linker when libraries are loaded.
The efficiency issue is not significant. Even for a completely dynamic object system like Objective-C, a good runtime will have hardly more overhead for a dynamic method call than a regular function call. Any of the systems based on static type checking I mentioned above would do even better. And Java, of course, can actually do better than C/C++ when it comes to libraries because the Java runtime can (and does) inline library code as native code at load/execution time.
Of course, sometimes, things just have to change incompatibly. But as far as I can tell, almost none of the changes in glibc (or most other C/C++ libraries I use regularly) should affect any user code. Almost any kind of library interface would be less problematic than what exists right now.
So, I agree: statically typed languages will not go away. But "DLL hell" is avoidable whether you use statically or dynamically typed languages. In fact, as I mentioned, you could even make it go away in C/C++ by introducing a special library calling convention that has a bit more information available at load time. However, why beat a dead horse?
I work for a company building a network montioring system available for FreeBSD, NT (and 2K), and both RedHat and Debian Linux. We're adding platforms as people request them.
:)
Really, RedHat 7.0 includes the libraries that shipped with 6.2, so while we only support RedHat 6.2 we still work out-of-the-box on RedHat 7.0. Why not use the compatibility libraries ? That's what they're there for - they're not performing worse or anything, they are just older versions of the library.
On UNIX-like systems you actually have VERSIONING on your system libraries. So you can have a perfectly running system with ten different versions of the C library, and each application will use the version it requires.
You're welcome to check out our beta-versions available from sysorb.com, if you don't believe me
First off, If you get anything out of this post it is this: DO NOT RUN A REDHAT X.0 RELEASE IF YOU DON'T UNDERSTAND LIBRARY VERSIONING
NO SUCH THING AS "GLIBC HELL"
There is _no_such_thing_ as "glibc Hell". UNIX (including Linux) has versioning on libraries -- right down tot he filename. _Unlike_ Windows, you can have _multiple_library_versions_ installed. Even Microsoft still has NOT addressed this (and I run into it daily) simply by versioning filenames of libraries. This is a _farce_ and the result of people not understanding the OS in front of them.
UNIX v. WINDOWS ON SYSTEM LIBRARIES
So, in a nutshell, you can have your library issues two ways:
- Recompile for new libraries (if you have source)
Otherwise, no option if you don't have source -- total SOL!Especially on Win2K which makes some libraries untouchable (and quite incompatible with a lot of existing software).
Microsoft does this for "stability", but it is a library- ignorant way of NOT addressing the _real_ issue, lib versioning.
LIBRARY VERSIONING AND SYMBOLIC LINKS
The main reason Windows cannot have UNIX-like library versioning and versioning on filenames is because it lacks symbolic links. With symbolic links, you can have multiple subrevision of a library, with one subrevision the "default" revision, with that (or another) the default "main" version. E.g.,:
libmy.so -> libmy.so.1
libmy.so.1 -> libmy.so.1.1
libmy.so.1.0 -> libmy.so.1.0.7
libmy.so.1.0.7
libmy.so.1.1 -> libmy.so.1.1.3
libmy.so.1.1.2
libmy.so.1.1.3
libmy.so.3 -> libmy.so.3.0
libmy.so.3.0 -> libmy.so.3.0.1
libmy.so.3.0.1
In the preceding example, there are actually only 4 library versions: 1.0.7, 1.1.2, 1.1.3 and 3.0.1. We could easily introduce more versions if programs required them. Most libraries are "parent revisioned" (I don't know what the "official term" is, but that's what I'll call it), so the latest "x.y.z" "version.revision.subrevision" is synlinked as "x.y" "version.revision" as well as "x" "version". As far as compatility between versions, anything goes (and is a per-library consideration), but the "general rule" is as follows:
Most OSS projects, including GLibC, have good versioning schemes that change subrevisions (the "Z" in x.y.z when updates, bugfixes, or non-structural changes are made -- meaning 1.1.2 and 1.1.3 are most likely header/function compatible. So, depending on the library, most programs are fine when linking against X.Y instead of x.y.z-- and do so to keep from requiring the user to have numerous differnet libraries installed. A simple x.y symlink to the latest x.y.z (latest being the max(Z)) is usually all it takes. Again, a "parent revision" symlink does the job.
Now different revisions (the "Y" in x.y.z) usually involve some header/function changes that _may_ be INcompatible. As such vendors usually do not link against just the major version (the "X" in x.y.z) for that reason. E.g., some programs work fine on any GLibC 2.y.z system (BTW, GLibC 2 is aka LibC 6), but most are tied to GLibC 2.0.z (RedHat 5.x), GLibC 2.1.z (RedHat 6.x) or GLibC 2.2.z (RedHat 7.x). Major version changes (again, the "X" in x.y.z) are left for radical, completely incompatible changes -- like LibC 4 (RedHat 3.x), LibC 5 (RedHat 4.x) and GLibC 2 (RedHat 5.x+ -- which caused a bigger stir than 7.0 awhile back ;-PPP).
DON'T RUN A REDHAT X.0 RELEASE UNLESS YOU KNOW WHAT YOU ARE DOING!
98% of the "bitching and moaning" about RedHat 7.0 comes from user naivity on library versioning. Yes, I _do_agree_ that RedHat did release 7.0 too earlier with unfinished components, but since patching GLibC and GCC through December, RedHat 7.0 is a _solid_ release. Never, never adopt a RedHat X.0 release unless you are a seasoned Linux user! Please get the word out on that (although the RedHat 7.0 README on CD 1 *DOES* stress that point too!!!)
[ Side note: The kernel is another matter though -- but understand RedHat cannot "time" the release of GLibC, GCC and the kernel since they are all independent development teams. ]
RedHat gives you _full_warning_ of the all-important GLibC change in a new release. It's always been RedHat's model -- introduce a new, and possibly INcompatible GLibC on a X.0 release. All revisions in a major release have the same GLibC and GCC, and are quite interchangable. I've said it before and I'll say it again, only RedHat seems to do this (although I haven't checked out Debian yet). So I know that going from 6.2 to 7.0 means issues, just like 4.2 to 5.0 did for me almost a half-decade ago. NOTHING NEW!
So, again, if you do NOT know what you are doing, stick with RedHat X.2 releases (or at least X.1)!!! At X.0 releases, most of the Linux world has NOT yet adopted the new GLibC versions -- hence the wealth of binary library incompatibilities. So if you are not familiar with how to deal with them, do NOT try to deal with a RedHat X.0 release!
REDHAT MAINTAINS LIBRARY COMPATIBILITY
Today's RedHat 7.0 (GLibC 2.2) release comes with full RedHat 6.x (GLibC 2.1) compat libs, even a compat devel libs and a compat compiler/linker. And you can also install compat libs from the RedHat 6.2 release for 4.x (LibC 5) and 5.x (GLibC 2.0) compatibility.
[ Side note: I *DO* have a "complaint" with RedHat for not including LibC5 and GLibC2.0 compatibility libraries with RedHat 7.0. And I've let Alan Cox know about them. They should _at_least_ be available on the Powertools CD. I know LibC4 shouldn't be included for security reasons, but LibC5 and, especially, GLibC2.0 should be! ]
ANY LIBRARY ISSUES ARE ALMOST ALWAYS THE PACKAGERS FAULT!
Now that we have the user out of the way, as long as a vendor/packager dynamically links against the specific library version, the binary will use that version. Too many vendors are used to just linking against what they have, especially if it is an older library on their development system. So when a user has a new version of the library, possible with function name, parameter and other structrual changes, core dumps will occur.
If a developer is really worried, there is always the option of statically linking -- i.e. putting the library in the binary itself (so no external library references/dependencies). Of course, there could be licensing issues with GPL or other OSS code to/from commerical and vice-versa. If people think this is limited to Linux, they are _gravely_mistaken_ as most commercial IDE and development tools introduce their own issues. Under Windows, where only one system library can be installed (and that library may be "fixed" in Win2K), you're in for a world of hurt trying to sort them out.
END-USER LIBRARY VERSION ADMINISTRATION
As previously discussed, symlinks are often used to "parent revision" a full "x.y.z" "version.revision.subrevision". Most of the time, RPMS and tarballs/makes installs do this for you. But sometimes, you'll have to administer and create them yourself. Again, doing this is usually easy (it's just a symlink -- "ln -s ") and the first thing to try when a program cannot find a library (e.g., libmy-2.0) where a library exists with a more complete version (e.g., libmy-2.0.1).
-- Bryan "TheBS" Smith
-- Bryan "TheBS" Smith
Independent Author, Consultant and Trainer
You have some interesting viewpoints, but I think you're avoiding the question rather than dealing with it.
In the programming language research community, the feeling is that dynamic languages are very good for things like scripting and prototyping, but are not as good an idea for large software systems.
The problem is twofold - first, as you mention, dynamic languages always get a performance hit. But the second reason - which you miss - might be more important - fewer errors can be detected at compile time... they would only turn up at runtime, or worse, end up as hard to detect bugs. Moreover, the runtime may fail in someplace other than where the error occured. For example, let's say you have a bunch of "polygon" objects in a linked list, and you mistakenly put a "circle" object in that list as well. Much later, you're traversing the list and expect to find a polygons, but instead you find a circle. Type error! But the real error was where you put the circle in the linked list. In a dynamically typed language, you'd have to look to see where the circle was inserted - and the bigger the software system, the harder that becomes. However, in a statically typed language, the compiler tells you right away "hey buddy, you're putting a circle in to polygon list. Fix that, or you don't get object code!".
I don't think that statically typed languages are going to go away. As it often is with issues with software development, the real problem is psychological rather than technological. If backwards compatibility across ".x" releases was a priority for the glibc team, perhaps we wouldn't have this problem. As it is, they are probably more driven to adding new features or fixing really bad old problems in ways which break compatibility... if there are people willing to work on the project who have different goals perhaps it may be time to fork libc again?
However, they offer clear conditions. First, they don't guarentee upwards-compatibility, that is code compiled against glibc 2.2 working with 2.1. Second, C++ is currently off limits (which will change with gcc 3.0). Third, it applies only to shared versions of the library. Fourth, private internal interfaces are off limits.
The Oracle problem is simple: they're using static libraries (ie, ar archives of object files). This doesn't work because symbol versioning (the magic that enables compatibility in shared libraries) isn't implemented for object files. HJ Lu has a page on this issue and possible resolutions.
90% of other compatibility problems result from using private interfaces. This happened to Star Office a while back.
The evaluation of an action as 'practical' . . . depends on what it is that one wishes to practice.
Are there alternatives? Plenty, actually:
You could probably invent a new calling convention for C together with some changes to the dynamic linker that would address this problem for C libraries. While you are at it, you should probably also define a new ABI for C++, something that avoids "vtbl hell" using an approach to method invocation similar to Objective-C. These new calling conventions would be optional, so that you can pick one or the other, depending on whether you are calling within the same module or between different modules. Perhaps that's worth it given how much C/C++ code is out there, but it sure would be a lot of work to try and retrofit those languages. Why not use one of the dozens of languages that fix not just this problem but others as well?
A related approach is to still write a lot of stuff in C/C++ but wrap it up in a dynamic language and handle most of the library interactions through that. That was the dream of Tcl/Tk (too bad that the language itself had some limitations).
Altogether, I think the time to fix this in C/C++ has passed, and COM-like approaches didn't work out. My recommendation would be: write your code in a language that suffers less from these problems, Python and Java are my preference, and add C/C++ code to those when needed for efficiency or system calls.