Debian Upgrade May Cause Serious Breakage

daria42 writes "Debian developer Bill Allombert has e-mailed the Debian community saying he estimates about 30% of users upgrading from Debian Woody to Sarge will suffer 'serious breakage'. Allombert says the upgrade process suffers from a number of bugs reported before the release went live several days ago. Chief among the problems, he said, were cyclic dependencies and the fact that software installation tool apt depended heavily on the changing C++ libraries. Allombert wants developers to test the upgrade cycle continuously during development and not just during the freeze period just before release."

Comments

Evidence of problems with packaging systems

[AKAImBatman]

Chief among the problems, he said, were cyclic dependencies and the fact that software installation tool apt depended heavily on the changing C++ libraries.

Let this be a lesson to those of you who claimed that "APT is unbreakable." There's no such thing as an unbreakable technology. There is however, such a thing as a robust technology that resists failure. As packaging systems go, APT is fairly good. However, my belief is that packaging systems are inherently flawed.

What you want in an OS, is a method for determining the precise core upon which you can base your applications on. Such a core would effectively be an immutable set of system APIs that cannot be changed. The upshot to this situation is that the given system is verifyable. i.e. I can have a script go through and ensure that everything that should exist does exist. From that information, I can then do a delta to find out what exists that shouldn't exist.

This is in direct opposition to a packaging system that builds an OS out of inter-dependent components. The problem with such a strategy is that using inter-dependent components only works if you're building from scratch. As anyone who has managed a version control system can tell you, things get extremely complicated (and tend to require manual intervention) as soon as files start branching. The same thing happens in packaging systems as soon as you start doing upgrades to individual components. Soon you find yourself with a mess of mismatched dependencies which require constant manual intervention to solve. Not a good situation.

In the case of a defined core, you can simply wipe out the old core and replace it with the new one. As long as testing has been done to ensure that the new components are still backward compatible with old software, everything should work fine after the upgrade.

Food for thought, anyway. To the Debian team: Thanks for the new release! Even if there are some growing pains, it's still nice to see you back in the game. :-)

Re:Evidence of problems with packaging systems

[Tharkban]

Give it a rest.

The Linux Standard Base is dead.

There is too much freedom for even the distributions to make cores effectively. Debian doesn't develop the software, they package it. They have no direct control over compatibility issues between versions in their software. This makes their job a whole lot harder than in commercial OS's where one entity controls both the core software and the packaging.

They also don't have the resources to making security patches for every package without upgrading to a newer version of said package (i.e. backporting). They really do a phenominal job given their constraints.

Re:Evidence of problems with packaging systems

[listen]

You again ;-)

Take a look at this Conary system. It has some interesting ideas that could certainly help in this kind of situation : especially transactions for upgrades. If a bit fails, the whole upgrade rolls back, and you can even rollback completed transactions.

I like this idea better than choosing some arbitrary core of code to upgrade as a massive lump, and statically linking hundreds of copies of anything not in the core into the separate apps. As to your verifiability detecting script, I see no reason this can not be done for a packaging system. And before you go on about corrupt databases, please remind yourself what a filesystem is: thats right, a corruptable database.

I will agree with you on compatibility: people should stop breaking ABI. I'm looking at you, Freetype...

Re:Evidence of problems with packaging systems

[AKAImBatman]

They also don't have the resources to making security patches for every package without upgrading to a newer version of said package (i.e. backporting). They really do a phenominal job given their constraints.

I agree wholeheartedly. I'm not attempting to "diss" the Debian distro or its maintainers. I'm only pointing out that the packaging system is beginning to strain under the pressure of so many packages. The complexity of the package system is quickly becoming too difficult to maintain. Especially since the packaging system mixes the core system APIS with the user applications. (Always a recipe for trouble.) Thus it is time to start thinking about something new.

The Linux Standard Base is dead.

The LSB was always about the "least common denominator" and not about "the most usable configuration". For what it was, it wasn't too bad. But a real standard at this point would have to define a lot more libraries, although perhaps at more of a library version level than trying to force the individual APIs.

With that in mind, I don't think that such a standard should be attempted across all distros. For one, that would limit their ability to be different and provide new competitive services. For another, it tends to be better to allow a few different standards to compete before you attempt to pick one or two out of the fold. For example, there used to be many standards for Linus base distros. Now all distros tend to fork from either RedHat or Debian. Standards thus emerged.

The same thing should happen today. We should see different distros attempt differing solutions to the issue and see which ones work best. Symphony is certainly one of the most interesting, but mostly because it's the first attempt to break away from the current designs that Linux is stuck in. :-)

Re:Evidence of problems with packaging systems

[KiloByte]

The only issue is: if you don't read the freaking release notes, you will have problems. The apt in Woody is broken. The release notes say that you need to update it first, to let it handle circular dependencies.
The only fault of Debian is not putting this in a bold enough font.

Also, this breakage gives us a yet another reason to bash C++ as a poor excuse for a language :p

Re:Evidence of problems with packaging systems

[Qzukk]

I mean, come on, there were 10,000 pairs of eyes looking at the source code and fixing bugs before it was released, right?

Right. And they fixed the bug, and told everyone that apt was broken and to upgrade to the fixed apt before attempting to upgrade to sarge.

And nobody listened.

Re:Evidence of problems with packaging systems

[runswithd6s]

They also don't have the resources to making security patches for every package without upgrading to a newer version of said package (i.e. backporting). They really do a phenominal job given their constraints.

I'm not sure what weed you're smoking, but Debian backports ALL of their security fixes from upstream software to the version packaged in stable. Really, consult the Debian Security FAQ for more details.

Re:Evidence of problems with packaging systems

[wobblie]

And there is always some post like yours, which clearly demonstrates you haven't even tried to figure out the answer to this simple question.

rpm systems: rpm -q --changelog
deb systems: /usr/share/doc/changelog.Debian

These are almost always more informative that the kind of crap I see on commercial unixes.

There is no such things as "patch levels" or "clusters of patches" in any linux distro I know of.

It is, in fact, a rather dumb idea anyway.

Each package is updated alone, as it should be.

To whom it may concern.

[ShaniaTwain]

Everything is falling apart. You may experience some discomfort. Just thought we would let you know. have a nice day.

Typical Debian!

[JimDabell]

Obviously this was a rushed job. Typical Debian, always cutting corners, never taking the time to do things properly :P.

I was waiting for Sarge but then came Ubuntu.

[Anonymous Coward]

Any reason why I should switch from Ubuntu to Debian?

Re:I was waiting for Sarge but then came Ubuntu.

[guyfromindia]

Exactly.. Ubuntu came in at the right time ... I dont think I will go back to Debian..

Re:I was waiting for Sarge but then came Ubuntu.

[varmittang]

Ubuntu is more of a desktop, latest updates type distro, while Debian is a strong, server type distro. So which do you need, depends on if you want a desktop or server, make your choice.

well duh

[Tharkban]

Well duh, if you wait that long between release cycles...you're going to have some major problems upgrading, as everything you had was ancient and everything you're upgrading to is mearly old.

I love debian for their philosophy; however, when I tried their distribution and it downgraded the kernel from 2.4 to 2.2 when 2.6 had already come out....I don't think I even started X before deleting it. Maybe I'd have had a different experience if someone had told me "testing" didn't mean what it usually does.

All of that said, it seems these problems could probably have been avoided with more testing, :( .

I've upgraded 6 boxes without problems.

[khasim]

What, specifically, are the apps that will cause the problems and how does he determine that 30% of the boxes out there will have those apps?

I've upgrade 6 boxes and have not had a single problem on any of them. They run a combination of Apache, perl, python, mySQL, php, bind9, DHCP, etc.

If there is a circular dependency problem on an app, but no one uses that app, then there won't be any problem upgrading.

Re:I've upgraded 6 boxes without problems.

[Qzukk]

So far I've seen one user with problems with TTF fonts, so if you're trying to pack every font possible on your computer, you'll end up getting stuck on "Regenerating font cache" (this particular user was stuck on ttf-bitstream-vera, so it may just be this particular font, or their language setting (french I think?)).

If someone does run into a circular dependency, I'd suggest using dselect to run the upgrade, or simply going into apt's package cache and using dpkg -i to install all the packages in the circle at once.

Upgrading a library that apt is using shouldn't be a problem, since the old library is loaded when apt starts, and will stay in memory while apt is running. Of course, if apt stops early, after it replaces the library and before it replaces itself, then you have a problem, but thats why apt isn't the only tool for the job. Use dpkg.

All of this assumes you know what you're doing, which by and large I've found most debian administrators fit the bill. That doesn't make this any less annoying, nor does it excuse apt's lousy circular dependency checking.

Mixing Lilo and some kernel configs not nice eithe

[Gentoo Fan]

We had a Woody to Sarge upgrade fail on boot because Lilo barfed a kernel panic on root mount. Installing grub fixed it. I forgot how the lilo was set up prior to sarge, but whatever. My suggestion if you have SATA root mounts: Install grub before installing Sarge!

Update took me two days ... grrr

[TekGoNos]

Nice to know I'm not alone.

Suddenly apt-get dist-upgrade didnt do anything good, I had to do an apt-get -f install multiple times until the dependancy stuff was sorted out. In the process, some packages (notably apache and ftpd) were simple de-installed and I had to re-select them manually.

Good for me that it was a server and apache and ftpd were the only important hand-selected packages. I fear for the desktop systems with several dozends of hand-selected packages.

So, I guess it is a good thing that Debian only releases a major update every two years :|

Re:Update took me two days ... grrr

[MarkSyms]

WTF were you doing using "apt-get dist-upgrade" anyway. If you'd read the release notes then you'd now that the recommended way of doing the upgrade was to use aptitude to prevent just those sorts of problems.

Re:Update took me two days ... grrr

[BeBoxer]

Suddenly apt-get dist-upgrade didnt do anything good, I had to do an apt-get -f install multiple times until the dependancy stuff was sorted out. In the process, some packages (notably apache and ftpd) were simple de-installed and I had to re-select them manually.

I can't say for sure that it would have helped, but the instructions specifically say to use aptitude because it handles dependencies better that apt. So while I feel your pain, I'm not sure it's a valid complaint.

Re:Mixing Lilo and some kernel configs not nice ei

[SpiffyMarc]

My brain exploded trying to parse this sentence.

And we wonder why we aren't taken seriously by management. ;-)

so long and thanks for all the FUD

[costela]

This is FUD, even by Slashdot standards.

The problems do exist, but the "severe breakage" described does not implicate unbootable machines or unusable software. Cyclical dependencies mostly mean the algorithm used to select packages for upgrade or instalation will not run as expected and probably leave the problematic package on hold.

This is not a new problem and affects Debian mainly because of it's distributed and loosely coupled model of organization, where integration problems can go by unoticed for quite some time.

The original mail intended to push more developers into taking action about these integration errors and make sure the upgrade paths are always clear, which is a very big and important task.

I, for one, hope his message doesn't fall on deaf ears, but also hope it doesn't generate more FUD like this.

Apt Would be Unbreakable

[Greyfox]

If they statically linked it. Which they should really do for a base level of core utilities anyway. I've been burned by library upgrades and crippled recovery processes several times in the past because the correct libraries were no longer available. For something that might have a library pulled out from under it like apt, it really makes sense to incur the size penalty so that you never have to worry about it dying on you when you replace system libraries.

Re:Apt Would be Unbreakable

[Spazmania]

That's not quite true. For example, the staticly linked apt in a previous upgrade could run in to trouble looking up DNS entries. The problem? /etc/nsswitch.conf got upgraded and the staticly linked DNS library didn't understand some of the new options.

However, offering a staticly linked apt would probably have helped.

Re:Mixing Lilo and some kernel configs not nice ei

[Qzukk]

SATA changed from IDE subsystem in 2.2 and early 2.4 to libata (and therefore part of the SCSI system) in 2.4 and 2.6

This is true.

[cdn-programmer]

I attempted an upgrade from woody to sarge about a month ago and it broke my system. I have 1000's of zombies running around. This shows up as a defunct process. Its not the end of the world mind you but you can't kill a zombie since it is already dead.

I have reported this and warned that there will be a lot of folks with broken systems. I was very surprised to hear that sarge went stable before this problem was sorted out.

A sarge install from scratch however is fine. Its just the upgrade that is broken and in more than one place.

Re:Cyclic Dependencies ?

[IpalindromeI]

I've flirted with the idea of installing Linux on a spare box. Is this nonesense the kind of stuff I should expect?

Do you have a reason to try Linux? Just from your tone you sound rather apprehensive of it in the first place. See if this describes you: "I'll just give it a shot so I can see why everyone is making such a stink about it. Then my condescending attitude will be justified because I actually did try it and didn't like it."

Frankly, even though you are obviously a "serious computer user" since you "create media" and "edit audio," if you don't have an idea of why you might want to switch to Linux, you aren't going to find a reason by just trying it out. What you'll probably find is that you can't figure out how to easily do the things you want to do in one afternoon. Or maybe you will, but they won't be any easier or wow-bang than just doing it in Windows. So you'll shrug your shoulders, wonder why everyone is making such a stink about it, and wipe the drive.

You should have a reason when you decide to do something, even if that reason is just to explore. If you were the exploring type, you would have already tried it, rather than just "flirted with the idea" of trying it, so that one is out. If you don't have another reason, you'll just be wasting your time. Honestly, it's the same with any decision in your life. Try thinking through things, rather than just randomly trying them because you know they exist.

Re:How to kill Debian

[runswithd6s]

The subject of the parent is itself suspect of reasonable objectivity. How does one kill a highly successful distribution that is 100% driven by the community at large?

"Take freaking forever to freeze for a release." There were a number of mitigating issues regarding Sarge, not the least of which was creating a new installation suite modular enough to work on all 11 ported architectures (not two dozen). Few can claim more portability. The second largest hold-up was the lack of an autobuild infrastructure for security updates. This was exhaserbated by hardware failures of key buildd daemons, etc. Regardless, time between releases is a sore subject for Debian Developers as well as the users. It is well-discussed on the lists, and in the public archive. Feel free to search debian-release, debian-project, and debian-devel for the relavent discussions.

"Take freaking forever to ship after freezing." I'm not actually sure what was meant by this. The freeze was done in steps, and once the actual freeze was announced, the release happened blazingly fast by most standards. However, this is subjective to POV.

"Ship a broken upgrade even after all the damn testing." How did Debian ship a "broken upgrade?" It created a few ISO images with a typo in /etc/apt/sources.list which prevented updates from an archive that contained no packages yet. What was broken? Additionally, published release notes and detailed installation instructions outlined the difficulties you might find during an upgrade from woody to sarge. What known breakages were hidden from view? What malicious intent did Debian have?

Seriously, to use your phrasology, the above post is nothing more than flamebait. If you don't like Debian's release cycle, either roll up your sleeves and participate in the process to improve it, or jump ship and use something like Ubuntu. Debian is not dead, is not in danger of dying, and could benefit more from helpful contributions than rants about its shortcomings.

I have failed in these posts by feeding the troll. I haven't provided a new defense or pointed out new facts. All of this information is available for those that would search (with little effort, mind you) for it. Happy hacking, and happy feeding.