Building A Better Package Manager

SilentBob4 writes "Adam Doxtater of Mad Penguin has published a preliminary layout for his proposed cross-distribution package manager capable of adding/removing software from any locale. He is suggesting the interface will basically allow for installation of several major package formats including RPM, DEB, TGZ, as well as source code with the ability to pass build time options. All of this will come at the price of standards of course, including naming, documentation, and package structuring. If this idea were to catch on, it would signify a major leap in desktop Linux usability. This might be a project that UserLinux might benefit from. Read the full column here (complete with GUI mockups)."

Comments

Autopackage?

[Deraj DeZine]

So this is a similar effort to Autopackage except that it plans on using the native package formats? Intriguing...

Re:Autopackage?

[IamTheRealMike]

By the way, it's funny this should be mentioned now but autopackage.org is in the middle of a DNS repropogation - it was switched to point to sunsite.dk literally hours ago.

For now, if it doesn't work, use autopackage.sunsite.dk and bear with us as we fixup the broken links etc.

Mnyah!

[American AC in Paris]

Bah, you kids, always inventing your own names for things!

When I was your age, we called 'em by their proper name--athletic supporters!

"Package manager", indeed...

Again?

[avalys]

This topic comes up how often? Like twice a year?

And yet nothing ever changes.

Re:Again?

[Deraj DeZine]

What are you talking about? These mockups are clearly slightly different from past years! Perhaps no code has been written, but everyone knows that impressive mockups are the key to building successful open source software with your newly-created SourceForget account.

Re:Again?

[pyros]

And yet nothing ever changes.

Not true. Red Hat's up2date supports apt repositories and the dpkg format is getting GPG signature/hash checking. From discussion late in the Fedora Core 1 beta stage it seemed that there is internal pressue to include apt in the core distro at Red Hat. Those are big changes, I think. I stopped reading the article since it's getting slashdotted, but it the author[s] can implement a single database that tracks installation by RPM, deb, and tgz, then I'd wager those features will be added to RPM and dpkg down the line. I honestly can't see either Debian or RedHat jumping ship to a new system, but they both borrow features from each other, so why not from this too?

it's not the package format

[ajagci]

People get so hung up on the package format. It really isn't about the package format, it's about the people and organization behind the packages and whether they produce a consistent distribution. A "better" package format or a better installer isn't going to help you when a piece of software expects libraries to be there that just aren't available, or when an install script assumes functionality you don't have.

Please explain....?

[Anonymous Coward]

For a simple Windows user, what are these "packages" and why do they need to be managed?

What is so special about this? It seems just eliminating the whole concept of packages would make life so much easier. Installation programs (like MSI files) are simpler, aren't they?

This is not a troll. Please answer my question, don't mod me down.

Re:Please explain....?

[cca93014]

A package is basically the same thing as a Windows MSI file.

The problem is that different distros have different directory layouts, configuration file layouts, different places to put binary files, different ways of updating the internal library database etc. etc. etc.

The problem is basically a manifestation of there being more than 1 distro of linux and having distro maintainers who have not agreed on a common standard for this stuff. It's Linux's major achilles heal IMHO.

Re:Please explain....?

[zapp]

Packages *are* the installers (like MSIs)... only each distribution of linux supports a different one (well, some of them support the same formats).

In windows, "Add/Remove Programs" is the "Package Manager". Think back to Windows 3.11 where if you installed a program and you wanted to remove it, you had to delete the directory, find any files it dropped in c:\windows, delete them, edit your autoexec.bat, config.sys files... etc.

Since there is no uniform package manager for linux, and a lot of stuff is just distributed as source (ie: NO package manager support, you're back to the plane old file drop method in win3.11), it can be kind of frustrating.

For example: Redhat, Mandrake, Suse (and others) all use RPM.
Debian uses DEB files
Slackware uses .tgz files
And anything can usually be found in source format, typically with the extension .tar.gz or .tar.bz2

It's rather sad when you're on Redhat, and you find a package and its either only in DEB format, or it's in SuSE RPM (which has different dependancies than redhat, so you might not be able to use it) or ... (you get the idea, it's a pain).

So the point is, we need something equivalent to "Add/Remove Programs" that just *works* on all linux distros.

Re:Please explain....?

[pyros]

So what's wrong with this old song and dance? ./configure && make && make install

No easy dependency tracking, no easy uninstall, no easy upgrade, no audit trail. On a server you don't usually want a compiler installed as it can be a security risk. It's really nice having a database of all the software installed, what versions of what other software it depends on, and reliable way to remove it without keeping the build tree around assuming the build system used has an uninstall method. The only way I would feel confident about not accumulating cruft due to upgrading big packages from source (gnome, kde, X) is if they are installed 100% into a single folder (like /opt/kde/3.2/(bin|lib|conf|man|...). Then I can safely uninstall by deleting that top version folder. Even then, I don't want to take the time downloading and compiling the source, I don't find it to be very recreational. I'd rather run `apt-get install kde` or `apt-get upgrade kde` or `apt-get remove kde`. With that remove command, it also removes packages kde depended on but nothing else does. You don't get that with source installations, you have to keep track of it yourself.

In the long run, unless you are meticulous about tracking which packages need which other packages, and where they were all installed, you are insuring you will have to rebuild your system from scratch at one point. Package managers like APT and Yum, and even up2date allow you to avoid this.

Re:Please explain....?

[IamTheRealMike]

Installation programs (like MSI files) are simpler, aren't they?

I think once you spent hours disassembling and debugging these "simple" installer programs to make them run on Wine you'd have a different view on the matter ;)

Let's do a quick review of how things are done on Windows:

• InstallShield/NSIS type installer programs. These embed all the data needed inside themselves, or in the case of InstallShield the actual installer is wrapped by the equivalent of a WinZip self extractor. Ever wondered why InstallShields take so long to start? Well, the first thing it does is extract a bunch of files (setup.exe, data.cab, some dlls etc) that comprise the installer, then it runs setup.exe which in turn extracts the InstallShield Engine to your local hard disk, possibly upgrading it in the process. Then it runs it and makes RPCs to it using DCOM, which starts the actual installation - done by iKernel.exe.

  This is sort of how autopackage works, except we do it in a much simpler way and don't rely on CORBA (the nearest equivalent of DCOM on Linux). These installers have no dependency management beyond "is this file the right version? No? replace it then" which has caused some truly horrific hacks like Windows File Protection.

• MSI packages. These are the closest Windows has to traditional RPMs/DEBs. You need to install the runtime support separately for MSIs to work. They are based on a bizarre kind of database, with its own dialect of SQL. MSIs are mostly data but are extendable via COM, iirc. They even deal with dependencies, via MSMs (merge modules).

  Yes, Windows apps have dependencies too. Check out this list to see..

  MSIs "deal" with dependencies by including the least common ones inside themselves, causing huge and bloated downloads, and leaving the user to figure out random breakage for ones that don't exist (how many times have you found that an app assumes the presence of some VB/VC++ runtime lib that just wasn't there?).

  They can get away with this because Windows is a platform and a whole pile of technology is guaranteed to be present. For instance, you know without needing to check that graphics support is available, because it's a part of the kernel and cannot be removed. On the server that's an achilles heel, on the client it's an advantage (in terms of packaging).

• MSI/InstallShield hybrids. [shudder]. Let's not go there. These things take evil to a new level.

• Zip files. All MS Windows binaries are relocatable. In contrast, virtually no Linux binaries are. That's partly because it's not at all obvious how to make them so - there is no glibc API available to figure out your absolute path, rather stupidly (and I'm too scared of Uli Drepper to submit one ;). We wrote a simple dropin C file to help C/C++ programs do this - making a program relocatable makes many users lives a lot easier, so do it today.

Because there is no standard Linux platform (the LSB never caught on), and the user can basically arbitrarily add, remove or upgrade individual components as they see fit (from the kernel to the theme in use) package managers are used to manage and maintain it all. Unfortunately, because there is no standard platform, the distro becomes the platform - of which there are many.

The freedesktop.org platform effort and the LSB are both worthy steps forward in this area and I hope they pick up steam. In the meantime, approaches like autopackage, being dependency-smart and having communities of packagers are the way forward.

OpenPKG

[chipster]

http://www.openpkg.org/

Why reinvent the wheel?

[El Cubano]

APT already handles debs and rpms. tgzs should not be a far stretch. The problem is establishing standards and getting everyone to follow them. For example, all debs in the Debian archive follow the Debian packaging standard, else they would not be accepted into the archive.

Naturally, third parties are free to create their own non-conformant debs. This is just the same as someone creating an rpm for RH9, but it not conforming to the conventions used by Red Hat.

I assert that the tools already exist. I.e., we don't need a new one. The emphasis needs to be on getting people to follow the standards, and possibly creaitng a cross-dsitro standard fo everyone to follow.

Unifying the Packages

[derphilipp]

Yeah !
Unify those packages.
I am so often confused the RedHat comes in a red box and SuSE in a green one. - Which of those should I buy ?
And Fedora comes with a box you have to fold yourself...

Oh you mean these packages....
(Fedora Linux is included in the RedHat magazine - which has a foldable page for creating a suitable box)

*BSD ports system?

[Anonymous Coward]

Why not leverage from the BSD ports system? It already builds directly from source, checksumming the downloads to ensure security, and applies BSD-specific patches. Shouldn't be too difficult to grow this so that source patches and binary packages are platform-neutral.

ps: BSD trolls are dying!

0Install

[Sanity]

What about 0-Install? It is simple, elegant, doesn't require root to do an installation, seamlessly downloads libraries and other dependancies as they are needed, and integrates nicely into the filesystem. I really think 0Install could be the future of installers, if only they can get someone to build a distro around it.

Re:Don't leave out Gentoo!

[bee-yotch]

Ebuild's aren't written in python, they're simple bash scripts. Portage, the system that manages the ebuilds, is written in python, as are eclasses.

I don't really see a point to including ebuild's in this package manager though, as the package manager should already be doing the work of the ebuilds maintain source packages.

Besides, portage will kick this package managers ass anyday. :-)

Re:Don't leave out Gentoo!

[revividus]

constant pro-Gentoo advocating on slashdot

Funny, all I ever see is Gentoo bashing. Are we reading the same sladhdot?

Learn from Apple

[Octos]

Maybe Apple can do this because they have a standardized directory structure, but what can be easier than dragging an app package to the Applications folder? Poof, it's installed. Don't like it? Delete it. If it's more complex, there's an installer program. Playing with dependencies and makefiles is the reason I gave up on Linux.

Good theory, poor practice

[arrianus]

The theory is fine. The problem is that package managers are, in many ways, incompatible. Debian packages, for instance, track dependencies based on the names of other Debian packages (libfoobaz-dev requires libfoobaz). I've seen package management systems that have dependencies based on files (libfoobaz-dev requires /usr/lib/libfoobaz.so). The former system won't recognize dependencies from packages installed in the latter format. Worse, the packages don't overlap. One distribution will have libgnome, whereas another will have 50 different packages, one for each of the Gnome libraries. The problem of dependencies there breaks almost completely.

There's also a matter of versions and security updates. On Debian, I run 'apt-get update; apt-get upgrade' and have a new version. Since the packages are all maintained by the Debian project (and a few smaller projects that target Debian), this works. Versions aren't linear -- Debian back-ports security fixes. The package manager has no way of knowing whether kernel-2.4.24 is newer than kernel-2.4.19-with-patches.

Basically, there is no clean way to install .rpm packages on a .deb system, or vica-versa, without breaking something. It is possible to install packages of one sort on the other system, but eventually, things will break. Each package management system relies on some set of information about packages to work, and each system has a different set of information it provides and needs.

There is room for improvement in package management -- a really good GUI for finding and installing packages would be nice. I wouldn't mind having more information about the packages I'm about to install -- links to project web pages, ability to browse installed files (the packages.debian.org/freshmeat.net/etc. databases either installed locally or quickly accessable from the system), the ability to view screenshots of GUI programs, etc. There's a lot of metainformation that could be added, and better search functionality that could be implemented.

At the same time, on the package build side, it'd be pretty simple to have a system where you make a configuration file of information about the package, and it builds .deb, .rpm, .tgz, etc. packages, and give easy-to-read information about what systems it'll work on. I've heard of tools similar to this, but I haven't seen them used. Adding something like this to the standard autoconf/automake/... process would certainly be nice.

The last solution is to have the groups work together to make sure all packages have the same set of metainformation (more than is needed for any given package system), so that cross-platform package installs become possible. In practice, I don't see this scaling across versions, as package management systems evolve.

One more thing to bear in mind is the perspective of the author of the article -- he says he runs Slackware, and builds most packages from source (something I've stopped doing maybe 3-5 years ago). Slackware's package management tools are very basic, manual, and crude. That gives a very different attitude towards package management than someone running a distribution like Red Hat, which has a much heavier-weight, more technologically-advanced, but somewhat fragile, somewhat inflexible package management system, or a user of Debian, which has a state-of-the-art ubermaintainable, uberupgradeable package management system, but that primarily relies on grabbing packages from one (or a small number) of sources. I apologize about the stereotypes in this paragraph -- they're not entirely true, but the package management systems differ a lot (more than most people realize, if you've ever tried to build the packages), and I'm just trying to make people aware that users of each of them will have a very different world view, and it's important to keep that in mind when reading these articles.

Re:Don't leave out Gentoo!

[polin8]

"emerge -u gaim" will upgrade its immediate dependencies.

"emerge gaim" will just upgrade to the needed packages, or only gaim.

A few things:

[Wordsmith]

Why hasn't anyone developed a system that, from the End-user perspective, works similarly to MSI installations (which work very well). Point, click, next next next. In principal, DEBS/RPMs work similarly to MSIs, but the installation isn't as obvious a procedure to end-users.

And for that matter, why not make the installer intelligent about the distro? Use a single package/installer, but that includes all sorts of scripting information about installation in variosu circumstances. The installer checks to see if it's on RH9, and if so it puts files where RH9 expects them, editing any configurations and making RPM database entries as necessary. If it's on Debian, it takes the appropriate measures there. And so forth.

Why do we see such absurd dependencies that don't seem to happen in the windows and mac worlds? Install a new version of a KDE app, and you need the latest minor revision to the core KDE libs, which in turn requires a minor upgrade to the font server, etc. In the windows world, occasionally you need to update something big like DirectX to install a latest-and-greatest app, but even then the dependencies are often packaged with the app itself. Why isn't this practice more common in Linux/Unix (not counting Mac OS X)? I undestand that many of these apps are under CONSTANT quick-release development and are often tied to bleeding-edge versions of their libs, but why aren't major releases at least more dependency-friendly? Installing an app can be a real pain in the ass even with something like apt, if you don't have the dependencies in the repositories you've defined. And adding new respositories isn't exacly grandma-friendly.

Packages

[arrianus]

A package is a file that contains information needed to install and uninstall a program. They are similar to MSI files, but have a number of advantages, mostly stemming from the fact that free software is, well, free, and so you can get it without buying it. Proprietary software comes on CDs, whereas free comes over the Internet. Upgrading free software is very "light weight" whereas upgrading proprietary software is usually very "heavy weight." This gives a different distribution model.

This has several effects. If I distribute a nonfree 10MB program UberTool, that requires the nonfree 20MB MegaLib, I'd better distribute MegaLib with UberTool. If both are free, I can distribute them seperately -- if the user already has MegaLib, he'll just install UberTool.deb. If he doesn't, the package management system will know where to grab MegaLib from, will download MegaLib.deb, and install it.

Furthermore, if I'm going from Office 97 to Office 2000, it's because I bought money on a CD, and I'm running an installer. In the free software world, upgrades are no-brainers, since they cost no money, and most free software programs are a smooth evolution, rather than major versions every several years. As a result, I'll generally be running the latest version of my office suite (as well as every other little utility on my system), and it is convenient to be able to do the upgrades all in one step (apt-get upgrade; apt-get update will grab all packages with newer versions, and install them, cleanly removing the previous ones). Most people never reinstall Debian -- I know installs from '96 that are still running today, at the latest versions, and there are almost certainly ones from before. I don't know of anyone who went from DOS/Windows 3.1 through Windows XP with just upgrades, and without a reinstall.

The next thing is Windows has a problem of bit rot. If you leave a Windows system without reinstalling the whole thing, adding and removing programs, etc. crap builds up. You get all sorts of registry keys you don't need, .dll files you don't need, weird interdependencies, and the system gets slower, more bloated, etc. This doesn't happen on Debian -- I installed my box maybe 3 or 4 years ago, and it's identical in functionality to if I installed it yesterday. Package management, well implemented, buys you that. You never reinstall the overall system, and upgrades are well-managed and don't break things.

The other place package management helps is in centrally-maintained networks. You can install the same package, with the same configuration settings, very easily from a centralized location.

So package management is, in effect, a fancy way to install and uninstall files. However, the fanciness buys you a lot. The new Windows installer is a form of package management, and gives some of the same advantages, although it's not yet as mature as the GNU/Linux ones (.deb has been around since at least '95, and .rpm even longer).

Re:FreeBSD ports collection

[pyros]

I will however say that in my experience RPM's are a poorly implemented packaging system. I'm sure plenty will mod me as a Troll or Flamebait, and other's who've never used a linux outside of RH or MDK will pipe up extolling the beneifts of RPMs and i'm just a stupid, lazy fsck, blah blah.....

I'm not going to call you a troll. But I am going to assume you dislike the RPM format due to the dependency-hell problems of the past, if that's not why you dislike it, then feel free to ignore the rest of my post, except the PS. The problem is that many distro maintainers selected the RPM format (Red Hat, Mandrake, Suse, Ark, even the LSB chose RPM as the standard format), and then packaged software with conflicting package names and file system layouts. So you go looking for an RPM for Red Hat, and find one from Suse, and it says it needs xfree86 3.0.3, even though you have XFree86 3.0.3 installed. Or perhaps it needs some particular .so to be in /usr/lib but the Red Hat package owning that file put it in /usr/lib/ssl.

These aren't flaws in the RPM format, these are the problems this project aims to fix. You would see the same problem with dpkg if there were as many popular distributions which used dpkg but didn't base themselves off of debian's repositories.

The other complaint commonly made is that apt is better. Apt needs a tool like rpm or dpkg behind to actually be useful. Apt is purely that part of the system that locates dependencies, like the part of portage that knows to build and install X before building and installing GNOME. It doesn't actually install the package or maintain the database of installed software. Apt also runs very nicely on RedHat. Connectiva and Ark use it as the default system. Mandrake and Suse implement their own dependency tracking system.

PS - In addition to Red Hat and Mandrake, I've also tried FreeBSD (years ago, didn't support all the hardware needed to install), Debian, Suse, Ark, and Slackware.

Re:Don't leave out Gentoo!

[V.P.]

I've been doing an emerge -u world for about 2 years now and have never ever had a problem.

Two years huh? How's the good ol' 386SX going?