Slashdot Mirror
Building A Better Package Manager
SilentBob4 writes "Adam Doxtater of Mad Penguin has published a preliminary layout for his proposed cross-distribution package manager capable of adding/removing software from any locale. He is suggesting the interface will basically allow for installation of several major package formats including RPM, DEB, TGZ, as well as source code with the ability to pass build time options. All of this will come at the price of standards of course, including naming, documentation, and package structuring. If this idea were to catch on, it would signify a major leap in desktop Linux usability. This might be a project that UserLinux might benefit from. Read the full column here (complete with GUI mockups)."
So this is a similar effort to Autopackage except that it plans on using the native package formats? Intriguing...
True story.
It's called pkgsrc.
http://www.openpkg.org/
What about 0-Install? It is simple, elegant, doesn't require root to do an installation, seamlessly downloads libraries and other dependancies as they are needed, and integrates nicely into the filesystem. I really think 0Install could be the future of installers, if only they can get someone to build a distro around it.
Well first off, since programs get installed into different, and multiple, directories specified by the linux distro, you would have to make an installer for your program for every distro!
Packages let you package your proram once, and the package manager on the distro chooses where to put things instead of an installer that comes with the program.
i get a bit fed up with the constant pro-Gentoo advocating on slashdot, but eBuilds do have one thing I praise them for (as a Debian user).
;-).
python.
I'm not even sure that you're allowed to do build scripting in Python for debian, but to force gentoo package maintainers to write in proper maintainable language was THE RIGHT THING.
now sod off till you support a few more architectures you little spratling.
A package is basically the same thing as a Windows MSI file.
The problem is that different distros have different directory layouts, configuration file layouts, different places to put binary files, different ways of updating the internal library database etc. etc. etc.
The problem is basically a manifestation of there being more than 1 distro of linux and having distro maintainers who have not agreed on a common standard for this stuff. It's Linux's major achilles heal IMHO.
Invoicing, Time Tracking, Reporting
I would jump on the Portage bandwagon, and continue with that effort. There already a couple projects working toward cross-distro compatibily, for instance Emerde.
Portage has the ability to install RPM's on a Gentoo system. While not as easy as emerging from the source, it has a very wide horizon ahead of it. I am not sure aobut Portage using DEB's though.
apt, rug, and yum also already do this.
Ebuild's aren't written in python, they're simple bash scripts. Portage, the system that manages the ebuilds, is written in python, as are eclasses.
:-)
I don't really see a point to including ebuild's in this package manager though, as the package manager should already be doing the work of the ebuilds maintain source packages.
Besides, portage will kick this package managers ass anyday.
Yes, and it's called Portage. Written in Python, and uses wget to retieve AND install ANY dependants. On the fly.
Packages *are* the installers (like MSIs)... only each distribution of linux supports a different one (well, some of them support the same formats).
.tgz files .tar.gz or .tar.bz2
... (you get the idea, it's a pain).
In windows, "Add/Remove Programs" is the "Package Manager". Think back to Windows 3.11 where if you installed a program and you wanted to remove it, you had to delete the directory, find any files it dropped in c:\windows, delete them, edit your autoexec.bat, config.sys files... etc.
Since there is no uniform package manager for linux, and a lot of stuff is just distributed as source (ie: NO package manager support, you're back to the plane old file drop method in win3.11), it can be kind of frustrating.
For example: Redhat, Mandrake, Suse (and others) all use RPM.
Debian uses DEB files
Slackware uses
And anything can usually be found in source format, typically with the extension
It's rather sad when you're on Redhat, and you find a package and its either only in DEB format, or it's in SuSE RPM (which has different dependancies than redhat, so you might not be able to use it) or
So the point is, we need something equivalent to "Add/Remove Programs" that just *works* on all linux distros.
no comment
This reminds me of something I read the other day in the gentoo forums: Installing Portage on Other Distros
====
Crudely Drawn Games
> Don't make me go hunting down 20 dependency packages but offer to install them for me.
c.f. apt, apt-get
SURELY NOT!!!!!
"emerge -u gaim" will upgrade its immediate dependencies.
"emerge gaim" will just upgrade to the needed packages, or only gaim.
So what's wrong with this old song and dance?
./configure &&
make &&
make install
Other than taking awhile to compile the occasional large program, it's always worked for me. As far as making a desktop linux for dummies, the idea shouldn't be to have some magic whiz-bang tool that does everything and works on every platform and.... you get the picture. If there is a massive desire to use linux from people who are not competent enough to compile their own packages from source, then major distributions like Mandrake and SuSE will likely offer their own upgrade RPMs or DEBs or what-have-you for download.
Slackware, what else when it must be secure, stable, and easy?
That is because you use the -u flag. If you leave that out it will only update gaim or whatever you were updating.
"emerge -u gaim" will upgrade its immediate dependencies.
"emerge gaim" will just upgrade to the needed packages, or only gaim.
Mod up parent please! I don't have any mod points and this concisely explains how portage works.
For a simple Windows user, what are these "packages" and why do they need to be managed?
.RPM package with a GUI front end, not unlike what you would find in a .MSI file in windows -- even more simple, to be honest.
.dll file) to run properly, so the package knows which libraries are required for it to install. Debian, Gentoo, and FreeBSD have great systems for automatically installing those dependancies when the user requests a package, and the .RPM-based linux distibutions are getting better at this too.
.RPMs or use Debian/Gentoo to install the program, the package management system knows how to handle many of the default questions, installs everything in an expected place, and presents any questions in a familliar manner.
With many windows programs, the source is closed and the developer creates a binary package and controls how the program will be distributed. But with free software, many people take those source files and distribute them in whatever way works best for them -- a package is simply a way to put programs in a file for distributing to others.
If you'd like you can think of of package as an installation program -- with modern end-user distributions the distinction is minor. A package is RedHat, Mandrake, and SuSE all have programs that will automatically install a
But it gets more complicated than that, because of the increased complexity of the *NIX world. Certain programs depend on external libraries (think of it like a
It seems just eliminating the whole concept of packages would make life so much easier. Installation programs (like MSI files) are simpler, aren't they?
Some applications, like the Sun Java JRE, OpenOffice, and the binary NVIDIA drivers (I'm sure there are many others) have their own installation programs. It's ugly and messy and doesn't work that well compared with how each distribution handles packages natively.
To put it more practical terms, if I download OpenOffice from openoffice.org and run their installer I see a custom installation program that they have developed. I have to answer a lot of questions about how my Linux distribution is set up and do this all in an unfamiliar environment. However if I install OpenOffice
I hope this helps answer your question.
501 Not Implemented
When I try to upgrade my version of GAIM and there happens to be a better version of GTK available, Portage will upgrade GTK first, regardless of whether you actually need the very latest GTK to run GAIM. I'd rather see Portage know what the minimum version a dependency has to be in order to get a program running. As far as I know, it'll just upgrade everything in the dep tree.
;-)
Basically, this is wrong. Sorry.
The "-u" parameter to emerge will make it work as you described. However, if you just typed "emerge gaim", it would only emerge the minimum required. You have to ask for the "emerge all depencies, too" behavior.
I quite frequently emerge -up world, then just pick and choose what I want updated.
(I just checked "emerge -p world" against "emerge -up world", and "emerge -up" did significantly more packages on my system, where over 100 packages can be updated. On Gentoo, IIRC, the "world" is the list of things that you explicitly emerged; "emerge mozilla" will put mozilla in the "world" list but not any of its dependencies. So "emerge world" can update the packages you cared enough about to explicitly ask for them, and -u will add all possible dependency update.)
It is when used on Slackware. Slackware packages add the metadata in a specially named file in the archive, very much like a Java JAR manifest. (A JAR is bascially just a .zip with a manifest.)
When you use pkgtool to install from the tgz file, the package database is updated with info about the package that you just installed. It can also check for dependencies, etc.
Just be sure to wear the gold uniform when you beam down -- you know what happens when you wear the red one.
OK, I'll bite...
/usr/lib for any shared libraries that are no longer in use, or if any files have been altered. Thanks to dependency handling, it is also safe to remove unneeded old stuff (i.e. you don't have to put up with a gazillion old .dlls in c:/WINDOWS/SYSTEM32).
Packages in typical Linux distributions pretty much do the same things as MSI files on Windows, except that they do much more.
1. They describe how to build from source. That is (obviously) a big deal on an open source platform, since it makes builds repeatable, and so not depending on the magical build environment of one company or person.
2. They deal with dependencies: package "foo" can dictate that it needs package "bar" to work correctly, and that it needs package "foobar" version 2.32 or higher to build. This is a Good Thing, as you don't have to find out what the dependencies are the hard way.
This causes some problems from time to time, since distribution X may not have package "foobar", but the real problem here is that distributions are different. This may also be seen as a good thing: package management is a way to deal with diversity.
3. Standardised package management in a distribution makes other Good Things possible, such as automatic installations of all dependent packages, or automatic upgrades, thanks to tools like apt and yum and the dependency information in packages. That means that you can make sure that every program on the system is up-to-date with just one command.
Another really Good Thing is that package managers allow a lot more control over installations: they know which files are installed by which packages. That makes it possible to check, say,
A package is a file that contains information needed to install and uninstall a program. They are similar to MSI files, but have a number of advantages, mostly stemming from the fact that free software is, well, free, and so you can get it without buying it. Proprietary software comes on CDs, whereas free comes over the Internet. Upgrading free software is very "light weight" whereas upgrading proprietary software is usually very "heavy weight." This gives a different distribution model.
.dll files you don't need, weird interdependencies, and the system gets slower, more bloated, etc. This doesn't happen on Debian -- I installed my box maybe 3 or 4 years ago, and it's identical in functionality to if I installed it yesterday. Package management, well implemented, buys you that. You never reinstall the overall system, and upgrades are well-managed and don't break things.
.rpm even longer).
This has several effects. If I distribute a nonfree 10MB program UberTool, that requires the nonfree 20MB MegaLib, I'd better distribute MegaLib with UberTool. If both are free, I can distribute them seperately -- if the user already has MegaLib, he'll just install UberTool.deb. If he doesn't, the package management system will know where to grab MegaLib from, will download MegaLib.deb, and install it.
Furthermore, if I'm going from Office 97 to Office 2000, it's because I bought money on a CD, and I'm running an installer. In the free software world, upgrades are no-brainers, since they cost no money, and most free software programs are a smooth evolution, rather than major versions every several years. As a result, I'll generally be running the latest version of my office suite (as well as every other little utility on my system), and it is convenient to be able to do the upgrades all in one step (apt-get upgrade; apt-get update will grab all packages with newer versions, and install them, cleanly removing the previous ones). Most people never reinstall Debian -- I know installs from '96 that are still running today, at the latest versions, and there are almost certainly ones from before. I don't know of anyone who went from DOS/Windows 3.1 through Windows XP with just upgrades, and without a reinstall.
The next thing is Windows has a problem of bit rot. If you leave a Windows system without reinstalling the whole thing, adding and removing programs, etc. crap builds up. You get all sorts of registry keys you don't need,
The other place package management helps is in centrally-maintained networks. You can install the same package, with the same configuration settings, very easily from a centralized location.
So package management is, in effect, a fancy way to install and uninstall files. However, the fanciness buys you a lot. The new Windows installer is a form of package management, and gives some of the same advantages, although it's not yet as mature as the GNU/Linux ones (.deb has been around since at least '95, and
zapp's comment below is good, but I feel the need to add some things.
There are still PLENTY of Windows applications that don't use Add/Remove programs. You have to find their uninstaller, if they have one. This is the same as downloading a tar.gz with the source and hoping it has a "make uninstall" target. However, free software is available to track packages you compile and the files they install. Software is probably available to help uninstall stuff under Windows too.
With Debian, I can find out all of the files a package has installed. I can re-run the initial configuration. When I uninstall the application, it won't remove the config files in case I re-install later (unless I tell it to "purge"). I can query the package database and find out what the package is actually supposed to do, which is often a problem for me on Windows systems which are not my own ("What is this?" "I don't know!")
Additionally, a good package management front-end like APT, coupled with well-organized packages, makes maintenance a breeze. I can update every application with two commands*. Libraries are only installed if I need them, and it's safe to share libraries because programs won't be trying to install incompatible versions of them (if they did, the package manager would notice and tell me).
So yes, installation programs like MSI files are simpler, but I wouldn't say they were easier.
* Except for the source-installed apps, of course, but those are a) not packages, and b) few and far between on a Debian system.
WMBC freeform/independent online radio.
Don't confuse the FreeBSD ports with a packaging system. FreeBSD has its own nice packaging system. (If you've used Solaris a lot you'll feel right at home) The FreeBSD ports all create and install packages for you "behind the scenes", and you can install any package as binary on FreeBSD as simply as:
# pkg_add ftp://ftp2.freebsd.org/pub/FreeBSD/releases/i386/5 .2-RELEASE/packages/archivers/rpm-3.0.6_8.tbz
Umm, proprietary? The .app framework specification is documented, and is based on the OPENSTEP bundle specification supported by GNUStep. You can even put binaries for multiple CPU and OS combinations in the same .app, so you just need to create a single bundle and can run it on any supported platform.
I am TheRaven on Soylent News
Are you joking? Linux has had sophisticated package management systems for years now. apt lets you install, remove, and update software with great ease. Dependencies aren't an issue and haven't been for a long time.
When was that, the initial design phase? up2date was written specifically to address the problems of dependencies and automated errata installation. In Fedora Core, up2date supports APT repositories (and YUM repos as well). My rhn_applet lets me know when the fedora.us and freshrpms.net Apt repos have new packages to upgrade to. It even lets me know when one of the three repos I have configured (RH's yum repo for core and updates, fedora.us and freshrpms.net Apt repos for extras) has a newer package then the version I have installed from a different repo. For example, I recently noticed apt/synaptic did not inform me that the gstreamer-* packages all jumped to 0.6.4 on freshrpms.net because I had the 6.3 set installed from the Red Hat repo. What actually happened was it noticed the two extra packages from freshrpms.net had upgraded, but not the base packages. Trying to upgrade based on Apt's selections told me I had to upgrade the some of the base pacakges, but not all. But that failed. In order to upgrade with apt/synaptic, I had to manual pin each of the gstreamer pacakges, in a particular order. This led to 6 other packages being installed, which apparently is why Apt wouldn't upgrade all the gstreamer packages automagically. up2date, on the other hand did notice, and alerted me, and then proceeded to upgrade them and install all new dependencies without any hassle.
Red Hat engineers are also reportedly working on a standardized repo format, so that maintainers don't have to apt-ify, yum-ify, urpmi-ify, and yast-ify their collections to satisfy everyone.
As usual I'll come out with my Gentoo Zealotry but I'd like to deflect some of the problems I'm seeing mentioned here.
Gentoo is a Linux distribution largely centric to the Portage package manager (there are other features of Gentoo, but Portage is by far the most conspicuous)
Portage is a package manager loosely inspired by FreeBSD's ports system. Portage maintains a global software configuration file called make.conf. Make.conf holds meta-configuration settings about your system. As Portage builds all programs from source for your machine, make.conf is the place where you describe your machine to Portage. make.conf also holds a collection of use flags. Use flags are global binary switches. They have a default value if they are unspecified, and if you include a Use flag (ie USE="java") then it turns that flag on, and if you include -flag, (ie USE="-java") then it explicitly will not use that feature which is globally recognized by Gentoo.
I see complaints that emerge VI tried to build X and thus portage is "smarter" than you as a sysadmin. This is patently false and ignorant. Portage lets you do your job as a sysadmin once and then never have to worry about doing it again. If you do not want X on a machine then you need merely put "-X" in your use flags.
It puts control in your hands. If you want an application built to support certain things you can have it. If you do not want to support other things explicitly it will do that. It defaults tod doing what's sensible for most people who use Linux casually. If you aren't a casual user, spend a week or so getting familiar with portage and it's configuration. emerge is an incredibly potent tool. All of my systems are patched automatically every day, from source, with the configuration I have specified for that system. My binaries are all built with -march for the CPU, and -Os. And I've never once had any of my systems have a failure caused by misconfigured dependencies. They stay up to date and I don't have to worry about it.
If you want to do all your dependency checking yourself, you're welcome to. However there's a good solution that takes care of all of the issues revolving around this available, freely, to the world. Click here to find out more about it.
"Give away the stone, let the oceans take and transmutate this cold and faded anchor." - Maynard James Keenan
I was a long time Debian user, and I've "switched" to Gentoo. The primary reason I feel the ports/portage system is better is that I am not forced to install packages that have dependencies on other packages I don't need. For example, take gaim. In Debian, gaim has a dependency on NAS (Network Audio System), so I'm forced to install it. I don't need NAS. I don't want to install NAS. Gentoo has a USE flag that allows me to declare that I don't want anything to use NAS.
.deb. Say a new release of your favorite software comes out, but the package maintainer hasn't gotten around to packaging it. In Gentoo, in most cases, you simply copy the old ebuild file, and possibly tweak the version number. You don't have to download, compile, and package it seperately, as you'd have to do in Debian.
Also, it is pretty easy to make a custom "ebuild" file (which is a shell script) in Gentoo, and relatively difficult to create a new
There is also a lot less political activity in Gentoo, and they seem to Get Things Done.
Lots of these posts have mentioned that different distros use different packaging schemes and that there can also be problems with package names and dependancy locations (e.g. required .so files being in the wrong directory), but I haven't heard anybody mention the problem of binary compatability. Suppose I have program a.out which requires a.so and b.so to run. Now a.so also uses b.so, but a.so was compiled against b.so version 3 while a.out was compiled against b.so version 4. This will lead to problems as the loader will load only ONE version of the library and require both a.out and a.so to use the same version of b.so. This is why rpm's of the same program are not compatible accross different versions of the same distro! We'll never get out of the dependancy hell until somebody fixes this problem (e.g. Windows works the way you'd want). There's more info on this issue in the Autopackage FAQ if you're interested.
BTW: suppose I've got a.so version 3 on my machine and I go to install myprog. myprog requires a.so version 5. Upgrading a.so will require upgrading 500 other binaries on my system as they also depend on a.so version 3. As far as I can tell rpm doesn't want to let me install a.so version 5 (e.g. a.so.5) along side a.so.3 without using the --force option even though that would/should work in many instances and make an upgrade much easier. Am I missing something??
Actually the apple way doesn't work on just a single file. What you see on the Desktop or the Application's folds as an app icon for a binary, is in fact a special folder, with an extremely rigid and well documented layout (see here for more information)
.tgz or a tar.gz, decompress it, pop it into the apps folder, and expect it to run" is exactly what you do on OSX!
So when people talk about moving, or dragging and dropping, or copying an app from one machine to another, or from one folder to another, what they are in fact doing is moving a whole directory tree, with the binary inside of it (as well as other files).
Typically when you are ready to distribute your app, you do the equivalent of create a tgz file from the top level app folder. Your comment "take a
The differences are:
A package declares what other packages are required to install. Imagine if you were installing a program that *required* IE6.0 and Media player 5.0: a Windows installer will start up, run, and then barf saying you need to install something. A package would allow to to determine BEFORE you start that you need other things installed.
A package lists all files to be installed, and a package manager tracks who installed what. Thus, when you encounter a file you don't recognize, you can ask the system "Where is this from?" and the system can tell you "This was installed when you installed Gator.1.2"
A package can be digitally signed so that you know who created it without a doubt, and you can tell your package manager who you trust and who you do not trust. Again, this key can be inspected without running the installer.
Because packages detail what files they will install and what packages they depend upon, it makes the creation of a repository easier - a repo is a collection of packages that can be accessed by a program, so that you can say "Let's see - I want that, that, and that, but not that. Install those, please."
www.eFax.com are spammers
from emerge --help
I've also been using gentoo for a while now, and reading the documentation (as with anything else) can stop misconceptions from happening, and make the experience better all around
Here's a similar package funded by a dutch institution:
A-A-P
DNA is the ultimate spaghetti code.
On Windows, the linker records which DLL a symbol comes from and you can explicitly specify it in the source/header files using some simple language extensions. On Linux, that isn't done, and worse ELF specifies unscoped symbol fixup semantics.
To rephrase the original question: if libA1.so is linked against foo_func() in libA2.so, and libB1.so depends on foo_func() in libB2.so, and binary Z links against libA1 and libB1, then the results will not be what you expect. Both libA1 and libB1 will be linked to the foo_func() in libA1, because that came first in the load order.
This causes problems when different parts of the program use different versions of the same library - they may be independent parts of the code but the wires get crossed inside ld.so and things go boom.
It'll need to be fixed at some point. Using the ELF grouping extension in Solaris seems to make sense. Somebody just needs to write the code and (this is the hard part) get it accepted upstream.