Slashdot Mirror
File Packaging Formats - What To Do?
Jeve Stobs writes: "It seems that nowadays, there are three ways of distributing a program. In a tarball (be it a .gz or .bz2), in a Debian package, or in a RPM. These are all fine methods of packaging a piece of software, but they each have their places, and they aren't as comprehensive as I would like. I really think that, as we move into a broader user base with the variety we have so far (not to mention the variety we are likely to have in the near future) that a new method of software distribution is needed. osOpinion has an excellent editorial piece which details some solutions to this growing problem."
One of the strongest points about Debian, I think, more than any minor technical distinction between .deb and .rpm, is their strong centralized packaging organization that solves the above problems. (Based on this, they can also make nicer tools e.g. for automated network-based updates.)
Now, if only Debian stabilizes their new release so I can install it on my PPC...
If a thing is not diminished by being shared, it is not rightly owned if it is only owned & not shared. S. Augustine
I am reposting this from a previous article because it makes more sense under this one.
Ok, as only partially-initiated, I must ask, in spite of the simplicity of the philosophy of Unix, why oh why are there so many damn interdependencies in applications? Example: I install RedHat (yeah, shut up, it was the only thing that would install over DHCP on this old ThinkPad, after trying FreeBSD, Slackware, NetBSD, and TurboLinux), and choose the most minimal of configurations, and also choosing some small tools like cvs, etc. Well all of a sudden it is prompting me for all sorts of other dependent packages. I could not believe it when it told me I needed the entirety of KDE, and *then* also GNOME to satisfy dependencies! That is bullshit. Tk, Tcl, Python, Perl, Expect (!)...how the hell many things do I need to install? Am I the only one who thinks that backending GUI or administrative applications by Perl is just a god-awful abuse?
Sure this is just one experience, but I've found the same general thing when installing other distributions. Is this just a commercial flaw? Or do other "non-commercial" distributions like Slackware and Debian not require this? I just boggle at the horrendous amount of crap that even the most trivial of applications is dependent upon.
Ok, I'm putting on my asbestos trousers...
It's 10 PM. Do you know if you're un-American?
Time and time again we hear the call for standardizing something. The common argument tossed out is that such and such standard is needed if Linux is to succeed. I say baloney! to that. Linux will survive, grow, and prosper, on the basis of what has made it do so, so far: its diversity and opportunity to try things in a different way.
If we are going to standardize something, then maybe what we should standardize on first is a single type of user. Afterall, if we only have one type of user to have to deal with, then we don't have to deal with a diversity of needs and we can focus on exactly what this one type of user does need. Then we can have just one distribution, one filesystem layout, one package format, one window manager. Hell, we might even be able to narrow it down to just running a single application. Then Linux will RULE!.
... in your dreams!
now we need to go OSS in diesel cars
A new package format that would combine the strengths of the current various is a good idea, but (and it's a big but) I hope the author isn't the guy to do it (he doesn't have a good grasp of security..)
.02...
He mentions making the package self-extracting - "just do a chmod +x on it, and away you go"
Sorry, but that's why "security" in Windows is so abysmal..
Think about it - to install, you're running this file as ROOT - would you run a binary from an unknown source as root? (I don't even untar source as root!) - this is inherently a bad idea.
Just my
Packages are nice, but really, if we want to see Linux make it in the Real World (TM) we need to advance to the 21st century and recognize that we must have a simple and easy means of installing new software. I propose that we follow Apple's lead in this area and move to Self Extracting Executables, ELF binaries that you run and will extract the software and install it for you, without the hassle of remembering arcane flags and what program you're supposed to use.
/opt/local/lib/install/.
Just look at it! The author of the article makes a very good point. How many of you really look over a binary before you install it? Do you just rpm a package and then run it? What do you have to lose from running this random binary? No more than if you downloaded the rpm and ran the binaries contained therein.
What we as the Open Source community need to do is create a standard install tool, like what windows programmers have with the install shield software. The software could be customised, and could be written to reconize the directory structure, and install the files in the correct locations, maybe through the use of certain files that would be kept in a standard location, like
So, whaddya think? Think we can get something whipped together? Even just a proof of concept done in a combination of perl and python? We certainly don't want to make the mistake OS/2 made and not at least keep up with what windows has had for years.
-ssd
GNOME and KDE also have a similar concept.
My Blog. Sela Ward can sell me long distanc
Is everyone stuck in Linux? It seems like nobody has used /usr/ports on a BSD machine lately. Personally, I think it's got what everyone needs, being a package system that's got the features talked about here, and it even builds from source. Now, if I'm seeing this correctly, the reason we _have_ source distributions of things is a) for the customizability of it all , b) to have the source at our disposal to do with as we please. /usr/ports allows one to do just that. Coupled with ease of use ('cd /usr/ports/category/myprogram/' 'make install'), I think we've got a winner. Too bad the linux world doesn't have that... Until then, FreeBSD and OpenBSD get my vote...
c
-- you too can have an annoying sig!
It seems to me that the author simply lacks experience with RPM. RPM in its more recent incarnations has a Prefix option so that you can define relocations from /usr to /usr/local or from /usr/doc to /usr/share/doc or whatever. You simply specify directories that can be relocated without breaking the package.
That solves the problem of different directories for different distributions. If you are not using the same paths as the packager, you can give options to RPM to move the files to where you want them. Obviously it's not perfect because some programs hard code path information, in which case the directory cannot be relocated and should not be specified in a Prefix line.
The author also says that a standard .tar.gz should be used. I totally disagree with this. CPIO is very similar to tar, but has some distinnct advantages over tar. One such advantage is the ability to read filenames to include from stdin. Of course there is the tar -T option, which may be able to take "-" as an argument, or possibly /dev/stdin to read files from input. Another advantage to cpio is that it deals with device special files and pipes/sockets better than cpio. Of course all of these are moot points, RPM is already using cpio, DEBs use tar. Either way it's about the same thing.
Another concern was the dependency problems. Sometimes package maintainers will list other packages on the requires line. This is a really bad thing to do. RPM will automatically find library dependencies, so there is no need to do this. Packagers who do this are creating the problem, not RPM itself. Do note that RedHat has a habit of doing this, so they are somewhat guilty.
One issue I would like to bring up is that RPM works best on a system when EVERYTHING is an RPM and you do not install any shared libraries as source. I am thinking of making a tool that will generate a small .spec file which given a directory where a program has already been built, it will make install to a buildroot and then package the files into an RPM. That would be usefull for keeping track of programs that are constantly changing, such as those you are updating from CVS and recompiling only the necessary parts.
Finally, there is a need for a better version of alien. Right now alien is severly broken in that you must run as root to get good results. Alien should instead read the permissions and any device special information from the source file and use that information when packaging instead of requiring root access.
Perhaps another way to fix the packaging problem is a tool sort of like alien that would conver an RPM from one distro to an RPM of another. Something that could relocate files, fix dependency info, and then create an output RPM with the changed information. Of course I still say if the package is made right in the first place, this is unnecessary
Well, that's a lot to think about, hopefully I got some other people thinking too.
-Dave
And I mean this not just in regard to installers and packages, but everything.
And no, I'm not proposing that what we need to do is make Linux look more like Windows or the MacOS.
But there are problems that others have solved and we can draw on their solutions, even if we can't use their source code.
(Even when I was working at Apple I would tell people about stuff from SunOS or Linux that I thought would go good in the Mac - they wouldn't hear of it).
I think an indicator of the problem we face in trying to bring Linux to the desktop was when I was corresponding with RMS about things I thought would be helpful to the users and I suggested an installer. He replied "What's an installer?"
The best installer I've ever come across on any platform, both to create packages with and for the user to install products with is Mindvision Vise.
It would be worthwhile to find a friend with a Mac and download it, and make a little toy installer that installs SimpleText and a readme file to try it out (you can download it for free - the installers created with it complain that you've lifted it until you get a valid serial number. It is possible to get a serial for free for installers for freeware).
It beats the living hell out of anything I've seen for Linux.
BTW - if you want to see an installer that really blows, check out PackageBuilder/Software Valet for the BeOS. The thing drove me to distraction. It wasn't just the way it would corrupt the data in my archives or crash while users were installing my software with it.
What really drove me nuts is that it had no concept of updating an installer when I had built new software to go in it.
With vise you just drop your new files in the folder next to your installer project and tell it to update. It gives you a list of files that have changed and you can approve or disapprove of updating them (or deleting the ones that are now missing).
PackageBuilder requires you to delete the old file from the installer project, which loses its settings, then you have to go and add your file back in and reset your settings. This is probably the number one reason for every time I've been reluctant to release a new version of my software on the BeOS - I enjoy programming it but I hate the damn installer.
-- Could you use my software consulting serv
Why do Unix users have to be Sysadmins, but Windoze and Mac users don't? Why are the rules different?
It doesn't look that way to me. It looks like they just want a system that works well. Lowering the sites to Windows would be pretty unambitious.
Obviously it would be nice for the user to be more clued up. But there are a lot of people who use computers these days, and making cluefulness a requirement just isn't realistic. If it continues to be a requirement for Unix, then then people who don't want to learn sysadmin stuff are just going to have to scratch Unix off their list of options. Do you want them to do that? If so, why?
---
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
Instead of having large numbers of disparate databases, all holding essentially the same information, most of which is never kept in sync with the filesystem (and therefore becomes stale very easily), I'd like to propose that Linux filesystems become package managers in their own right.
And not just package managers. Most of the information that such tools as 'configure' generate is already known -to- the filesystem, albeit in a form that's not readily accessible (hence the need =FOR= 'configure'). But if the filesystem keeps an actual database of all installed packages and files (along with where they currently reside), then 'configure' scripts would become largely simple database searches, which would have increased speed and increased reliability. (No more having to tell 'configure' where non-standard installations are. It'll still find them.)
Also, finding files would be quicker. A linear search is horribly slow, when all you really need is a sorted index that you can do an n-ary search on.
Does this sound a bit like Windoze? Nope! NTFS and FAT32 keep very primitive caches, I believe, but not a comprehensive database. That's why you need installers with Windoze. The filesystem itself is not capable of acting as an "installer" in it's own right. The system I'm proposing would. A simple copy would be as "uninstallable" as an RPM, DEB or SRP. A tarball would have equal functionality as any package manager.
In a day & age where computers can perform feats so astonishing that nobody, even 10 years ago, could have imagined them, and where integration & interoperability have long been established, we are STILL using totally incompatiable methods of doing identical operations on identical spaces. This is stupid.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
The major advantage that RPM has over tar/make is the ease of removal.
Call me a wuss, but I like to know that I'm not going to have to spend an hour tracking down file all over the disk when something I installed sucks and I want rid of it.
The vast majority of RPM packages don't have any valid reason for being installed as root. The main reason you have to have root right now is to be able to write the RPM database. Fix that. For example, give me a local branch of the RPM database and give root the ability to merge that with the main, shared RPM database.
Miscellaneous other reasons for needing to install as root:
- Making the package available system-wide. Why can't I install a package in my home directory, then ask via a root-privileged command to link it from
/usr/bin? Or, more permanently, to move it there. Such a privileged command would still require security checks of course, but these checks can be very well-defined, and it would be easy to ensure that no suid programs were being installed. With a little more work, it could satisfy itself that no common user commands were being overridden (a technique favored by script-kiddies for hiding their root-kits).
- Installing device drivers. Solution: user-space device drivers. We're some distance away from being able to do that just now. For now, elevate privilege to root just long enough to install the device driver, and require root's password to install such a package.
- Installing daemons and privileged programs. Obviously, only root should be installing them. In many cases a daemon doesn't really need root privilege, and in that case it should run in user space and be installable by a normal user for their own use, or be shareable by the mechanism mentioned above.
- Changing system config files. The author should try to write the program so it does this only as a last resort. We should have a way of overriding certain system config files locally, for example, resolv.conf, so that we have a per-user view of the system configuration. As a last resort, the installer can invoke a root-privileged utility just to change the config files, and require root's password.
- (Other reasons, please enumerate.)
This is all by way of saying that we can have mindless (read: stressless; user-friendly) auto-installing packages just like Windows, without breaking security.--
When all you have is a hammer, every problem starts to look like a thumb.
I don't think it's quite so much an issue with the existing package formats as it is an issue with most of the Linux developers out there not really knowing what they want to do. It seems to me Linux is in this phase where a few leaders have been established, they're going their own way, and are becoming increasingly divergent.
It's basically a standards issue. I've used Slackware for a long time.. like it. Tar and gzip is all I need to install a package. But once stuff started creeping in that only came out in rpm.. I was forced to coax RPM work on my Slackware installation, or go RedHat. Maybe that's great for the l33t Linux hackers out there, as it gives them options, but for someone who's more concerned with getting software running on their machine, it's a hassle.
IMO, someone the entire Linux community trusts needs to answer some of these questions:
"Where do we install stuff?"
"What format should the distributed files be in?"
"How do we handle upgrades or patches to our software?"
Every answer to these questsions are subjective. This is why Debian and RedHat differ. I think that someone just needs to tell us how it's going to be, and let us start doing it. If it doesn't happen, in a few years, one really WILL be able to safely refer to Linux as "RedHat" or "Debian."
And I know some people out there are trying to answer these questions. But are they making ANY impact, whatsoever? The only reason the linux directory structure isn't complete spaghetti is because people kind of do what most other people are doing. That works great today.. but I think it's going to implode on us sometime in the future.
Now maybe good 'ol Torvalds doesn't want to take a dictatorship stance like that. I can understand that. But these questions of "standardized distribution schemes" will continue to rise as long as no one in authority stands up and makes a decision. Linux may truly only be the kernel.. but that's not how the world is viewing it.
I guess I'm okay with the issue remaining unsolved, but if that's going to be the case.. I think the community needs to start spreading that news. "No package standard is forthcoming, choose your favorite, live with it, and stop asking for something unified."
But of course, there's additional problems that appear with a statement like that..
There is an alarming lack of education about what parts of the linux file system are for. I'm sure regulars at #linux can attest to the massive volumes of new users asking questions such as "what's /proc?" or "should I install (app) into /usr, /usr/local, or /usr/local/mysql ?".
It would also appear that many application authors don't know where their apps should live. For example, mysql by default (in the INSTALL) wants to go to /usr/local/mysql, but other applications want to sit in the /usr/local/[bin/etc/lib] heirachy.
We need to get someone to provide a definitive explanation of what each part of the file system is for, and how they should be used, so that we'll be able to say "RTFM", and have a sounder understanding of our own operating system.
nf
That's worth thinking about. The MacOS has had something like this for years; programs are registered in a database when an executable is placed in a folder. Programs are found based on what they are, rather than where they are. Configuration files go in a standard place (the Preferences folder) and, by convention, can be deleted without major trauma. This simplifies the usual case for installation enormously.
So suppose we have a filesystem that holds "packages", rather than files. A "package" is a directory tree, and is normally installed intact and unmodified. Where the package is isn't relevant; there's a database of packages and information about them maintained by the file system. This database replaces "PATH"-type variables; everything that looks at at "PATH" type variable needs to become a query to this database.
Unlike the Windows registry, the package database needs to be locked to the file system. It's a cache for finding stuff, not something you edit. There should be something you can put in a package which causes a cache of (package-name package-version attribute value) items to be updated.
You also need a place to store preferences-type data. This should not be mixed in with the installed data; for one thing, much preferences-type data needs to be per-user. This looks like a database of (user-id package-name package-version attribute value)
It's too advanced an idea for Linux, though. The UNIX world is too heavily tied to explicit pathnames. BeOS, maybe.
I've only used Debian indirectly, it's never run on any of my personal boxes (because I'm too lazy), but I really appreciated the idea of having dependancies based on roles, rather than a particular version of a particular program. For example, you can specify that your software "requires a mail transport agent", rather than specifying "requires sendmail".
Debian also got a lot of things right by specifying up-front a standard for package naming. I'm sick of all of my dependancies breaking because I dared to install a non-RedHat RPM.
--
The more I learn about the Internet, the more amazed I am that it works at all.
Now, this might get moderated down as flamebait or as troll, but I have to say this: I don't think you guys should be allowed to run a *nix of any description if you are going to start installing stuff left, right and centre without knowing what files are going where. You're all going to think I'm crazy, but I really think the 5 or 10 minutes or so it takes to read and understand what happens when you type 'make install' is worth it. This is not Windows. You are trying to make Linux aspire to be Windows. This is not a Good Thing. It's not that you should make the interface neccessarily easier, but that you should attempt to make the user more clued up. I'd much rather see comprehensive documentation on the lines of "Makefiles for Dummies" being shipped about for the newbies than some new package format being created.
:-)
But seriously, to just randomly pull down a package from a site you don't know from Adam, and then as root say "Oh, go on then, do whatever you want" is plain madness, but then, you guys are going to flame me to death anyway, so perhaps I should just go and be quiet somewhere.
Ever heard of the MentalUNIX Packaging System(mpkg)? It takes a radically new approach to packaging. It's truly amazing what it does. The site is at
/home). It then checks the binaries against a build in database of known software, and can ad the program to the database. So, you can have rpm, deb, and plain ol soure code distributions that CO-EXIST! i suggest that you join the mailing list, and read the site. it has lots of information. Sccording to one somebpody said on the mailing list, sometime this week a detailed mpkg spec is going ot be released.
http://mentalunix.sourceforge.net
They currently have a 4 or 5 page "mentalunix paper" in the entire distribution, with a small part on mpkg. From what I've read(in the paper and on the mailing list), mpkg is going to be amazing.
No more binary packages! Everything distributed as source code. The nice thing is that it can be interactive. yes, interactive. If you load up the packager with the normal options, it will load up a simple console GUI(they are supposedly going ot make a Gtk+ based front-end too), and you can configure the program. It stores everything as XML. The congfiguration is handled in a config.xml file, the interface in interface.xml, the dependencies in dependencies.xml. The interfaces will be build using XML, TCL, PHP, and javascript(well, XML and any of those languages added). Imagine -- The kernel pacakge will have xconfig load when you open it! mpkg will also feature a daemon that montitors when new programs are installed in the the standard directories(everything but
-------------
HAL 7000, fewer features than the HAL 9000, but just as homicidal!
For stuff that isn't packaged with the applications, the package format should contain version information and checksums for any other files it depends on. It should declare those dependencies. The operating system should then be able to identify what is needed, possibly fetch it over the Internet, and give the package access to those files under a symbolic path name.
Such application bundles with dependencies should not (normally) be allowed to access the raw file system for their installation at all. In fact, even most applications should probably not be allowed to reference absolute pathnames. All references should be through symbolic paths. That way, one actually has a prayer of figuring out what the thing depends on, and it would also help with security.
Furthermore, the notion of "install scripts" is broken because it is difficult for anything to figure out what went wrong in the bowels of some script, and because scripts may do things to the system that are difficult to undo. Information about how an application bundles and their needs should be entirely declarative.
Another way of looking at that is that applications and application installation needs to work similar to objects and software components. In the past, programs consisted of functions that looked for, and modified, global variables all over the place. These days, good software components have well defined interfaces, get access to their environment only through those interfaces, and rarely use global variables. That's the kind of change that also has to happen at the level of applications.
The system that is closest to that in many ways is Sun Java: with jar files and the JavaBeans standards, there are well-defined ways for software to get installed and interact with the rest of the system, yet the environment can limit what new software components can do. We need something similar for Linux packages. That would also require additional naming and name translation support for Linux, similar to what Plan 9 offers (however, Plan 9 never adopted a good way of packaging applications based on their naming model--a shame, they were pretty close).