Slashdot Mirror
Jordan Hubbard On Next-Generation Packaging
GlobalEcho writes: "Developers associated with Darwin are beginning to think about package management and source building. At issue is whether something like dpkg, RPM or *BSD's ports could suffice, or whether they are all just way too mid-90's. Jordan Hubbard himself (now of Apple) weighed in with his opinions (user and passwd 'archives'). Apparently he thinks it is time for something more advanced, and he gives some ideas about what that might look like. Does anyone else have good ideas?"
If I'm not mistaken the whole Ports thing was one of Jordans great inventions. It's succeeded quite well using standard distributed tools (ie. makefiles, compilers, and the like).
Perhaps I'm wrong. Nice to see he's still having great thoughts. Hope whatever packaging system they come up with is portable enough to work on a large chunk of systems (linux in various configs, bsd's, solaris, darwin, etc.).
Rod Taylor
Quite frankly, dpkg isn't all that bad. It has MAJOR issues; no dobut about that, but has many great concepts which can't be found anywhere else (correct me if i'm wrong. they're still good ideas, though!)
The dependency and dependency resolution system- dpkg has the most advanced dependency system known to unix. No dobut to that... To solve these dependencies, dpkg goes to it's list of package locations (complete with http and ftp locations, cdroms, etc.. if necessary) and grabs the required packages from the net (the user is prompted on this, of course)
--Easy upgrades. No other system allows me to bring my system up to date in less time (note: debian isn't updated often, so this is generally unappreciated)
$apt-get update
$apt-get upgrade
(hit y to confirm)
All from the command prompt.
I'm not sure what else there is that makes it good. But RPM certainly doesn't have these features.
What it lacks:
It's buggy as hell - it's easier then signing up for aol to nuke your system this way (in other words, it happens quite often by accident)
No good front-ends - There is no good program to browse available packages, install them, enter configuration information (more on that in a sec) and remove them. You should enter the package you want to install. a wizard is displayed, it grabs the package from a mirror or local source, solves dependecies, installs it and any dependent packages, configures it, and exits.
Configuration - dpkg has a system that allows the package to prompt for a few options before it is installed. this is a good thing, but the packages usually don't ask enough. users need full customization (nothing nitpicky. big stuff... so you dont have do manually edit configuration files by hand.
Available packages - this is where dpkg falls flat on it's face. 95% of unix packages are rpms. that never helps. a unified packaging system needs to be put into place
i dunno what i forgot?
-- If you try to fail and succeed, which have you done? - Uli's moose
But in practice I just don't see how it's gonna be any different in the long wrong. I mean with the xml you'll be able to do alot more stuff, a simple database etc it starts to get a little bit big for a porting system. How long until this becomes obsolete because the database is too big to search effectively.. I like it but i just think that it's implementation is gonna be the hard part. Then again implementation is always the hard part.
Absolutely! sorta.
The G4 differs from the G3 in a lot of piddly little ways, and one great big, huge way. The G4 has the AltiVec engine (a vector computation unit, alongside the Integer and Floating Point Units). However, it's a little specialized and (afaik) there aren't any compilers that automatically generate AltiVec-optimized code. On the other hand, programmers can use AltiVec (vector engine) accellerated functions, which will then be much faster on G4's than on G3's.
As an example of an AltiVec accellerated function, look up the man page for writev(). Basically, it's a vectorized (and thus accellerated) version of write().
It's buggy as hell - it's easier then signing up for aol to nuke your system this way (in other words, it happens quite often by accident)
Huh? I suspect user error. I've been using Debian/dpkg since pre 1.0 days, and I can count the number of times dpkg has had system wrecking errors on one hand. I can count the number of times that it actually wrecked my system on one finger -- after that, I got a little more cautious about upgrading dpkg in the unstable tree. (i.e. wait a few hours and read debian-devel), There are ways to tell dpkg to hose your system, but those aren't bugs, those are options with big nasty warnings next to them.
Now, there have been many more occurences of buggy packages screwing things up, but that's hardly dpkg's fault. And if you live on unstable, well, that's what you get.
No good front-ends -
apt-get install aptitude
(Not in stable, but coming soon[1] to a release near you.
Configuration - dpkg has a system that allows the package to prompt for a few options before it is installed. this is a good thing, but the packages usually don't ask enough.
Again, not a dpkg issue. If the package doesn't provide sufficient configuration flexibility, it's an issue with the particular package.
Available packages - this is where dpkg falls flat on it's face. 95% of unix packages are rpms. that never helps. a unified packaging system needs to be put into place
I don't know where you got that statistic. Yes, maybe 95% of packages you see floating around random websites are in rpm, but I doubt that 20 times as many software packages available as rpms vs. debs. Most upstream developers don't provide debs, because there's a debian developer to do so for them; the fact that mozilla.org has rpms but not debs doesn't mean there aren't .debs of Mozilla. (I'll allow that the ratio for non-free software is much worse, for fairly obvious reasons.)
Steve
[1] "soon" in Debian terms, at least :-)
We use a simple XML file-based (i.e., you can edit everything with vi) object-oriented database. The project isn't just about package management, but we implemented a full multi-platform build-from-source-and-install-sitewide package management tool. It also handles dependencies etc.
What I would hate to see are any major revisions if it's just gonna add some feature; I would rather see that time spent on developing the ports and packages themselves. Make is a good, simple, foundational and almost always present solution. Adding other languages would be a waste of time IMHO.
Let me condense what I think should be pursued from the ports perspective: documentation and ease of use. One can always make readmes and get mini-descirptions, but that really should be expanded upon, both for beginners and seasoned users who just don't know what that software is about. It would be nice to have some options like, info that would go thought the ports tree and build more verbose information. If those documents are built in a consistant manner (such as xml), then any ol' front end can be built to pull the info on the port and automate building the port and the flavors available. For example, a simple curses interface that will inform you of the dependancies that will need to be built first, estimates the size, and gives you a list of flavors to add into your build. Hit ok and it monitors the progress for you, logs the process and keeps the messages out of sight (from those who get scared easy).
I agree that something should be done to be able to automagically build a package from a port. I think this area would be the best to pursue. Even better, if we bsd types could get a system like checkinstall /installwatch consistantly, not most of the time, but consistantly working on BSD. This project essentially is a wrapper script that records everything make install does. In current form, it gives you the option of building an RPM from that make install. What should be pursued is making this work -well- on bsd, with the option to build a package along with documenting it's dependancies and/or recording the install info into the existing system to that all one has to do to remove what you just built is 'pkg_delete'. THAT would be cool!!
Democrats and Republicans only disagree about how to enslave you
None of these ideas are new. We at the OpenPackages project have already discussed these ideas and more. We have a pretty solid plan, but unfortunately no one seems to have the time to turn the proposals into a design document and get going. Like Jordan said in his post and i said in my follow-up, time is the critical issue. This is a big job.
t p://openpackages.org/pipermail/op-tech/2001-Apr il/000764.html/ op-tech/2001-May /000826.htmlp -tech/2001-Dec ember/001454.html
Here are some references i included in my darwin-devel post:
http://openpackages.org/html/pkg_design.php
ht
http://openpackages.org/pipermail
http://openpackages.org/pipermail/o
Check out the fink project
http://fink.sourceforge.net/
600+ OS X ports so far, automatic updates,
database indexing, built on top of dpkg.
Gentoo Linux is aiming to create a much enhanced collection system.
The username/pass are mentioned in the story. :-)
Many unix users seem to think existing solutions are great. You know, I've taken to Cocoa since I got my Mac last month, but there is one thing that drives me crazy and thats figuring out this freaking package manager. Its not easy for people use to using Windows Setup or other installation packages for Windows development tools.
The problem is UNIX isn't designed for the average user. When you look at it from a Unix perspective, it works great. When you look at the rest of the world, it doesnt. And anything thats a single platform does not create a big issue. You won't see it used on all your open-source console apps, mainly MacOS X applications. This is definately needed in Mac OS X.
www.atacomm.com - The Leader in VoIP Product Distributi
The dependency and dependency resolution system- dpkg has the most advanced dependency system known to unix. No dobut to that... To solve these dependencies, dpkg goes to it's list of package locations (complete with http and ftp locations, cdroms, etc.. if necessary) and grabs the required packages from the net (the user is prompted on this, of course)
Dpkg doesn't do that. APT does. Its not that these aren't great and massively useful features, its just that just like urpmi, APT works its magic across RPM (the LSB standard packaging system, which many app packagers primarily package for) too. I maintain an apt repository for my workplace with around 3000 packages, all in RPM format for Red Hat 7.2, and it works like a charm. Debian's policies are postable too - Connectiva has a similar set of guidelines. The unique advantages of Dpkg is suggested / recommended dependencies, something RPM desperately needs (Red Hat themselves cheat and use the `comps' file to provide this logic themselves in the installer, but us users don't have the luxury). RPM has some unique advantages too tho - transaction handling in the database (thanks to DB3) does wonders for my piece of mind. In the end, I'll stick with RPM and Apt-get because of the LSB stuff, and avaliability, but I do hope they add suggested / recommended dependencies soon.
Actually, if you want to see a system which kicks both their arse in many ways, look at QNX. They have apt-like features, a nifty `package filesystem', and a GUI installer that reallycraps all over every other software install system I've ever seen.
Jordan, I've been using FreeBSD since about '94. I can't really remember it being without ports. It shouldn't be to hard for a hacker of your class to make a GUI front end for it. Or at least some of the Apple hackers. Hell, I wish I could, but I've given up on anything more complex than a PHP/database driven website. All I know is, the FreeBSD style ports system has saved my ass many, many times. Thanks for all the cool stuff!
--- Think of it as evolution in action ---
Hurbbard states :) for any number of potentially
"1. All the descriptive text which current goes into a Makefile like the
port's name, version, dependencies, URL for source bits, and so on,
should go into an XML file. This immediately allows the port to be
indexed, documented and modified from automated tools which traverse the
"ports collection" (sorry, I have to keep using that terminology since
it's what I'm most familiar with
interesting reasons. The reason a lot of these tools don't exist for
*BSD today is that extracting data from Makefiles generally sucks from a
parsing perspective so people aren't encouraged to get too creative."
To reprase.
Metadata should be in XML, this makes it much easier to process than if it has to be extracted from Makefiles
But why does it have to be XML (or SGML or RDF for that matter). The only reason i can think of is that there are pre-existing tools to parse it, the tradeoff is that any Markup language will bloat your metadata and make it unreadable.
Shouldnt a packaging system be important enough to warrant the development of its own parsing routines. i.e. just design it to be the best it can be
Is http://www.openpackages.org/ dead? why did it die?
You can wail on about this average user but you must be careful about this cliche. Because the implicit fallacies is that there is an average user of these systems, that they have far less experience than you or I do, and that they aren't already happy with what they are using now. If there is anything more certain that can be said at all about these average computer users is that they probably don't want to change operating systems right now. In fact, not only would there quite possible be no reason for them to switch operating systems right now but it would mean erasing the skills they have learned using the current operating system. Its no coincidence that those who do migrate to other platforms have little to loose. If you think that these so-called average users spend all their time surfing web pages and sending email then there would probably be a lot more migrators to other operating systems.
I think it is useful to consider a few things I learned in a Cultural Geography class last semester. I know these things are pretty much common sense but I think its not only useful to consider these ideas but to introduce some new terms when dealing with these things (rather than using impoverishments like average user). When people migrate from one region of a country to another, they do so for a number of reasons. There are push factors and pull factors.
So why would someone move to a Free operating system? It seems that freedom itself isn't much of pull factor (but this would change, surely, once many of these software laws and licenses are really enforced against end users, not just distributors).
Let me say something about ease-of-use. While it would seem to be an obvious pull factor--the days of easy to use general-purpose operating systems are long over, I think. Perhaps the first Macintoshes were among the easiest systems to use and the reason for this is quite simple. The needs and expectations of users have gone up quite a bit since then. While I have never used these early computers nor do I know the intentions of the Apple staff (these things are probably clearly documented somewhere...I'm too lazy to look right now), I would suspect that they were trying to make as easy as possible to type out documents with relatively sophisticated typessetting (compared to typewriters!) and then to file these documents into a filing system.
Today's systems are expected to require quite a bit more. Many of the posters here on slashdot carrying-on on what these operating systems need to be successful (in what ever definition of success, most do not say) give examples:
The paradox is that this average user needs all of this. This seems extremely unfair to anyone trying to implement an operating system...nowadays to any team of programmers or consortium of developers contributing to an existing free software project.
Now lets consider what relevent about talking about the average user. Like I said before, I doubt this user would switch his or her operating system for any reason. This is because for everyday this user masters his/her OS, the push factor from every other OS becomes stronger and stronger. Unless there exists a push factor from the OS he is currently using, he's gonna stay.
So lets forget this average user since it isn't relevent or even interesting. Lets consider, instead, a different class of users. Lets just create a class of users who might or will definitely switch to another operating system. Now, awaiting to be smacked around with a stack of statistics proclaiming otherwise, I would guess that this set of users would have the following things more or less in common:
And where would you find this average set of OS migrators? Probably on the internet: in newsgroups and web forums. Specifically, you would find that many of these people read and post to slashdot regularly.
And thats the point to this entire post. I find it interesting to hear slashdotters condemn the intelligence of the average users, how they can't program, or they can't figure out the command line. This might be true, but they are revealing their own experiences more than anything else. They are their own breed of software users.
In conclusion, You Are the Average User.
Jordan said "I think 10,000 entries is going to be something of a stretch even for the FreeBSD ports team, but I don't see that number being entirely improbable for some Macintosh equivalent since there are a lot more Macfolk than there are FreeBSD users." This comment tickles at a core problem with the transition from OS9 to OSX for many Mac users. There may be more Macfolk than FreeBSD users, but how many of them want to rethink package utilities? A lot of Jordan's comments are good suggestions (for instance, implementing the new standard of XML into the package process), but it seems that this problem will be tackled by *nix users like himself moving over to the OSX platform, and not by OS9 users delving into *nix for their first time. Still, it IS a good time to rethink things before momentum makes it difficult to change. I also agree with him that Apple should have spearheaded the process, and they probably will in time. Apple has always focused on userfriendlyness first, then streamlined what was under the hood last. Compared to the past, Apple seems to be responsibly balancing the development of OSX across the board. However, if OSX is to succeed with the typical Mac audience, it will need to be a lot more stable and userfriendly to the simplest Mac user. I can't imagine my parents enabling their root account to reconfigure folder permissions as I had to do recently. Heck, my dad can barely figure out his email. LOL, nevermind launch a shell and interface via CLI. I imagine streamlining the package utility mechanism is low on Apple's priorities. Still, it's comforting that Jordan is mulling the problem over.
The Splintered Mind - Overcoming
I, personally, think the FreeBSD Packages/Port system is damn near perfect from a user/sysadmin standpoint.
I've tried fink, and it seem terribly unintuitive and clunky to me, and I don't really use it, prefering to get packages from macosx.forked.net, because they just install, and I don't have to screw with funky menus that don't work simply, and I don't have to hunt for fink packages that aren't in my menu, even though I grabbed the latest list.
Sure, I could spend the time to figure fink out and get it working properly. Hell, I could be using netcat to write this comment, but I'm not going to.
People who actually need to get stuff done, need intuitive and simple tools. Fink and the like are fine for home hackers/users, but when you need to get stuff done, fink doesn't cut it, I'm sorry.
FreeBSD Ports/Packages is very simple, and easy to use, and gets the job done well. It's probably the main reason I use FreeBSD servers (aside from the insane levels of stability).
I can understand Jordan wanting to move above and beyond, I just ask that he keep things simple and intuitive. Something I can get around in with a small, half-page cheat sheet.
I second the motion!
Subpackages and package variations are two major things missing from all the candidate systems. I mean, Windows can manage to do this, why can't we? Why can Windows users select "custom" during install but we have to take whatever the packager decides to give us?
One example: Dia. Under FreeBSD the port and package requires Gnome. But Dia will build just fine without the Gnome libraries. Everytime I want to install or upgrade Dia I have to go in and edit the Makefile.
Proposal 1A: All meaningful configuration options should be easily available to the user of ports and any successor to ports. All major configuration options should be easily available for packages. Let me choose "custom->without-gnome" when I install Dia.
Another example: KDE. This already exists as a meta package in FreeBSD, but a metapackage is not the same as a set of subpackages. Uninstall KDE and nothing actually gets uninstalled, as all the dependencies are still there. At an even finer grain of detail, maybe I don't want to install everything in kdegames. Maybe I just want shisen-sho and patience. Allow me a way to install just what I want out of a package. The Debian way of splitting packages into smaller packages (instead of subpackages) is not an optimal solution.
Proposal 1B: A port or package that contains optional or secondary components should allow the user to choose what components they want installed.
The defaults should remain for obvious reasons. But the package manager should allow -full, minimal, and -custom installs. For a CLI installer, these could be switches (with -full being the default), and for GUI installers they should be selectable options. For some packages it won't make sense to have more than one option, but for most it will.
These proposals may mean the replacement of metapackages with superpackages. It will mean more work for the package developers, but that's outweighed by the benefits to the user.
A Government Is a Body of People, Usually Notably Ungoverned
I think for a next generation packaging system Apple ought to adopt concepts from their frameworks method of binary packaging. A package would contain an XML header relaying to the installation program information like the file contents, checksums, a list of required and a list of optional packages, install and make instructions (including file destinations), a hash of the compressed file, and a dependancy list. The installer program would scan the header and ask the user which parts they wanted to install or install options based on arguments like installprogram -a packagename would install the entire package or some such. Since there's a hash of the compressed file it could be checked against a hash stored on the server the package came from for confirmation, from there the installer would figure out using information of the header whether it had to just cp and chmod some files or if it had to compile them from source. Packages could come in source only or binary only or in combination packages containing both. The frameworks the installation fromworks would output to (or just plain directories) could contain a header telling which files exist on the drive and a master lister of installations could be kept by the installation program. So you could whip it open and remove GNOME lets say and it would consult a relatively small XML file telling it where the GNOME XML resource file is and then procede to uninstall the files listed there. The XML header for the package could be even compressed inside the file and follow a standard naming convention so the install program could grab the header out of the compressed file in order to do the rest of the work, this would cut down space needed for the XML file since a good majority of headers are being repeated several times and just taking up space.
I'm a loner Dottie, a Rebel.