Slashdot Mirror
Jordan Hubbard On Next-Generation Packaging
GlobalEcho writes: "Developers associated with Darwin are beginning to think about package management and source building. At issue is whether something like dpkg, RPM or *BSD's ports could suffice, or whether they are all just way too mid-90's. Jordan Hubbard himself (now of Apple) weighed in with his opinions (user and passwd 'archives'). Apparently he thinks it is time for something more advanced, and he gives some ideas about what that might look like. Does anyone else have good ideas?"
If I'm not mistaken the whole Ports thing was one of Jordans great inventions. It's succeeded quite well using standard distributed tools (ie. makefiles, compilers, and the like).
Perhaps I'm wrong. Nice to see he's still having great thoughts. Hope whatever packaging system they come up with is portable enough to work on a large chunk of systems (linux in various configs, bsd's, solaris, darwin, etc.).
Rod Taylor
Quite frankly, dpkg isn't all that bad. It has MAJOR issues; no dobut about that, but has many great concepts which can't be found anywhere else (correct me if i'm wrong. they're still good ideas, though!)
The dependency and dependency resolution system- dpkg has the most advanced dependency system known to unix. No dobut to that... To solve these dependencies, dpkg goes to it's list of package locations (complete with http and ftp locations, cdroms, etc.. if necessary) and grabs the required packages from the net (the user is prompted on this, of course)
--Easy upgrades. No other system allows me to bring my system up to date in less time (note: debian isn't updated often, so this is generally unappreciated)
$apt-get update
$apt-get upgrade
(hit y to confirm)
All from the command prompt.
I'm not sure what else there is that makes it good. But RPM certainly doesn't have these features.
What it lacks:
It's buggy as hell - it's easier then signing up for aol to nuke your system this way (in other words, it happens quite often by accident)
No good front-ends - There is no good program to browse available packages, install them, enter configuration information (more on that in a sec) and remove them. You should enter the package you want to install. a wizard is displayed, it grabs the package from a mirror or local source, solves dependecies, installs it and any dependent packages, configures it, and exits.
Configuration - dpkg has a system that allows the package to prompt for a few options before it is installed. this is a good thing, but the packages usually don't ask enough. users need full customization (nothing nitpicky. big stuff... so you dont have do manually edit configuration files by hand.
Available packages - this is where dpkg falls flat on it's face. 95% of unix packages are rpms. that never helps. a unified packaging system needs to be put into place
i dunno what i forgot?
-- If you try to fail and succeed, which have you done? - Uli's moose
Doesnt OS X only run on G[3,4,5] processors any way?
Wasnt the hole point of fere bsd ports section so that i could have a version optimized for my arch?
If so wouldnt binary versions suffice eg apt?
i like apt but would love to have th eoption to have stuff build for my box ony but dont see the point for a platform where the processors are more similar. Or am i missing out on something?
Is a bin optimized a G3 different from G4 ?
But in practice I just don't see how it's gonna be any different in the long wrong. I mean with the xml you'll be able to do alot more stuff, a simple database etc it starts to get a little bit big for a porting system. How long until this becomes obsolete because the database is too big to search effectively.. I like it but i just think that it's implementation is gonna be the hard part. Then again implementation is always the hard part.
the damn platform needs something! The Updater should be called Crasher. The only time my macnix crashes is during an Update.
Yes when it reboots the update works but still.
This
This is great on a open system that can be changed by anyone at any time, and if someone screws up and a feature get ruined, it can be easily changed before it is really catastrophic.
On a production OS that will be used by people who don't know what a CPU is, and can't tell
He doesn't really say this, he suggests modifications for his protocol, including:
Meta data in xml files instead of the makefile
Front-end development facilitation
Use a database to keep the data to keep the descriptions to make it easier to use
All of these are manageable to implement seperately on FreeBSD or wherever before putting them into a new OS, where a screwup would be a huge disaster.
I'm a concientious
why not package the source for most apps that come out & having them build as part of the install??
RPM's are basically a script at teh start of an archive taht tells rpm where to put the binaries.
then ya go get teh source RPM.. balh blah...
why not jsut put it out to one package, and have teh local compiler build the executables before it installs em?? autoconf works great most of the time, and if they got super keen about it, some gui or menu to custom tweak it wouldnt be too difficult an extension.
sure, some people dont install a compiler or other utilities like make or autoconf, but unless they have to seperate production from development environments that way, they probably wont be installing many apps anyway, right??
it'd make a lot more sense to 'mould' each app to teh architecture rather tahn have one package for x86, another for ppc, another for alpha by putting it into one package and have it 'melt' into the optimal configuration.
just some thoughts...
It's buggy as hell - it's easier then signing up for aol to nuke your system this way (in other words, it happens quite often by accident)
Huh? I suspect user error. I've been using Debian/dpkg since pre 1.0 days, and I can count the number of times dpkg has had system wrecking errors on one hand. I can count the number of times that it actually wrecked my system on one finger -- after that, I got a little more cautious about upgrading dpkg in the unstable tree. (i.e. wait a few hours and read debian-devel), There are ways to tell dpkg to hose your system, but those aren't bugs, those are options with big nasty warnings next to them.
Now, there have been many more occurences of buggy packages screwing things up, but that's hardly dpkg's fault. And if you live on unstable, well, that's what you get.
No good front-ends -
apt-get install aptitude
(Not in stable, but coming soon[1] to a release near you.
Configuration - dpkg has a system that allows the package to prompt for a few options before it is installed. this is a good thing, but the packages usually don't ask enough.
Again, not a dpkg issue. If the package doesn't provide sufficient configuration flexibility, it's an issue with the particular package.
Available packages - this is where dpkg falls flat on it's face. 95% of unix packages are rpms. that never helps. a unified packaging system needs to be put into place
I don't know where you got that statistic. Yes, maybe 95% of packages you see floating around random websites are in rpm, but I doubt that 20 times as many software packages available as rpms vs. debs. Most upstream developers don't provide debs, because there's a debian developer to do so for them; the fact that mozilla.org has rpms but not debs doesn't mean there aren't .debs of Mozilla. (I'll allow that the ratio for non-free software is much worse, for fairly obvious reasons.)
Steve
[1] "soon" in Debian terms, at least :-)
We use a simple XML file-based (i.e., you can edit everything with vi) object-oriented database. The project isn't just about package management, but we implemented a full multi-platform build-from-source-and-install-sitewide package management tool. It also handles dependencies etc.
What I would hate to see are any major revisions if it's just gonna add some feature; I would rather see that time spent on developing the ports and packages themselves. Make is a good, simple, foundational and almost always present solution. Adding other languages would be a waste of time IMHO.
Let me condense what I think should be pursued from the ports perspective: documentation and ease of use. One can always make readmes and get mini-descirptions, but that really should be expanded upon, both for beginners and seasoned users who just don't know what that software is about. It would be nice to have some options like, info that would go thought the ports tree and build more verbose information. If those documents are built in a consistant manner (such as xml), then any ol' front end can be built to pull the info on the port and automate building the port and the flavors available. For example, a simple curses interface that will inform you of the dependancies that will need to be built first, estimates the size, and gives you a list of flavors to add into your build. Hit ok and it monitors the progress for you, logs the process and keeps the messages out of sight (from those who get scared easy).
I agree that something should be done to be able to automagically build a package from a port. I think this area would be the best to pursue. Even better, if we bsd types could get a system like checkinstall /installwatch consistantly, not most of the time, but consistantly working on BSD. This project essentially is a wrapper script that records everything make install does. In current form, it gives you the option of building an RPM from that make install. What should be pursued is making this work -well- on bsd, with the option to build a package along with documenting it's dependancies and/or recording the install info into the existing system to that all one has to do to remove what you just built is 'pkg_delete'. THAT would be cool!!
Democrats and Republicans only disagree about how to enslave you
What really happened is the slashdot folk noticed 150+ posts on some of the Macslash threads a couple of weeks ago.
cat
None of these ideas are new. We at the OpenPackages project have already discussed these ideas and more. We have a pretty solid plan, but unfortunately no one seems to have the time to turn the proposals into a design document and get going. Like Jordan said in his post and i said in my follow-up, time is the critical issue. This is a big job.
t p://openpackages.org/pipermail/op-tech/2001-Apr il/000764.html/ op-tech/2001-May /000826.htmlp -tech/2001-Dec ember/001454.html
Here are some references i included in my darwin-devel post:
http://openpackages.org/html/pkg_design.php
ht
http://openpackages.org/pipermail
http://openpackages.org/pipermail/o
Check out the fink project
http://fink.sourceforge.net/
600+ OS X ports so far, automatic updates,
database indexing, built on top of dpkg.
The link to the article seems to require some sort of authentication. Anyone have an "authentication-free" link?
Gentoo Linux is aiming to create a much enhanced collection system.
Many unix users seem to think existing solutions are great. You know, I've taken to Cocoa since I got my Mac last month, but there is one thing that drives me crazy and thats figuring out this freaking package manager. Its not easy for people use to using Windows Setup or other installation packages for Windows development tools.
The problem is UNIX isn't designed for the average user. When you look at it from a Unix perspective, it works great. When you look at the rest of the world, it doesnt. And anything thats a single platform does not create a big issue. You won't see it used on all your open-source console apps, mainly MacOS X applications. This is definately needed in Mac OS X.
www.atacomm.com - The Leader in VoIP Product Distributi
The dependency and dependency resolution system- dpkg has the most advanced dependency system known to unix. No dobut to that... To solve these dependencies, dpkg goes to it's list of package locations (complete with http and ftp locations, cdroms, etc.. if necessary) and grabs the required packages from the net (the user is prompted on this, of course)
Dpkg doesn't do that. APT does. Its not that these aren't great and massively useful features, its just that just like urpmi, APT works its magic across RPM (the LSB standard packaging system, which many app packagers primarily package for) too. I maintain an apt repository for my workplace with around 3000 packages, all in RPM format for Red Hat 7.2, and it works like a charm. Debian's policies are postable too - Connectiva has a similar set of guidelines. The unique advantages of Dpkg is suggested / recommended dependencies, something RPM desperately needs (Red Hat themselves cheat and use the `comps' file to provide this logic themselves in the installer, but us users don't have the luxury). RPM has some unique advantages too tho - transaction handling in the database (thanks to DB3) does wonders for my piece of mind. In the end, I'll stick with RPM and Apt-get because of the LSB stuff, and avaliability, but I do hope they add suggested / recommended dependencies soon.
Actually, if you want to see a system which kicks both their arse in many ways, look at QNX. They have apt-like features, a nifty `package filesystem', and a GUI installer that reallycraps all over every other software install system I've ever seen.
Jordan, I've been using FreeBSD since about '94. I can't really remember it being without ports. It shouldn't be to hard for a hacker of your class to make a GUI front end for it. Or at least some of the Apple hackers. Hell, I wish I could, but I've given up on anything more complex than a PHP/database driven website. All I know is, the FreeBSD style ports system has saved my ass many, many times. Thanks for all the cool stuff!
--- Think of it as evolution in action ---
Hurbbard states :) for any number of potentially
"1. All the descriptive text which current goes into a Makefile like the
port's name, version, dependencies, URL for source bits, and so on,
should go into an XML file. This immediately allows the port to be
indexed, documented and modified from automated tools which traverse the
"ports collection" (sorry, I have to keep using that terminology since
it's what I'm most familiar with
interesting reasons. The reason a lot of these tools don't exist for
*BSD today is that extracting data from Makefiles generally sucks from a
parsing perspective so people aren't encouraged to get too creative."
To reprase.
Metadata should be in XML, this makes it much easier to process than if it has to be extracted from Makefiles
But why does it have to be XML (or SGML or RDF for that matter). The only reason i can think of is that there are pre-existing tools to parse it, the tradeoff is that any Markup language will bloat your metadata and make it unreadable.
Shouldnt a packaging system be important enough to warrant the development of its own parsing routines. i.e. just design it to be the best it can be
I know that the bundles used for the upper layers of MacOS rely on the Cocoa (NeXTStep) libraries, but couldn't something similar be done for Darwin? At least that way packages would be compatible between teh Desktop and CLI versions of the OS.
Is http://www.openpackages.org/ dead? why did it die?
I have been using Gentoo Linux for a while now... It includes a package management system called Portage. Portage is basically a reimplementation of the BSD Ports system. Its goal is to become much more powerfull than ports ever was.
http://www.gentoo.org
I have to say that portage is by far the easist package management system I have used so far.
One nice thing about gentoo btw, is that the entire distrib builds it self when you install, downloading the latest packages from the net... So your installation is always current.
Its also incredibly easy to customize.
You start off with a very basic system... Then say you want kde? you would type "emerge kdebase/kde"
and portage will take care of downloading, compiling and installing everything you need (inc xwindows) to have a working kde desktop.
ps: Anyone else notice that slashdot is now 'apple' candy like? LOL ERaa.. when did that happen?
Luke
You can wail on about this average user but you must be careful about this cliche. Because the implicit fallacies is that there is an average user of these systems, that they have far less experience than you or I do, and that they aren't already happy with what they are using now. If there is anything more certain that can be said at all about these average computer users is that they probably don't want to change operating systems right now. In fact, not only would there quite possible be no reason for them to switch operating systems right now but it would mean erasing the skills they have learned using the current operating system. Its no coincidence that those who do migrate to other platforms have little to loose. If you think that these so-called average users spend all their time surfing web pages and sending email then there would probably be a lot more migrators to other operating systems.
I think it is useful to consider a few things I learned in a Cultural Geography class last semester. I know these things are pretty much common sense but I think its not only useful to consider these ideas but to introduce some new terms when dealing with these things (rather than using impoverishments like average user). When people migrate from one region of a country to another, they do so for a number of reasons. There are push factors and pull factors.
So why would someone move to a Free operating system? It seems that freedom itself isn't much of pull factor (but this would change, surely, once many of these software laws and licenses are really enforced against end users, not just distributors).
Let me say something about ease-of-use. While it would seem to be an obvious pull factor--the days of easy to use general-purpose operating systems are long over, I think. Perhaps the first Macintoshes were among the easiest systems to use and the reason for this is quite simple. The needs and expectations of users have gone up quite a bit since then. While I have never used these early computers nor do I know the intentions of the Apple staff (these things are probably clearly documented somewhere...I'm too lazy to look right now), I would suspect that they were trying to make as easy as possible to type out documents with relatively sophisticated typessetting (compared to typewriters!) and then to file these documents into a filing system.
Today's systems are expected to require quite a bit more. Many of the posters here on slashdot carrying-on on what these operating systems need to be successful (in what ever definition of success, most do not say) give examples:
The paradox is that this average user needs all of this. This seems extremely unfair to anyone trying to implement an operating system...nowadays to any team of programmers or consortium of developers contributing to an existing free software project.
Now lets consider what relevent about talking about the average user. Like I said before, I doubt this user would switch his or her operating system for any reason. This is because for everyday this user masters his/her OS, the push factor from every other OS becomes stronger and stronger. Unless there exists a push factor from the OS he is currently using, he's gonna stay.
So lets forget this average user since it isn't relevent or even interesting. Lets consider, instead, a different class of users. Lets just create a class of users who might or will definitely switch to another operating system. Now, awaiting to be smacked around with a stack of statistics proclaiming otherwise, I would guess that this set of users would have the following things more or less in common:
And where would you find this average set of OS migrators? Probably on the internet: in newsgroups and web forums. Specifically, you would find that many of these people read and post to slashdot regularly.
And thats the point to this entire post. I find it interesting to hear slashdotters condemn the intelligence of the average users, how they can't program, or they can't figure out the command line. This might be true, but they are revealing their own experiences more than anything else. They are their own breed of software users.
In conclusion, You Are the Average User.
This whole thing is based on the fact that someone got another package managment system (netbsd's) to run on darwin in addition to having freebsd ports and dpkg already going. So people started talking about how what really needs to be done is a complete new package management system.
But nobody plans on doing anything. This wasn't even started by Apple employees. This isn't going to start Apple's new packaging system. It's about creating one for all systems, much as Open Packages was supposed to. Only this is all talk, not action. There is nothing that is going to come from it.
And finally this in no way relates to Mac OS X. Apple already has it's on little installation program format, and it's most likely not going to change. It's great that slashdot started a Apple section, but at least post real news.
There is one bit in your post I thought about too for quite some time:
A GUI FrontEnd for the ports collection etc. would be great ! Especially for the folks coming from point-and-click-MacOS 9.
Having used Fink for a while now, I have to admit that it is indeed pretty cool, but I can tell you that a *lot* of people I know that are migrating from MacOS 9 to X right now won't touch anything like it at least for years.
Then again, they may not be exactly *the* target audience, but hey, more users is always a nice thing.
So when Jordan and his "helping hands" are building the next-generation ports collection basically from scratch, it would be nice to consider an optional graphical FronEnd for it too !
Jordan said "I think 10,000 entries is going to be something of a stretch even for the FreeBSD ports team, but I don't see that number being entirely improbable for some Macintosh equivalent since there are a lot more Macfolk than there are FreeBSD users." This comment tickles at a core problem with the transition from OS9 to OSX for many Mac users. There may be more Macfolk than FreeBSD users, but how many of them want to rethink package utilities? A lot of Jordan's comments are good suggestions (for instance, implementing the new standard of XML into the package process), but it seems that this problem will be tackled by *nix users like himself moving over to the OSX platform, and not by OS9 users delving into *nix for their first time. Still, it IS a good time to rethink things before momentum makes it difficult to change. I also agree with him that Apple should have spearheaded the process, and they probably will in time. Apple has always focused on userfriendlyness first, then streamlined what was under the hood last. Compared to the past, Apple seems to be responsibly balancing the development of OSX across the board. However, if OSX is to succeed with the typical Mac audience, it will need to be a lot more stable and userfriendly to the simplest Mac user. I can't imagine my parents enabling their root account to reconfigure folder permissions as I had to do recently. Heck, my dad can barely figure out his email. LOL, nevermind launch a shell and interface via CLI. I imagine streamlining the package utility mechanism is low on Apple's priorities. Still, it's comforting that Jordan is mulling the problem over.
The Splintered Mind - Overcoming
Good package management is really going to help in competing with the status quo (ahem...Microsoft) and helping open source projects gain mainstream acceptance. If installing software from packages were as easy as everything else on the Mac, then the advantages of the system would sell itself and gain many Windows converts.
Open source software continues to improve by leaps and bounds. Unfortunately, the uninitiated tend to be terribly confused about how to get it, install it (including installing dependencies etc), update and maintain it. I think the problem stems from a few things including:
- The somewhat archane nature of open source software and unix...keeping out those who aren't in on the joke so-to-speak.
And, more importantly,
- Competing package management systems
I think competition is a great thing. I think it is far to late to advocate the abandoning of the existing ports collection, RPM, deb, etc systems because they are entrenched. But the end result is that a project must have someone who does packaging for all these systems if they want full exposure.
This is why I think we need something like a package repository run by volunteers who create, maintain and store the packages. If it were a single system (distrbuted across many servers around the world of course) then you could have a single, standard interface for searching and downloading packages. This way if someone knows how to use the system on Red Hat, they'll know how to use it on OS X.
Eventually, a dominant package format may emerge. Hubbard has some good suggestions (XML descriptions is a no brainer). Ultimately, the ease of use of this stuff is key. You want developers to be able to serve all package communities with as little effort as possible, and you want the users to have the most comprehensive access to packages available.
Once this happens, people buying Macs or installing Linux will have a huge advantage over their Windows mired breathren because once they log in they'll be a couple of clicks away from thousands of ever evolving applications. While Windows users will be thousands of dollars away from a couple of applications.
there's an XML parser built in to OS X (in Core Foundation).
XML is also pretty easy to work out for anyone else who comes along later and wants to get at the data, you feed it to the parser and end up with a nice structured set of data that you can walk through, it often enforces quite a strict and readable structure on the data too, which is verified as it's read in, so you know that it's a valid structure when you get it.
Unreadable as opposed to what? a binary format??
I, personally, think the FreeBSD Packages/Port system is damn near perfect from a user/sysadmin standpoint.
I've tried fink, and it seem terribly unintuitive and clunky to me, and I don't really use it, prefering to get packages from macosx.forked.net, because they just install, and I don't have to screw with funky menus that don't work simply, and I don't have to hunt for fink packages that aren't in my menu, even though I grabbed the latest list.
Sure, I could spend the time to figure fink out and get it working properly. Hell, I could be using netcat to write this comment, but I'm not going to.
People who actually need to get stuff done, need intuitive and simple tools. Fink and the like are fine for home hackers/users, but when you need to get stuff done, fink doesn't cut it, I'm sorry.
FreeBSD Ports/Packages is very simple, and easy to use, and gets the job done well. It's probably the main reason I use FreeBSD servers (aside from the insane levels of stability).
I can understand Jordan wanting to move above and beyond, I just ask that he keep things simple and intuitive. Something I can get around in with a small, half-page cheat sheet.
I think that a moderna packaging system should have some kind of subpackages. So If you have a package for GIMP for example, you should be able to choose which localizations you want to install for example.
Basically some parts of the package should be optional.
Of course this could be extended to binaries as well. So you can have a single package for several platforms, the package installer chooses the binary for the correct platform.
Erik Dalén
I think for a next generation packaging system Apple ought to adopt concepts from their frameworks method of binary packaging. A package would contain an XML header relaying to the installation program information like the file contents, checksums, a list of required and a list of optional packages, install and make instructions (including file destinations), a hash of the compressed file, and a dependancy list. The installer program would scan the header and ask the user which parts they wanted to install or install options based on arguments like installprogram -a packagename would install the entire package or some such. Since there's a hash of the compressed file it could be checked against a hash stored on the server the package came from for confirmation, from there the installer would figure out using information of the header whether it had to just cp and chmod some files or if it had to compile them from source. Packages could come in source only or binary only or in combination packages containing both. The frameworks the installation fromworks would output to (or just plain directories) could contain a header telling which files exist on the drive and a master lister of installations could be kept by the installation program. So you could whip it open and remove GNOME lets say and it would consult a relatively small XML file telling it where the GNOME XML resource file is and then procede to uninstall the files listed there. The XML header for the package could be even compressed inside the file and follow a standard naming convention so the install program could grab the header out of the compressed file in order to do the rest of the work, this would cut down space needed for the XML file since a good majority of headers are being repeated several times and just taking up space.
I'm a loner Dottie, a Rebel.