OpenBSD Ports and Packages Explained

jpkunst writes "As reported on undeadly.org: an interesting interview with OpenBSD developer Marc Espie about the internals of and the philosophy behind the OpenBSD ports and packages system."

Comments

The original 'gentoo' ... explained :)

[Gopal.V]

BSD has been where everything seems to find a crowd of whackos to implement stuff. Or that's the only reason I can find about it having source compilation first (the first time I compiled gcc it took me 18 hours.. in 1999).

The real difference between gentoo now and bsd then is the people who use it.

Re:The original 'gentoo' ... explained :)

[DrSkwid]

wtf are you on about ?

they are source code because they need to support 11 architectures and you might want different options to the pre-compiled binaries

nearly *all* the ports have binary packages for your particular source port, if you stick with GENERIC make options

you can even install them from ftp !

# ftp -a ftp.openbsd.org

ftp> cd pub/OpenBSD/$version/packages/$arch
ftp> get gcc-3.3.2.tgz "|pkg_add"

Personally I prefer OpenBSDs ports to FreeBSDs because OpenBSD will create the binary as a package so you can compile with your options once and install the package on different machines without re-compiling on each box

even better is that *some* of the optional parts also have pre-compiled packages
I was rather pleased to find the rc shell pre-compiled with readline support

# ftp -a ftp.openbsd.org

ftp> cd pub/OpenBSD/$version/packages/$arch
ftp> get rc-1.6-readline.tgz "|pkg_add"

how does that makes me a whacko ?

HIBT ?

Re:The original 'gentoo' ... explained :)

[stox]

FreeBSD will be glad to make you a binary package when you build a port, just type "make package".

Re:The original 'gentoo' ... explained :)

[compass46]

OpenBSD does package first then install. FreeBSD does install first then package. The former method is useful when you want to build a package on a faster machine that will not be using the package. For example, building Apache on one machine for another slower machine when the fatser machine will never use Apache. The process using FreeBSD on the faster machine is build, install, package, deinstall.

OpenBSD's ports system is geared primarilly towards being a packaging system for the OpenBSD developers where FreeBSD's is more towards a general purpose installation, management, and packing framework. My personal wish is that FreeBSD would move to a portstree structure and management more like OpenBSD's.

Re:The original 'gentoo' ... explained :)

[Curtman]

BSD has been where everything seems to find a crowd of whackos to implement stuff.

There's a lot of very intelligent people working BSD. I'm a Linux zealot myself, but I really enjoyed reading this interview with Theo de Raadt, and Christos Zoulas. It's very interesting how much they seem to different in what they believe. One of the more striking ones was:

When we find that a change must be made to the system (security or otherwise) we can therefore force such a change into the system by changing it all the way from userland through the libraries down to the kernel. We can change interfaces as we want to. We can move quickly. Sometimes changes are even made which break previous executables; but if we need to, we can choose to make such decisions.

This gives us great flexibility to move forward fast. If something is designed wrong, and the fix depends on changes in more than just the kernel, we can fix it by. We change all the required pieces in the right places. We don't need hacks in the wrong place to fix a problem.

-- Theo de Raadt

-vs-

Linux's code is much newer and it keeps constantly being re-factored. This has the nice side effect of keeping the code simple and readable (at the base system layers such as VM and FS), but stability is suffering.

-- Christos Zoulas

It's hard to imagine that they are even talking about the same things.

Re:The original 'gentoo' ... explained :)

[theapodan]

Forgive me if I'm wrong, but cant you "make package" without installing after a "make"?

Actually, make package may hit the install target at some point, still necessitating a make deinstall, but at any rate, I don't think that you need "make install", I think "make ; make package" works fine.

Re:The original 'gentoo' ... explained :)

[Homology]

Forgive me if I'm wrong, but cant you "make package" without installing after a "make"?

OpenBSD installs into a "fake root" (you need root privs for this), and makes a package based upon this.

Re:The original 'gentoo' ... explained :)

[theapodan]

Sorry not to be clear, I was talking about the FreeBSD system.

Just out of curiosity, can you not change the directory prefix when you "make" in order to install the application in /tmp, then "make package" from there? I was adressing the OP who was talking about having to deinstall software when making packages he did not want on the building computer.

Wouldn't this avoid the issue of actually installing the software on FreeBSD?

Re:The original 'gentoo' ... explained :)

[Anonymous Coward]

Please excuse the typo's. I was in a rush. ;)

Re:The original 'gentoo' ... explained :)

[compass46]

Hopefully yes... But some ports (including some of my own, see the link, and yes it's on my TODO list) are not prefix safe. Files get lost and occassionally ports fail to build because of other lameness. Setting PREFIX _should_ work but doesn't always.

Re:The original 'gentoo' ... explained :)

[turbidostato]

"When we find that a change must be made to the system (security or otherwise) we can therefore force such a change into the system by changing it all the way from userland through the libraries down to the kernel."

The thing being this is untrue or not relevant (regarding its Linux counterparts).

Any given Linux distribution can do that on whatever codebase they regard as "official", from the kernel up to the highest userlevel app just in the same as any *BSD can.

"Linux's code is much newer and it keeps constantly being re-factored [...] but stability is suffering"

vs

"we can therefore force such a change into the system by changing it all the way from userland through the libraries down to the kernel. We can change interfaces as we want to. We can move quickly. Sometimes changes are even made which break previous executables"

I can't manage to see any difference.

Re:The original 'gentoo' ... explained :)

[Curtman]

"I can't manage to see any difference."

That's just it.. Christos is saying

Linux's code is much newer [than BSD] and it keeps constantly being re-factored [...] but stability is suffering [in relation to BSD]

and Theo is saying

We can change interfaces as we want to. We can move quickly [in relation to Linux]. Sometimes changes are even made which break previous executables [IE: sacrifice stability for flexibility, AKA re-factoring]

The point of the interview was to ask them questions about how BSD differs from Linux, and how they cooperate. It was a followup to similar questions asked of Linus. They just seem to have a very different view of how their projects relate to Linux. One seems to be a bit more political, if that's the right word.

Re:The original 'gentoo' ... explained :)

[Anonymous Coward]

I think you've both missed the boat on this one. Development of a Linux kernel vs. the BSD system is different in exactly those regards which I've emphasized in italics.

Theo et. al. have complete control of the kernel all the way down to the user-land. In Linux, however, userland tools are GNU (hence the rage over Linux and GNU/Linux naming) and maintained closely, but separately. That means changes have to be rippled rather than made at once, atomically.

The effect is that it is more likely you will see a shipped distro with broken tools due to the constant changes, or that you can become inconsistent with kernel version and tool version. In the recent 2.4 -> 2.6 changeover there were several kernel/user tool packages that changed drastically, making it very difficult to maintain a system with both kernels.

Also, don't be fooled by the word "refactoring." A refactoring is a change of interface or architecture, and by definition breaks an interface contract - whether that is a public or private contract only influences the scope of the breakage. Theo talks a lot about making and rolling out comprehensive fixes quickly, and describes refactoring as doable "when necessary." The Linux quote, OTOH, describes constant refactoring as a nuisance and a hindrance to Linux stability.

I believe this is the point the OP may be trying to make with these two quotes.

Re:The original 'gentoo' ... explained :)

[Curtman]

"Theo et. al. have complete control of the kernel all the way down to the user-land. In Linux, however, userland tools are GNU"

I am aware of that. To a point. BSD machines, very often contain a lot of GNU software as well in userland.

"A refactoring is a change of interface or architecture, and by definition breaks an interface contract"

And 'stable' is not subject to sudden or extreme change or fluctuation and consistently dependable. When Theo says 'We can change interfaces as we want to. We can move quickly', he's saying it is a good thing that they can do exactly what Christos is accusing Linux of doing.

Theo is saying that if they change the kernel interface, then they can change libc as well basically. When Linux changes their API, glibc changes as well. The fact that it's one group and not two intermingled groups (RedHat maintains glibc, and is a Linux distributor, and contributor for example), doesn't mean much as far as stability is concerned. The difference is mostly political as far as I'm concerned.

Re:The original 'gentoo' ... explained :)

[turbidostato]

"I think you've both missed the boat on this one. Development of a Linux kernel vs. the BSD system is different"

No, I bet *you* missed my boat. I was not talking about Linux kernel but about *Linux systems*.

"Theo et. al. have complete control of the kernel all the way down to the user-land"

Yes, they have.

"In Linux, however, userland tools are GNU"

No, they aren't. Not to the point of interest here. They are open source released under the GPL, hence Debian people (to name one *Linux system*) have exactly the same control level about their code than Theo et al. about theirs. No more, no less.

"The effect is that it is more likely you will see a shipped distro with broken tools due to the constant changes"

That strongly depends on the distribution itself. Linux distributions are free to put together a half-assed bunch of packages from different sources OR follow whatever process to insure the quality of their "official" releases, including extensive modifications from the "upstream" codebase exactly the same *BSDs are.

"making it very difficult to maintain a system with both kernels."

Thus, some decided to support both of them and others only one branch.

"Theo [...] describes refactoring as doable "when necessary [...] The Linux quote, OTOH, describes constant refactoring as a nuisance and a hindrance to Linux stability."

So? Theo refactors when he feels it is needed. Any given distribution refactors when feels it is needed.

Still I can't manage to see any *real* difference (yes I know what BSD folks *feel* to be the diference, cathedral vs bazaar, and all that stuff, but still, except from the preference for BSD-like licenses vs GPL ones, I can't see any procedural difference between any given BSD project and any given Linux one: in both cases we are talking about open source and in both cases the teams involved are in absolute control about what does go in their "official" codebase and what doesn't).

Re:The original 'gentoo' ... explained :)

[tek.net-ium]

Any refactoring would be hopelessly limited. The GP was discussing an API change that would require a simultaneous change in both glibc and the kernel. This is unlikely in Linux for a few reasons:

1. Most real world environments upgrade kernels and other packages, such as glibc, at different times. With BSD, you could make such a change across releases, but with something like Debian, you would need to make such a transition smooth.
2. You would abandon any users who had vanilla kernels, since it would require a special patch, something most users aren't willing to do.
3. It's unlikely to even happen in the first place, since, at least in my experience, most packagers don't have developer-level knowledge of the package.

Such an API change is possible, yes, in theory but hopelessly limited in practice.

Re:The original 'gentoo' ... explained :)

[turbidostato]

"Most real world environments upgrade kernels and other packages, such as glibc, at different times."

No, they don't.

"With BSD, you could make such a change across releases"

Correct.

"but with something like Debian, you would need to make such a transition smooth."

No. You "could make such a change across releases". There's no change Theo can't overtake from, say OpenBSD 3.7 to 3.8, Debian can't take from 3.0 to 3.1. I miss to see how changes from Woody to Sarge are any different from those between two OpenBSD releases.

It *useful* transitions between Woody and Sarge are smooth, because no Debian user wants to reinstall just to grab the lastest Stable, as well as it is usefull transitions between OpenBSD versions being smooth, since I can't imagine OpenBSD users really like reinstalling their boxes from time to time just for them to "act the same".

"You would abandon any users who had vanilla kernels, since it would require a special patch, something most users aren't willing to do."

What do you mean? OpenBSD users can use vanilla Linux kernels? OpenBSD can use OpenBSD kernels. Debian users can use Debian kernels AND Linus' vanilla kernels. Even if Debian users could use Linus' vanilla kernels nomore they wouldn't be any worse than any OpenBSD user, since they already can't use anything but the OpenBSD kernel.

"It's unlikely to even happen in the first place, since, at least in my experience, most packagers don't have developer-level knowledge of the package."

That again depend greatly from distribution to distribution. Are OpenBSD packagers reknowed developers per chance?

"Such an API change is possible, yes, in theory but hopelessly limited in practice."

It is not "hopelessly limited"; it is limited because it pays great dividends. Having something mostly compatible with that of your neighbour makes easier to share the development load. It is not that Debian, to stay with the example, can't "break laces" with any other Linux distribution in the world, it really can, it is only it doesn't really make sense. If it would do, it would be as hopelessly isolated from the vibrating Linux community no less (but no more, either) than the different *BSD communities are currently isolated among each other, and thus its ability to affront future would be greatly dimished.

yay for ports!

[Leroy_Brown242]

I'm in love with ports. It just makes things so damned easy.

I'd venture to say that without ports on FreeBSD, I'd never have learned so much.

I've always thought that with a little more polishing, it could be good enough for even my mom to use.

Eased our transition from Debian to NetBSD!

[CyricZ]

We recently transitioned some of our smaller servers from Debian to NetBSD. Indeed, the BSD packages and ports system is fantastic. It allowed us to get our new servers up and running in no time. Kudos to all who have perfected the BSD ports and packages system.

Re:Eased our transition from Debian to NetBSD!

[BerntB]

[...] from Debian to NetBSD. Indeed, the BSD packages and ports system is fantastic.

Uh? Are you saying it is neater than Debian with apt-get?!

Not a troll, but sure... mod me down if you think so. (Just wait until someone answered, plz! :-)

Re:Eased our transition from Debian to NetBSD!

[sp0rk173]

having used apt-get and ports extensively, yes. The various BSD ports are better than apt-get. However, pacman (in arch linux) beats them all to hell. Simple, clean, and elegant.

Another Newb

[mugnyte]

Like some ./ readers, I got the OpenBSD image when 3.7 was released just this year. I dropped it onto an older box and have been exploring ever since. I used the packages for several items (like Gnome) but ports for others (Postgresql 8).

I've been very happy with several concepts:

- The dependency trees are spot-on and very automated. Correct versions and complete coverage.
- The ability to undo or rollback a package is smooth (like when I took a 7.x Postgres package and pkg_delete'd it to try the port)
- The published docs, man pages and organization of the system is superb. I picked up "Absolute OpenBSD" and "BSD Hacks" and have been toured confidently around the system by these and the man pages they point to.
- The post-install notes are a great help.

For me, it's a great "warm and fuzzy" to gather the documentation sources into a list and be able to dive down rabbit holes for long periods without feeling like a flea market is on my box. Cheers to the BSD folks, especially the package maintainers.

Great Work!

[Anonymous Coward]

I loved the Interview, and recall Espie's comment from maybe a kerneltrap interview: Working on OpenBSD has made me a better day job worker.

now all I wish for is a command that would list the packages sorted by dependencies, so if I wanted to see what is impacted by removing one package and adding another, I could easily see what woudl be affected.

Maybe it is already there, finding out is just a man page away.

Why Perl?

[marcovje]

I hate having to have install Perl, Ruby and Python (m4?) for all kinds of trivial stuff. (or have it in the base systems as more and more systems do).

First, these scripting languages are slow. People that don't agree, please compile X, and watch the font making in Perl. It eats more time than actually compiling X

Second, every stupid package uses another one, so you end up with a bunch of them.

Third, it requires more languages to learn to access the system. Moreover, the amount of languages that can interface to Perl modules is significantly lower than the amount that can interface to C (virtually all). This can also be seen in the article: the perl backend is unused, everybody works around in shellscript.

The first two could be solved by e.g. limiting the tools to a perl subset that is compilable to C, and not use too many modules.

The third is harder to solve. Best would be to code the package system in C, and have a C callable library. Or maybe using 2C kind of stuff.

Re:Why Perl?

[Anonymous Coward]

A good reason for perl is that it *works*.

A second good reason is that it's already part of the OpenBSD base install.

A third good reason is that the old package system was in C, and was downright broken.

A fourth good reason is that the current perl system is over 200K of perl code. Try rewriting that in C, and see how big it gets.

Finally, maybe perl isn't that slow... I've rewritten all kind of C/shell code in perl, and it consistently is faster than the old C code. Why ? because I can use smarter algorithms, and better caching. The package system of OpenBSD is another example of the same. Dude, the perl version of pkg_add is about twice as fast as the old perl version.

As far as X font building goes, you can't really know if it's slow because perl is, or if it's slow because there's a large amount of stuff going on. Look at the sizes involved, look how much data is processed, realize that usually, everything gets compressed with gzip.

Now, you can think all you want that it's slow because perl is slow. Well, I've got at least two other reasons for it to be slow.

What you have is an opinion, and not really well informed. How about backing it with some actual facts ?

Re:Why Perl?

[Anonymous Coward]

Use the right tool for the job. If you think that it is slow, stop and think beyond, not only about speed. Perl, python, ruby and so on, they all help reduce complexity, costs and time needed to develop software.

Computing power are getting bigger every day, and cheaper too, so the speed penalties are not that bad.

These languages all help you to express your algorithms easier than with C.

Try to write text processing programs with C, then try to write the same program using Perl, compare it. I bet youll find perl much easier to program on.

however, if it you really want to, you can write your own tools, using your language of choice ;)

Re:Why Perl?

[marcovje]

"The right tool " mantra works both ways. Both from the developer perspective and from user perspective.

The dependancies (install a few apps, and you have 100MB on various script languages, and extra modules) and speed are the trouble.

I'm talking about slow from the viewpoint of a recent Athlon64. Even if it gets twice as fast in the two next years, it is still annoyingly slow.

I have had the trouble to be forced to program in Perl, to fix some predecessors code. My conclusion was that while it was nice to get something under 1000 lines working quickly, IMHO the development speed went down for anything non-trivial, often to below even C speed. C, though a bit too spartan for my taste, is at least debuggable and predicatble.

Note that I'm not a C advocatists. I have other preferences. But they are not viable, partially because of the same problems as I don't deem the
scripting languages viable: dependancies.

Another thing that might play for me is being *BSD user, doing a lot of source building (manual or ports). One get the builddepends as well as the rundepends.

C based packaging system for openbsd

[Triumph The Insult C]

is just a cvs diff away. where are your diffs?