FreeBSD Documentation: An Interview with Tom Rhodes

An Anonymous Coward writes "FreeBSD has been known for excellent documentation and here is a rare sneak peak behind the scenes of the FreeBSD document project with FreeBSD's very own Tom Rhodes."

Comments

Only slightly OT

[numbski]

I have a question for those interested in FreeBSD documentation:

Let's say you have a production environment running FreeBSD 5.x (I know, boo, hiss, only -RELEASE, not -STABLE...blah blah blah), and with the upcoming release of 5.3-STABLE (my understanding anyway), how would you recommend a minimal downtime upgrade?

I have 2 nameservers running the stock Bind8, 2 MX's running stock sendmail. One 'users' box running Sendmail with spamassassin and spamassassin milter, along with apache2 and squirrelmail for webmail.

None of these boxes have the full sources installed, and in the past I've taken the boxes down and done a binary upgrade from CD. Is this the fastest method?

Re:Only slightly OT

[TheLink]

You can build everything on another box, and then copy /usr/obj and /usr/ports/distfiles over and then shutdown to single user and do the installs from there.
See this for background.

There are many ways to do it depending on whether you want it built from source or just want the binaries.

Re:Only slightly OT

[Bishop]

I have always been a big fan of installing fresh on a new machine and copying the data over. This applies to most OSes. This method gives you a chance to test the new release before going into production. And once in production you can always switch back to the older machine if something goes wrong. Not enough people test an upgrade or have a downgrade procedure.

If you don't have a spare server don't be affraid to use an intermediate temporary server. It involves installing the os and copying data twice, but it is not as big a hassle as it sounds. If possible use fresh harddrives saving the old OS and data as a 'warm' backup.

Unfortunately if you are running a colocated server you probably can't do this. My only advice then is start Tuesday morning. Everyone knows not to start an upgrade Friday afternoon, but so many people still do. If you follow the instructions in the FreeBSD handbook your upgrade should be problem free.

Re:Only slightly OT

[JQuick]

It depends on how you measure quick, and on your risk tolerance.

If by quick you mean the least time start to finish, yes. If you mean as measured in system downtime, no. Each has a different risk profile which depends heavily on how much additional software you have installed.

I too have been running 5.x as a server environment since mid 5.0 days. I have performed 2 source based upgrades in the interim to bring me to 5.2. My preference for source based upgrades is based partly on my desire for quick response time re: security. It is also conditioned by my rather complex setup in which I have multiple jailed environments each running a large number of packages. A binary upgrade is less attractive since I would need to install dozens of different ports and possibly face conflicts or temporarily broken ports.

You have very few ports running, and from your statement they are pretty stock configurations. From this standpoint a binary upgrade should be relative painless. However, it might require more downtime.

If I were you and were running a GENERIC kernel, and was running a late 5.1, or 5.2_RELEASE, I would suggest a source base approach. if you are running an earlier 5.x version I would still do so myself but would counsel you to assess your comfort and knowledge with compiling the code and following /usr/src/UPDATING to the letter. If you are unsure, opt for a binary install.

If you do use a source base approach, I would prepare by installing the cvsup tools from the ports tree to mirror the source code and the ports tree. Then you can compile using buildworld and buildkernel, and even compile and install ports (using and alternate paths for the package db and destroot) to test versions of installed ports which might be newer.

Read UPDATING thoroughly and study any differences which you are unsure of. Then when you are ready, use install* targets and mergemaster to finish.

This is initially a longer, more time consuming approach, you must install sources, and configure cvsup to keep them up to date. Once that is done, however, they are always up to date. At each site which I have maintained FreeBSD, I use cvsup to mirror ports and sources on a single box. In fact, I mirror the cvs trees, enabling each host in the network to choose what particular version to check out. I then check out source trees via cvsup, and run a buildworld and a buildkernel via cron either weekly or monthly.

Thus, I always have a recent binary distribution ready to install when I feel like it. I upgrade rarely, but when I do, I typically have a 10-20 minute downtime. On boxes where I have configured multiple drives with sets of boot, usr, and var partitions, I configure and install to the alternate drive using the DESTROOT variable, and can take care of merging changes while running on the old version. Then downtime, is boot time + time to select the new boot partition.

Re:Only slightly OT

[feargal]

I usually upgrade from source. You don't need the current sources to upgrade, just the new ones, so grab them with cvsup.

I do the initial builds offsite and usually well in advance (perhaps leave them to work on a Friday evening).

1. make buildworld
2. make buildkernel

Once onsite, I:
3. make installkernel - takes a few minutes, doesn't count towards downtime.
4. reboot
5. mergemaster -p - takes about a minute
6. make installworld - takes maybe 5 minutes at most
7. mergemaster - this takes the longest - I usually manage it in about ten minutes, as I've become pretty familiar with it, and make the right decision pretty instantly.
8. reboot

I've timed myself, and I end up with 15-20 minutes downtime, depending mostly on the speed of the machine.

Going more off-topic, but I had an idea on how to make this process faster, and to make mergemaster much less scary.

Most of the files that are affected by the mergemaster process are rarely actually changed. On a stock server, you'll probably only ever change files in /etc/mail, /etc/namedb, /etc/ssh, and perhaps one or two other file. All the rest of the files are usually unchanged by the sysadmin. The mergemaster process however asks you if you want to upgrade all the files that have changed in CVS. This takes a lot of time and involves repeating the same keystrokes, and is probably the source of most accidents: shift-G, i, shift-G, i, shift-G, i, "oops, shouldn't have overwritten that one!"

It should be trivial to, pre-upgrade, traverse /etc, extract the version numbers, download their originals from CVS, diff the two, and build an "auto-list" of files which have never been altered.

When mergemaster is run, it can then automagically upgrade all of the files in the auto-list; if nobody saw fit to change /etc/mtree/BSD.usr.dist, chances are the new one will do just fine. Meanwhile, the sysadmin only has to think about that matter to him.

Also, prior to doing the upgrade, he would be able to get a list of files which he *has* changed, so he can figure out what exactly he was thinking when he decided to hack /etc/rc.d/initdiskless to bits.

Any reason this wouldn't/shouldn't work? Obviously mergemaster should give Big Bloody Warnings before using the list. I reckon I'd save at least 25% of my downtime doing this.

Re:Only slightly OT

[cperciva]

My preference for source based upgrades is based partly on my desire for quick response time re: security.

Entirely off-topic, but if you're concerned about security, binary updates are a better option than source patches -- both because FreeBSD Update is more secure than the cvsup mirror system, and because I normally have patches available via FreeBSD Update within a few minutes of the code being committed to CVS and the security advisory going out. (I have the advantage of seeing the source patches in advance, thanks to being on the FreeBSD security team.)

Of course, this only applies to tracking the security branches, but if you're concerned about security that's what you should be doing anyway -- we don't issue security advisories for issues which only affect -current.

I have been impressed...

[brilinux]

As someone who has used multiple Un*x-like OSes, such as FreeBSD, OpenBSD, Gentoo Linux, Debian GNU/Linux (and I am not a zealot for any of them - imagine that!), and others, I have found that if I want to know about saomething or how to do something, FreeBSD has always been the best at having the information availiable. It is very easy to find what I need to know, and everything seems done very logically. Good Job, guys!

For Such A Critical Yet Thankless Job...

[devphaeton]

Thankyou to all the folks that have created the world-class documentation system to go with the world-class OS that is FreeBSD.

*thumbs up*

Re:Tom Rhodes

[Anonymous Coward]

Please ignore this troll. It's the same guy who has been pestering every *bsd (yes, even NetBSD) announcments trying to discredit FreeBSD developers. He usually goes on to say just how FreeBSD devs hate Dragonfly, how they unfairly kicked out Matt, and so on.

Sometimes he links to a message posted by DES on FreeBSD-advocacy in his signiture. If you take the time to see how that thread started, you'll see that the original "quesiont" was quite rude, and follow-up messages from the same person were written in a "I'm a famili member of the former Nigerian royal familiy and want to deposit large sums of money" style. Also if, you follow the thread further, you'll see this reply from a FreeBSD developer:

> BTW, I've spent a lot of time looking at the DragonFly approach, and I met
> with Matt for quite a while at USENIX to talk to him about the approach. I
> have a number of concerns about it -- I think the premise is very
> interesting, but that the results aren't yet there to prove the model. In
> particular, there's a huge volume of code in their system that has not
> been addressed, and a lot of complexity that will need to be handled
> before the SMP primitives they're using have proven that they offer the
> desired performance advantage. We have the opportunity of using a hybrid
> model, and have been exploring some of the ideas present in DFBSD (and,
> one should point out, many other SMP systems).
>
> A lot of other systems have opted to use elements similar to those
> primitives, but in a much more limited way due to the performance costs.
> For example, locking services into particular CPUs prevents the scheduler
> from balancing load between the CPUs in an service-transparent way. In
> the DFBSD model, load balancing must be implemented separately for each
> service, requiring extensive modifications to the services. I.e., the
> model may indeed offer benefits, but the cost of doing the work will be
> high, and the time to complete it long. We'll adopt elements of the
> design as they prove to make sense, as we do with all other open source
> operating systems (and they do with us!).

For your interest, Matt still posts occasionally to -current list, in fact, he even helps out a bit here and there. This troll's problem seems to be with DES, PHK, Bosko, but he is ready to extend his warm words towards anyone, even, it seems, to someone associated with the documentation project. Oh, btw: you'll see the same message by Doug-Furlong Smorgreff on Osnews as well. ~molnarcs

Re:Take a hike

[Anonymous Coward]

This pathetic troll is trying to put FreeBSD and DragonFlyBSD developers against each other. He already spammed the FreeBSD mailing lists, www.osnews.com, and slashdot.
He makes unsubstantiated claims, then asking other people to provide explanations to his trolling bullsh*t statements - a troll technique as old as the internet.
I heavily suspect this is the brainless GNU zealot that has spammed /. BSD section with innumerable FUD-spreading "death" messages - which, OTOH, don't seem to have provided the desired deleterious effect. :-)

If I may, I'd suggest to wait for him to provide proof of the BS he's uttering, and in the meantime, just to leave him in his misery, since every argumentative answer will just (ehm... you know) feed the troll. On the other hand, I think insults are ok. :-)
Thank you for your eventual attention.

All documentation available online and offline

[phoenix_rizzen]

One of the nicer things about the FreeBSD Documentation Project is that everything is available both online and offline. All the man pages for every release of FreeBSD (going all the back to 1.0), along with OpenBSD, NetBSD, and several Linux distros, are available at http://www.freebsd.org/cgi/man.cgi

And, if you selected the docs distribution during the install, you'll find all the articles, books, and papers under /usr/share/doc, including the Handbook and the Porter's Handbook. If you didn't install the docs during the initial install, they can be fetched (and/or updated) using cvsup. There's a samples docs supfile in /usr/share/examples/cvsup. Just be sure to set DOCS_LANG in /etc/make.conf to the language you want, otherwise you'll get every language availables. :)

Having all the documentation available offline is a boon for those days when you break the network. :)