FreeBSD Documentation: An Interview with Tom Rhodes

An Anonymous Coward writes "FreeBSD has been known for excellent documentation and here is a rare sneak peak behind the scenes of the FreeBSD document project with FreeBSD's very own Tom Rhodes."

Comments

Only slightly OT

[numbski]

I have a question for those interested in FreeBSD documentation:

Let's say you have a production environment running FreeBSD 5.x (I know, boo, hiss, only -RELEASE, not -STABLE...blah blah blah), and with the upcoming release of 5.3-STABLE (my understanding anyway), how would you recommend a minimal downtime upgrade?

I have 2 nameservers running the stock Bind8, 2 MX's running stock sendmail. One 'users' box running Sendmail with spamassassin and spamassassin milter, along with apache2 and squirrelmail for webmail.

None of these boxes have the full sources installed, and in the past I've taken the boxes down and done a binary upgrade from CD. Is this the fastest method?

Re:Only slightly OT

[TheRaven64]

I don't know if this is true of the 5.x series (I assume it is though), but on my 4.x box I did the upgrade from 4.8 to 4.9 by doing a source upgrade. Everything new was installed while the system was running, and mergemaster updated all of the config files. After a reboot, I was running 4.9. Total downtime was less than a minute, and most of that was the BIOS POST.

This procedure is not recommended for moving between major version (e.g. 4.x -> 5.x), but for your setup it should be fine. Just cvsup the src tree and follow the upgrading from source instructions in the handbook.

Re:Only slightly OT

[TheLink]

You can build everything on another box, and then copy /usr/obj and /usr/ports/distfiles over and then shutdown to single user and do the installs from there.
See this for background.

There are many ways to do it depending on whether you want it built from source or just want the binaries.

Re:Only slightly OT

[Brandybuck]

From 4.x to 5.3-RELEASE, you're going to have to deal with a whole bunch of stuff, so that an in-place upgrade isn't recommended. Heck, it's going to take you much longer than a minute to run mergemaster.

In the meantime, I would be putting 5.3-RC on a test system to work out the issues.

Re:Only slightly OT

[Bishop]

I have always been a big fan of installing fresh on a new machine and copying the data over. This applies to most OSes. This method gives you a chance to test the new release before going into production. And once in production you can always switch back to the older machine if something goes wrong. Not enough people test an upgrade or have a downgrade procedure.

If you don't have a spare server don't be affraid to use an intermediate temporary server. It involves installing the os and copying data twice, but it is not as big a hassle as it sounds. If possible use fresh harddrives saving the old OS and data as a 'warm' backup.

Unfortunately if you are running a colocated server you probably can't do this. My only advice then is start Tuesday morning. Everyone knows not to start an upgrade Friday afternoon, but so many people still do. If you follow the instructions in the FreeBSD handbook your upgrade should be problem free.

Re:Only slightly OT

[JQuick]

It depends on how you measure quick, and on your risk tolerance.

If by quick you mean the least time start to finish, yes. If you mean as measured in system downtime, no. Each has a different risk profile which depends heavily on how much additional software you have installed.

I too have been running 5.x as a server environment since mid 5.0 days. I have performed 2 source based upgrades in the interim to bring me to 5.2. My preference for source based upgrades is based partly on my desire for quick response time re: security. It is also conditioned by my rather complex setup in which I have multiple jailed environments each running a large number of packages. A binary upgrade is less attractive since I would need to install dozens of different ports and possibly face conflicts or temporarily broken ports.

You have very few ports running, and from your statement they are pretty stock configurations. From this standpoint a binary upgrade should be relative painless. However, it might require more downtime.

If I were you and were running a GENERIC kernel, and was running a late 5.1, or 5.2_RELEASE, I would suggest a source base approach. if you are running an earlier 5.x version I would still do so myself but would counsel you to assess your comfort and knowledge with compiling the code and following /usr/src/UPDATING to the letter. If you are unsure, opt for a binary install.

If you do use a source base approach, I would prepare by installing the cvsup tools from the ports tree to mirror the source code and the ports tree. Then you can compile using buildworld and buildkernel, and even compile and install ports (using and alternate paths for the package db and destroot) to test versions of installed ports which might be newer.

Read UPDATING thoroughly and study any differences which you are unsure of. Then when you are ready, use install* targets and mergemaster to finish.

This is initially a longer, more time consuming approach, you must install sources, and configure cvsup to keep them up to date. Once that is done, however, they are always up to date. At each site which I have maintained FreeBSD, I use cvsup to mirror ports and sources on a single box. In fact, I mirror the cvs trees, enabling each host in the network to choose what particular version to check out. I then check out source trees via cvsup, and run a buildworld and a buildkernel via cron either weekly or monthly.

Thus, I always have a recent binary distribution ready to install when I feel like it. I upgrade rarely, but when I do, I typically have a 10-20 minute downtime. On boxes where I have configured multiple drives with sets of boot, usr, and var partitions, I configure and install to the alternate drive using the DESTROOT variable, and can take care of merging changes while running on the old version. Then downtime, is boot time + time to select the new boot partition.

Re:Only slightly OT

[feargal]

I usually upgrade from source. You don't need the current sources to upgrade, just the new ones, so grab them with cvsup.

I do the initial builds offsite and usually well in advance (perhaps leave them to work on a Friday evening).

1. make buildworld
2. make buildkernel

Once onsite, I:
3. make installkernel - takes a few minutes, doesn't count towards downtime.
4. reboot
5. mergemaster -p - takes about a minute
6. make installworld - takes maybe 5 minutes at most
7. mergemaster - this takes the longest - I usually manage it in about ten minutes, as I've become pretty familiar with it, and make the right decision pretty instantly.
8. reboot

I've timed myself, and I end up with 15-20 minutes downtime, depending mostly on the speed of the machine.

Going more off-topic, but I had an idea on how to make this process faster, and to make mergemaster much less scary.

Most of the files that are affected by the mergemaster process are rarely actually changed. On a stock server, you'll probably only ever change files in /etc/mail, /etc/namedb, /etc/ssh, and perhaps one or two other file. All the rest of the files are usually unchanged by the sysadmin. The mergemaster process however asks you if you want to upgrade all the files that have changed in CVS. This takes a lot of time and involves repeating the same keystrokes, and is probably the source of most accidents: shift-G, i, shift-G, i, shift-G, i, "oops, shouldn't have overwritten that one!"

It should be trivial to, pre-upgrade, traverse /etc, extract the version numbers, download their originals from CVS, diff the two, and build an "auto-list" of files which have never been altered.

When mergemaster is run, it can then automagically upgrade all of the files in the auto-list; if nobody saw fit to change /etc/mtree/BSD.usr.dist, chances are the new one will do just fine. Meanwhile, the sysadmin only has to think about that matter to him.

Also, prior to doing the upgrade, he would be able to get a list of files which he *has* changed, so he can figure out what exactly he was thinking when he decided to hack /etc/rc.d/initdiskless to bits.

Any reason this wouldn't/shouldn't work? Obviously mergemaster should give Big Bloody Warnings before using the list. I reckon I'd save at least 25% of my downtime doing this.

Re:Only slightly OT

[cperciva]

My preference for source based upgrades is based partly on my desire for quick response time re: security.

Entirely off-topic, but if you're concerned about security, binary updates are a better option than source patches -- both because FreeBSD Update is more secure than the cvsup mirror system, and because I normally have patches available via FreeBSD Update within a few minutes of the code being committed to CVS and the security advisory going out. (I have the advantage of seeing the source patches in advance, thanks to being on the FreeBSD security team.)

Of course, this only applies to tracking the security branches, but if you're concerned about security that's what you should be doing anyway -- we don't issue security advisories for issues which only affect -current.

Re:Only slightly OT

[Eivind Eklund]

I've made patches to mergemaster that does this a long time ago. These patches are available from my homepage at http://people.freebsd.org/~eivind/. They have not been integrated into mergemaster due to lack of time to review them on the part of the mergemaster maintainer.

However, this is just a more specific case of doing a 3-way merge. I was planning to add this to mergemaster, but due to the issues of maintainer timeout (over a two-year period), I've instead written a new tool: See /usr/ports/sysutils/etcmerge.

This does a full 3-way merge, which means that you usually need no manual interaction (as opposed to all you need to do with mergemaster.)

I'm presently preparing this tool for possible import into FreeBSD, so there will be some enhanchements in the next couple of days.

Eivind.

I have been impressed...

[brilinux]

As someone who has used multiple Un*x-like OSes, such as FreeBSD, OpenBSD, Gentoo Linux, Debian GNU/Linux (and I am not a zealot for any of them - imagine that!), and others, I have found that if I want to know about saomething or how to do something, FreeBSD has always been the best at having the information availiable. It is very easy to find what I need to know, and everything seems done very logically. Good Job, guys!

C'mon, trolly...

[Anonymous Coward]

C'mon, stop it. I'm starting to feel pity for you and I don't like that feeling!

Nearly 2.5 Million Active Sites running FreeBSD

um its pretty good, but not great...

[josepha48]

I have a FreeBSD 4.10 system running off a cdrom.. I had to do some hunting and research on how to get a bootable floppy image for the cdrom and also some info on how to deal with things like ssh, /dev, tmp and memory disks. Basically read the scripts and hack.

Now I am trying to get a bootable 5.2.1 cdrom. I finally found section 16 of the manual, which describes cdboot. It doesnt really say much else in the way of what do I need to put in the loader.rc file, if anything, or do I need one. It doesn't say if I need to have just cdboot or also boot0 and the loader. Doing just what they suggested left me with a none bootable cdrom.

I'd like to know more information on what I need to do to the boot directory to get a working bootable cdrom. Well I should give it some credit, it does boot, but then it stops and says could not find / . Do I need to specify load /kernel in the loader.rc?

I will say one thing, using cd-rw's and bochs has saved me a few cd-r's.

Re:um its pretty good, but not great...

[Strog]

Why not use Freesbie?

You can setup a FreeBSD live CD in no time. It's in ports (sysutils/freesbie). I've set up a couple live CDs playing around with it. I need to get serious and make a nice 5.3 CD after it's released.

Re:um its pretty good, but not great...

[josepha48]

I don't want all the GUI stuff. Does it allow for customization and reducing whats on there?

Is it 5.2?

For Such A Critical Yet Thankless Job...

[devphaeton]

Thankyou to all the folks that have created the world-class documentation system to go with the world-class OS that is FreeBSD.

*thumbs up*

Take a hike

[Anonymous Coward]

Go take your bullsh*t where it has a chance to be considered.

FreeBSD & DragonFlyBSD are great projects led by top notch developers, they both rule.
You, on the other hand, suck.

Get a life, moron - or, better, learn to code, before going around defaming people who can.

Re:Take a hike

[Anonymous Coward]

This pathetic troll is trying to put FreeBSD and DragonFlyBSD developers against each other. He already spammed the FreeBSD mailing lists, www.osnews.com, and slashdot.
He makes unsubstantiated claims, then asking other people to provide explanations to his trolling bullsh*t statements - a troll technique as old as the internet.
I heavily suspect this is the brainless GNU zealot that has spammed /. BSD section with innumerable FUD-spreading "death" messages - which, OTOH, don't seem to have provided the desired deleterious effect. :-)

If I may, I'd suggest to wait for him to provide proof of the BS he's uttering, and in the meantime, just to leave him in his misery, since every argumentative answer will just (ehm... you know) feed the troll. On the other hand, I think insults are ok. :-)
Thank you for your eventual attention.

Re:Take a hike

[Anonymous Coward]

Hi,

You're an asshole.

For your information, Tom Rhodes is a decent friend of asmodai@ and hmp@. Well, to the best of his knowledge. His complaints only came when he noticed a rant from David Rhodus which he felt was not usefull in the least. Tom also has no problems with DragonflyBSD, nor an account on slashdot. And how do *I* know, because I am ...

"The real Tom Rhodes" ... and I should create an account here, for amusement purposes.

--
Tom Rhodes

Re:Tom Rhodes

[Anonymous Coward]

Please ignore this troll. It's the same guy who has been pestering every *bsd (yes, even NetBSD) announcments trying to discredit FreeBSD developers. He usually goes on to say just how FreeBSD devs hate Dragonfly, how they unfairly kicked out Matt, and so on.

Sometimes he links to a message posted by DES on FreeBSD-advocacy in his signiture. If you take the time to see how that thread started, you'll see that the original "quesiont" was quite rude, and follow-up messages from the same person were written in a "I'm a famili member of the former Nigerian royal familiy and want to deposit large sums of money" style. Also if, you follow the thread further, you'll see this reply from a FreeBSD developer:

> BTW, I've spent a lot of time looking at the DragonFly approach, and I met
> with Matt for quite a while at USENIX to talk to him about the approach. I
> have a number of concerns about it -- I think the premise is very
> interesting, but that the results aren't yet there to prove the model. In
> particular, there's a huge volume of code in their system that has not
> been addressed, and a lot of complexity that will need to be handled
> before the SMP primitives they're using have proven that they offer the
> desired performance advantage. We have the opportunity of using a hybrid
> model, and have been exploring some of the ideas present in DFBSD (and,
> one should point out, many other SMP systems).
>
> A lot of other systems have opted to use elements similar to those
> primitives, but in a much more limited way due to the performance costs.
> For example, locking services into particular CPUs prevents the scheduler
> from balancing load between the CPUs in an service-transparent way. In
> the DFBSD model, load balancing must be implemented separately for each
> service, requiring extensive modifications to the services. I.e., the
> model may indeed offer benefits, but the cost of doing the work will be
> high, and the time to complete it long. We'll adopt elements of the
> design as they prove to make sense, as we do with all other open source
> operating systems (and they do with us!).

For your interest, Matt still posts occasionally to -current list, in fact, he even helps out a bit here and there. This troll's problem seems to be with DES, PHK, Bosko, but he is ready to extend his warm words towards anyone, even, it seems, to someone associated with the documentation project. Oh, btw: you'll see the same message by Doug-Furlong Smorgreff on Osnews as well. ~molnarcs

Re:Tom Rhodes

[MavEtJu]

Hello troll, who let /you/ out?

Re:Tom Rhodes

[molnarcs]

Haha, little troll, I checked out your link, and it doesn't prove your point if you follow the whole thread. Points are made pro and con, and actually it is Matt who is a little bit more arrogant than, let's say PHK. :)))

Are you the frustrated HawkinsOS guy btw? Some things points to it ... if it is true, than I understand your frustration. You are on a crusade against FreeBSD developers, because they pointed out that 'your' os violates a number of licences that are part of the BSD system. Let me just say for all who would buy into your bs: you were given a good advice in a polite manner, and you reacted to it as if someone close to you was murdered by the FreeBSD devs.

Of course, it is possible that you have nothing to do with HawkinsOS. If that is the case, well, nevermind. ;)

People... hurry!

[ulib]

Come on... PHK, Long, Watson, DES, Rhodes, and anybody else he names... quickly, you all get out of FreeBSD immediately!
Otherwise, the Troll isn't giving back his enhancements! :-D

P.S. He sold 2,000 copies of a *beta* system! :-D

uh.. I didn't want *your* eventual attention. [nt]

[Anonymous Coward]

[nt]

Evilness documentation ...

[Anonymous Coward]

Why had they deleted a part of document about "Colouring the pages of processes's maps"?

It's appeared in the old documentation, but doesn't appear in the new documentation, so why?

open4free ©

Re:Tom Rhodes

[molnarcs]

You're right. As I pointed out some time ago, my OS does indeed respect the (C) notices. All I got when I pointed out the abysmal SMP support in FreeBSD 5.3 was a lot of flamage. I've sold nearly 2,000 copies of my beta system so far, and was planning to give back the enhancements to the FreeBSD team. I won't, at least not until they remove assholes like Scott Long and DES from the team. phk? Who cares, he's the most annoying SOB I've ever seen.

I usually don't reply to trolls, but this one is soo funny, I can't resist. I just want to say that I noticed that your list of assholes is expanding: now you added Scott Long as well (and the doc people for God's sake!!!). As ulib noted, you'll have almost all FreeBSD developers leave FreeBSD development for your 'enhachments.

To add more fun to what ulib noted: did you spell-check the announcments on your site? I mean, how should anyone take you and your 'enhachments' seriously if you can't check your own site for errors. Not being English is not an excuse in this case (what, you don't have the resource to get one man proof read it for you?). You are way out of touch with reality my friend. And to think that you probably made your threats with a straight face, and to think that you are trolling on slashdot... what would your customers say?

Haha, maybe there is a connection between your "Nigerian royal family member" style and the fact that you are selling FreeBSD renamed to HawkinsOS for 138$! Oh, it is only 1000$ if you buy 8 packages! YESSS, that's really the best deal if I ever heard one. You are one little entertaining troll. And this, in your about HawkinsOS box on your website:

About

HawkinsOS is being developed by T.J.HAWKINS. It relies heavily on the FreeBSD Foundation. More information will become available soon.

Yes, I'm sure the FreeBSD Foundation is glad to hear that they are writing FreeBSD. And I'm sure DES and PHK are really eager to get your 'enhachments'. Some more entertainment:

Consulting

Documentation of HawkinsOS

Community (!)

Well, you get the picture. Everything will be availabe if you deposit some money there after june 15 2004. :))))))

Re:Tom Rhodes

[Anonymous Coward]

Look, I've been doing consulting work for nearly 15 years. About the site, I had to put up an old copy because after that thread several FreeBSD members broke into my network and nearly deleted everything. I'm 75% sure that it's the work of Dag-Erling and his friends, as most of the packets and exploit attempts (until they succeded) were coming from Norway and Denmark.

I've sold 2,000 copies and made money off *my* work. How many copies has the FreeBSD foundation sold? Oh, that's what I thought, becuase 5.3 is not usable in a production environment, not without my patches. And guess what, they won't be getting my patches unless I see public apology from DES and PHK.

Security

[JQuick]

I'm glad you brought that up. I've been so long out of a well supported branch that I forgot about Software Update entirely. I agree with your assessment. Those who are on well populated branches of the code tree are best served by FreeBSD Update for security. Since security is a moving target, running on a STABLE branch and keeping up to date is the best way to ensure both stability and security. You correctly pointed out that if security were my primary concern, I should have been doing that.

My requirements several years ago made deploying a 5.0 snapshot release very worthwhile. The combination of functionality I desired was too compelling, and likely not to be added to 4.x for some time. Thus, I have balanced functionality, stability, and security by making certain trade offs. Obviously, a breach in security affecting the main cvs tree or cvsup could prove catastrophic.

However, once I decided to live in a relatively unsupported part of the code tree, it has served me well. Since I have a local mirror of the cvs repository, periodically checking out the sources and building via cron is very efficient and requires no labor whatsoever. Many security updates require me to simply ensure that my latest compile went smoothly, cd to an appropriate subdirectory, and run a script to perform a make install to the main system image and once for each jail DESTROOT, then restart service or the system as required.

If such a largely automated ability to upgrade the system or a subsystem was not present, I would have been foolhardy to run a production server so far from a well supported release. In retrospect I am extremely happy to have run so stably and so securely for the past 3 years. I have not had any crashes or unplanned outages during that time (except for a drive failure). I have also to my knowledge (there's always a possibility!) had no security breaches either to the host or to any of its jails.

In my opinion the true measure of security and stability is measured over time, not by a particular point in time. Security is a constantly moving target. If you remain still, you may eventually become vulnerable, or discover that you always were vulnerable, and just did not know it yet. The true beauty of FreeBSD, is that on a stable branch, it is possible with little effort and negligible risk to update your system frequently. Moving gracefully, in small steps as security, stability, and functionality improvements become available.

For a variety of reasons, I have chosen a much less traveled path. My trajectory has not been as graceful. However, my stability, and reasonably high level of security, has been maintained with very little extra effort. This is a testament to the rigor, and high quality of the FreeBSD security and release engineering processes.

I do look forward to rejoining the STABLE fold, soon after 5.3 is released. It will give me more options for rapid response to security, and make more frequent minor upgrades a viable approach again.

Re:Tom Rhodes

[ulib]

Thanks a lot for providing the full picture, and above all for revealing one of the funniest trolls I stumbled into throughout my goddamn internet life. Just FYI, this afternoon I spent a considerable part of my spare time genuinely LMAO. This is indeed a unique specimen, a kind of a troll champion: I thought the "death" troll was hilarious enough, but *this* guy is waaay beyond that... this is another planet altogether. :-D

All documentation available online and offline

[phoenix_rizzen]

One of the nicer things about the FreeBSD Documentation Project is that everything is available both online and offline. All the man pages for every release of FreeBSD (going all the back to 1.0), along with OpenBSD, NetBSD, and several Linux distros, are available at http://www.freebsd.org/cgi/man.cgi

And, if you selected the docs distribution during the install, you'll find all the articles, books, and papers under /usr/share/doc, including the Handbook and the Porter's Handbook. If you didn't install the docs during the initial install, they can be fetched (and/or updated) using cvsup. There's a samples docs supfile in /usr/share/examples/cvsup. Just be sure to set DOCS_LANG in /etc/make.conf to the language you want, otherwise you'll get every language availables. :)

Having all the documentation available offline is a boon for those days when you break the network. :)

Re:Tom Rhodes

[DashEvil]

When you left -current I had some respect for you. You didn't seem to really have a problem with anyone -- just wanted some information.

Now look. You're a run of the mill BSD troll on SlashDot. What happened to you?

Personally, and your "customers" may disagree with me on this one, I would never buy an OS from someone that spends all of his time demeaning the work of others on public forums.

Your critizism isn't even constructive, it's vague and without merit. Like that conversation between DES and Matt Dillon. I don't know what you see, but I see two skilled coders disagreeing about something. Yes, normal disagreements can involve some harsh words. Yes people get angry; they get frustrated. Shit happens. But at the end of the day you have two projects, two ways of doing things, and it's YOUR choice as to which one performs better.

So what do I see? I see two groups of people getting into a fight about something and setting out to prove their ways. Everyone benefits from this. I look at you and what do I see? I see a man that fights only with hollow words.

Good luck to both the FreeBSD and DragonFlyBSD team.