Debian Elevates KFreeBSD Port to First-Class Status

Reader tail.man points out this press release from Debian which says that the port of the Debian system to the FreeBSD kernel will be given equal footing alongside Debian's several other release ports, starting with the release of Squeeze. Excerpting from this release: "The kFreeBSD architectures for the AMD64/Intel EM64T and i386 processor architectures are now release architectures. Severe bugs on these architectures will be considered release critical the same way as bugs on other architectures like armel or i386 are. If a particular package does not build or work properly on such an architecture this problem is considered release-critical. Debian's main motivation for the inclusion of the FreeBSD kernel into the official release process is the opportunity to offer to its users a broader choice of kernels and also include a kernel that provides features such as jails, the OpenBSD Packet Filter and support for NDIS drivers in the mainline kernel with full support."

Comments

OK

[Profane MuthaFucka]

But, does it run Linux?

Re:OK

[Jurily]

GNU/FreeBSD

Flamewar starting in 5..4..3..

Also vi, KDE3, Gentoo and K&R.

Re:OK

[incripshin]

You son of a bitch. I use vi, KDE, Gentoo, and K&R.

Re:OK

[Korin43]

I think you're confusing "important part of an operating system" with an operating system. Linux is definitely not an operating system, it's just a common term to refer to Linux-based operating systems (because the average person doesn't care). Just like the FreeBSD kernel alone isn't an operating system. Debian and FreeBSD are operating systems. GNU appears to be a complete operating system (although not finished). I think you can install it from here: http://www.gnu.org/software/hurd/hurd/running/gnu.html.

Need a simple proof that Linux isn't an operating system? Download the kernel and boot it (oh wait, you can't because GRUB isn't part of the kernel), do some stuff on the command line (bash is not part of the kernel either), maybe update some programs (apt isn't part of the kernel). Oh wait you say, I don't need apt, I can just download the source and compile it myself... but wget and gcc aren't part of the kernel either.

And don't take this to mean I think we should call it GNU/Linux. People can call their operating system whatever they want. If they wanted every piece of software that uses GNU to be called GNU/Software, it should be in the license. My point is just that kernel != operating system.

Cool

[Faw]

First apt based distro with ZFS? Something worthy of a post about...

I know about Nexenta, but FreeBSD has more drivers than OpenSolaris, right?

Re:Cool

[spun]

First apt based distro with ZFS? Something worthy of a post about...

I know about Nexenta, but FreeBSD has more drivers than OpenSolaris, right?

You seem to be asking some interesting questions, but fail to do so in a timely fashion.

Re:Cool

[characterZer0]

Why would you not want to use APT on a server? What part of automatic dependency handling, automatic unneeded package pruning, easy security update application, and secure package retrieval do you not want on your servers?

Re:Cool

[Timothy Brownawell]

Why would you not want to use APT on a server? What part of automatic dependency handling, automatic unneeded package pruning, easy security update application, and secure package retrieval do you not want on your servers?

Possibly the "automatic unneeded package pruning". It could be dangerous if your custom apps don't specify their dependencies correctly (say, they rely on something that had been automatically installed by one of their other dependencies).

Re:Cool

[Timothy Brownawell]

If your custom apps required you to install a package, it'll already be listed as manually installed, so it'll never be automatically uninstalled.

The idea is that $app depends on $foo and $bar. But because $foo also depends on $bar, someone was able to goof up and only document that $app depends on $foo. So when $foo gets updated and drops its dependency on $bar, $bar goes away (due to being automatically installed) and $app stops working.

Re:Cool

[Chris Mattern]

If your custom apps required you to install a package, it'll already be listed as manually installed, so it'll never be automatically uninstalled.

Not if the required package was already installed because a third package that required it and correctly specified it was installed. Uninstall that package, which seems to be utterly unrelated to your custom app, and BOOM, custom app breaks.

Re:Cool

[Chris Mattern]

"nearly here". No other words can strike such fear into the heart of a production system sysadmin. How about something that's seen production use for years, instead?

Re:Cool

[Jurily]

Do we need Linux any more, now that HURD is nearly here, based on the same ideas but with Linux's design problems known about and worked around?

Re:Cool

[Sancho]

Wow, you got modded Troll and Timothy got modded Offtopic for describing a legitimate concern. Man, this place has gone downhill.

The correct rebuttal to your statement is that you don't mess with things on production machines. You don't uninstall that third package. If you want to make changes like that, you do it on your test machine first. Timothy was concerned with packages dropping dependencies, but that shouldn't happen within -stable.

Re:Cool

[oldhack]

God damn foobar apps.

Re:Cool

[bfields]

Sure.

Options are to keep the autoremoval turned off, or build simple custom packages for your third-party aps with the proper dependencies.

Or you could do without any package management entirely. That doesn't strike me as likely to make anybody safer....

Re:Cool

[ToasterMonkey]

There has been a lot of hype about ZFS but what use is it in a desktop system? And honestly, while APT is great for desktop systems, I really wouldn't use it much on a server. So unless there is some amazing benefit for the average user with ZFS why even have this port as a main system?

You must be kidding. You can snapshot your whole root, or your home directory, or anything and automate it for backups. There is even integration with GNOME's Nautilus to browse ZFS snapshots at the file level. You can create new filesystems and snapshots on the fly, compress them, enforce quotas, export via NFS, send snaps to a remote system or dump as flat file, etc, etc, etc.. if you don't have enough disks to use single or dual parity RAID-Z, you can even have ZFS record multiple copies of each block. ALL of those have uses on desktop or workstation systems.

I'm not sure if freeBSD supports ZFS root or not, but Solaris does if you want a taste. If anything, ZFS-root is under-hyped.
When troubleshooting updates, instead of booting into your old kernel with a trashed userland, you can boot into an old BOOT ENVIRONMENT. As easy as picking a different grub entry.

From OpenSolaris (other ZFS-root capable systems may use different commands, output BUTCHERED for junk filter)
beadm list
BE Active Mountpoint Space Policy Created
opensolaris 25.05M static 2009 04 01 2033
snv_111b 111.89M static 2009 06 03 1846
snv_121 38.18M static 2009 08 31 1617
snv_122 42.10M static 2009 09 14 1522
snv_124 NR / 19.09G static 2009 10 01 2354

Those are whole root file system snapshots I can boot, consuming a piddly ~50MB each.

Junkfilter is retarded.. there is a moderation system for a reason, and this is a technical forum.

Re:Cool

[agnosticnixie]

HURD is, and will always remain, the OS of the future.

Re:Cool

[stevey]

Raises hand.

The only time when I've worried about running (Debian) stable software on production servers has been towards the end of a releases' life.

In that case signatures for things like clamav were often useless. However right now I'm running entirely stable software and don't expect that to change for the forseeable future.

A lot of people seem to want the latest and greatest releases of software for no appreciable reason. (If they were hitting specific bugs I'd understand ..)

Re:Cool

[domatic]

There are more and less correct ways to go about running "non-stable" packages. Just jamming in a foreign binary deb and it's deps is generally the wrong way to do it. The right way to do it is to basically do what backports.org does....and check if backports has it first. If you're not running a Debian derivative then research the equivalent for your distro or BSD flavor:

1. add an apt-src line from the "non-stable" distro. I've even done this when Ubuntu had the latest SpamAssassin and I wanted a late model debianized SpamAssassin. Note well I said "apt-src" and NOT an "apt" line. apt-get update, yadda yadda

2. mkdir package-of-interest; cd package-of-interest; apt-get build-dep package-of-interest. If you're lucky this pulls in everything needed to build the package. If not you'll have to recurse with apt-get build-dep package-of-interest-newer-dependency; apt-get source package-of-interest-newer-dependency and come back to step 2 until you have everything you need to build your "non-stable" package.

3. apt-get source package-of-interest (or -newer-dependency)

4. cd package-of-interest-version; fakeroot dpkg-buildpackage -b. If all goes well a backport of the newer software that is built against your nice stable distro's libraries will be built one directory level up.

5. If you had to build a newer library your package needs install it then go back to #2. rinse and repeat until you have your newer software. Myself, I'll build one or two before giving it up as a bad job. Generally I succeed at this. Bread and butter stuff like Clamav and SpamAssassin that is well behaved tends to work well with this procedure.

The point of this is to avoid having to replace heeby jeeby inducing things like glibc, libstdc++, or even Perl since half your system scripts depend on such things. You avoid replacing packages that are contributing to the stability of the rest of the system and that are still getting critical bug and security updates. It keeps the "foreign" footprint to a middle and gives you a "foreign" package that is more or less compliant with the rest of the system.

Re:Cool

[X0563511]

You know about Debian-Volatile right? What's even better, is this is part of -stable and has been included in your default repository config since Lenny released.

This is meant specifically for moving targets like ClamAV.

Re:Cool

[X0563511]

By the way, there are ways to hold packages too

This is called pinning, if anyone is looking for the solution.

or to make lists of packages required, even for your own scripts.

"equivs" can be used to create empty packages for the sole purpose of manipulating dependencies. I usually use it to kill packages that are otherwise demanded in other important metapackages, though you could also use it to 'hold' dependencies for a broken third-party .deb package.

Linux vs. FreeBSD

[Boawk]

As a UNIX/Linux veteran, I have to admit that I've almost no experience with FreeBSD. Could someone summarize why one might prefer it over Linux?

Re:Linux vs. FreeBSD

[Anonymous Coward]

Features such as jails, the OpenBSD Packet Filter and support for NDIS drivers in the mainline kernel.

Re:Linux vs. FreeBSD

[Bill, Shooter of Bul]

That depends upon what you mean by veteran, and what you mean by UNIX. FreeBsd is closer to Unix due to its BSDness. So if you are used to kernels that are more Unix-y than Linux-y you may prefer it for that reason. If you are simply a fan of OSS that runs it as a desktop, there may not be any obvious advantages and perhaps some disadvantages due to lack of desktop like software. It should also include ZFS & dtrace which may entice you. Its also just a different kernel with a different schedule that may perform better for your specific tasks. Osnews carried a story about a benchmark between FreeBSD and Ubuntu the comments from osnews readers are also pretty insightful which is why I linked to them and not the source article. .

Re:Linux vs. FreeBSD

[Niten]

In my opinion, the biggest advantage of FreeBSD is how coherent the system is. Everything, from documentation to userspace utilities to the kernel, was developed and tested and released as a single project.

This allows for neat features that require cooperation between several system components, which would be more difficult to implement in the Linux world. For instance, in FreeBSD you can press ^T while cp is copying some huge file, and this will send SIGINFO to cp, causing it to print a progress report to STDERR. Handy.

So it seems to me that this Debian project defeats the most attractive feature of the FreeBSD operating system (by separating its kernel from its tightly integrated BSD userspace), while simultaneously casting aside Linux's advantages over FreeBSD (more drivers, more supported architectures, somewhat better performance, and--this may be controversial--in my experience, better stability). On the other hand, maybe Debian really can improve on the FreeBSD experience; apt rocks, and the Debian project does perhaps a better job than anyone of combining the disparate parts of the GNU/Linux ecosystem into a coherent operating system. So kneejerk reactions aside, I guess I shouldn't judge this until I have the chance to try it... still, I don't see myself trading in my Debian GNU/Linux anytime soon.

Re:Linux vs. FreeBSD

[blind biker]

One thing I noticed: a working and consistent sound system.

Re:Linux vs. FreeBSD

[XanC]

For instance, in FreeBSD you can press ^T while cp is copying some huge file, and this will send SIGINFO to cp, causing it to print a progress report to STDERR. Handy.

Isn't this an internal feature of their cp implementation? I don't see what this has to do with the kernel, or indeed any program besides cp, at all.

Re:Linux vs. FreeBSD

[poopdeville]

On the other hand, maybe Debian really can improve on the FreeBSD experience; apt rocks, and the Debian project does perhaps a better job than anyone of combining the disparate parts of the GNU/Linux ecosystem into a coherent operating system.

I am not a big fan of the BSD userland, and I typically install "prefixed Gentoo" on my Macs. (Basically, it brings in a GNU user land, a fresh compiler chain, etc. It works well, but the repositories are very basic. It can help set up a Unixy programming environment, not a feature complete Unixy desktop system)

kFreeBSD Debian can potentially make Apt a real option on Macs. Fink sucks. Debian's repositories are much better.

Re:Linux vs. FreeBSD

[ducomputergeek]

As someone who has had a lot of experience with both, I switched to BSD in 1999. Back then the main reason was Ports. Needed to install MySQL: /usr/bin/ports/databases/mysql/ make && install. Then go grab a cup of coffee come back and it would fetch everything it needed, compile, and run. Or you could fetch a pre-compiled binary via pkg_add -r mysql. Hell, the first few version of PostgreSQL I used, the only way I could get the damn thing to work was to use BSD ports. The best you had with Linux was RPM and that was dependancy hell at times.

Also, back in the day it had a better tcp/ip stack and was generally more stable as a server platform and decent SMP support. And frankly it was far easier to support than "linux" was back in the day because there was a single FreeBSD, not umpteen different flavors.

Today it has ZFS and Dtrace from solaris ported over. I know ZFS hasn't made it into Linux as of yet, not sure about DTrace. But both are handy tools.

Currently we're deployed 100% on FreeBSD for our web, mail, and database servers running PostgreSQL. But that has more to do with using Pair Networks than any other single factor. They've been 100% FreeBSD and consistently in the top 10 in terms of uptime according to netcraft.

For the past 10 years, I've found FreeBSD to be a stable, secure server operating system that doesn't take a lot of system resources to run. It seems like Linux takes about 256MB of ram these days in most default configs to run a web server whereas our BSD machines were using closer to 150MB for the core OS. And was both systems running Apache 2.

Re:Linux vs. FreeBSD

[cbhacking]

A lot of wireless hardware still doesn't have drivers (at least, not working ones) on Linux (or FreeBSD). For obvious reasons, NT drivers do exist. FreeBSD's kernel supports directly loading those NT drivers. Linux has ndiswrapper, a project to allow the same thing (ndiswrapper itself is a Linux kernel module that attempts to load the NT driver binary) but the FreeBSD NDIS support is a feature of the kernel itself, and supported as such.

For TL;DR folks: if you've ever had trouble making WiFi work in Linux, this might help.

Re:Linux vs. FreeBSD

[Sancho]

The grandparent was trying to make a point, but failed. Similar behavior exists throughout the FreeBSD userland--you can send SIGINFO with ctrl-t to many userland processes to get information on what they're doing. The point is that FreeBSD's kernel and userland were designed as a system, and little touches like this show that off.

Re:Linux vs. FreeBSD

[Guy Harris]

For instance, in FreeBSD you can press ^T while cp is copying some huge file, and this will send SIGINFO to cp, causing it to print a progress report to STDERR. Handy.

Isn't this an internal feature of their cp implementation?

No. The fact that ^T sends SIGINFO, just as ^C sends SIGINT, is a feature of the "tty driver" (standard tty line discipline). The fact that a particular program catches SIGINFO and prints a progress report is a feature of the program.

I don't see what this has to do with the kernel...

The standard tty line discipline referred to above is in the kernel.

Re:Linux vs. FreeBSD

[TheRaven64]

I would say mod parent up, but you're already at +5. This was what made me switch from Linux around 2002: Sound Works. I had a cheapy AC97 CODEC in my computer at the time. It didn't do hardware mixing. I installed Linux. There were two drivers, an OSS one and an ALSA one. Neither one let the majority of my programs play sound at the same time. For example, I couldn't have xmms playing music while I played BZFlag. Both KDE and GNOME came with their own sound daemon, which meant that either KDE programs could make sounds or GNOME programs could make sounds. Oh, and I think if I used the ALSA drivers then two programs that had had their sound output rewritten to use ALSA could play sounds at once... sometimes. Then I tried FreeBSD.

In FreeBSD, there was one driver for the sound device. This was back in the 4.x days, so sound devices needed a little bit of extra configuration. I had to set the number of virtual channels and then tell each device to talk to a different one. I set up 4, one for GNOME, one for KDE, one for xmms and left the default one for whatever apps just wrote to /dev/dsp. I could have music playing, BZFlag sound effects in the game, and notification beeps when I got an email or IM. Then came FreeBSD 5 and all of that manual configuration went away. To play sound, a program opens /dev/dsp and writes audio data to it. That's it. No libraries to link against; it's all done via standard UNIX system calls (open(), read(), write(), and ioctl()). It's trivial to program for and it just works. With FreeBSD 8, you now get per-channel volume controls and a rewritten high-performance mixing algorithm, as well as support for all of the new OSS 4 APIs (backwards - binary - compatible with the old OSS 3 ones) if care about those things (i.e. if you are a programmer).

A few other things that I like about FreeBSD:

1. Documentation. The man pages and the handbook are written by people who seem to understand both English and the subject that they are writing about. They've also been translated into other languages, but I've only read the English ones.
2. Sane and simple init system. RCng isn't as all-singing, all-dancing as something like Launchd, but it is sufficiently powerful and easy to understand.
3. Interfaces don't change. Once you learn how to do something on FreeBSD, you know how to do it. Tools don't get arbitrarily replaced with ones that are in some way better but have completely different interfaces.
4. Clear separation between FreeBSD and Other Stuff. Ports go in /usr/local/, everything else goes above that.
5. Jails. If you want to isolate something, or you want a simple testing environment, jails are great. They are almost the same as chroot, except each jail has its own set of users. Root in a jail is not root outside the jail and so can create arbitrary other users inside the jail but can't escape. With FreeBSD 8, jails have virtualised network stacks and now work recursively, so root in a jail can create a new jail. They work especially well when combined with ZFS, because you can make an O(1) ZFS clone of a clean install of the base system to create new jails.
6. ZFS. RAID without the write hole, per block checksums, top-to-bottom transaction support, O(1) snapshots and clones - what's not to like?
7. Ports / packages. Not really anything special anymore, but nice and clean. Ports build from source and can have custom build options easily set. Packages are built from ports. Use them interchangeably.

There are probably some things I've forgotten. I was going to say that the only thing I miss on FreeBSD was valgrind, but it turns out that Valgrind 3.5 has been in ports for a little while now and I just wasn't paying attention (so, obviously, I don't miss it that much).

We have to put an end to their monopoly of awesome

[SafeMode]

It's nigh time that we look at the sheer scrumtrulesence of Debian and realize that it's reign of End All Be All of OS's must be curtailed and possibly even put an end to. No single OS should be this awesome. And we can no longer ignore the fact that it is.

Awesome! But...

[Anonymous Coward]

This is a really cool thing, except that I wonder how much this is going to be used? I'm sure there's a group of people who will be interested in this, and it might be a great stepping stone for those that want to move to/from FreeBSD to/from Linux, but a lot of the FreeBSD community is heavily focused on the fact that FreeBSD is developed as a complete OS. The userland and the kernel are developed by the same people and integrated. So while this is exciting, I'm not sure how much interest you're going to get from the FreeBSD community. Similarly, a lot of the Linux people who use Debian don't think of using Debian but think of using Linux, Debian just happens to be the distribution they choose.

Now, what may be interesting that'll come out of this is packages with better FreeBSD compatibility. That is something I look forward to.

Scrumtrulesence

[sexconker]

Scrumtrulesence is a perfectly cromulent word.

Re:I still dont see the point

[PCM2]

if you want FreeBSD, use it.. If you want Linux, use it instead.

Yeah, that's what Debian said.

Re:Will Debian move to BSD permanently?

[IgnoramusMaximus]

Is this a stepping stone to Debian moving from Linux to BSD permanently?

Unless the elected leaders of Debian all go insane at the same time, not very likely.

I'm trying to figure out if the FreeBSD licenses are more compatible with the Debian philosphy, or less.

Far less. If Debian were to ever go BSD as its primary license (see point one), somewhere in the vicinity of 90% of its contributors would leave, probably to start a new GPL-ed distribution.

Inclusion of the BSD kernel is not the same as an adoption of the BSD philosophy, as the kernel is an interchangeable component of the whole, very much like the much maligned Hurd is, of which there is also a Debian-based experimental distro.

Re:I still dont see the point

[acey72]

Debian is in effect raising BSD from the dead. IMO it's a good thing, the more OSes there are, the better.

If being made into the un-dead means becoming more like GNU/Linux, I'd rather just keep me and my demonic servers six feet under please.

Re:I still dont see the point

[vlm]

And this happens now on BOTH sides of the fence, so mixing this improves the situation how? I see it making it worse if anything.

Software that compiles and installs on BOTH BSD and Linux, has not been all that unusual since, perhaps, 1991-1992.

Re:I still dont see the point

[vlm]

I really don't get why Debian would do this though because of the fact that it will take away from its primary user base (Linux users) to help fill a possible niche of users (KFreeBSD users) that are small in number.

That type of question makes sense when asked about Microsoft, but doesn't even make sense when discussing Debian. "Why would Debian do this" is like a zen koan, until you're enlightened it makes no sense, or when it makes sense it means you're enlightened.

Debian developers do what they want to do, within the legal framework and societal tolerance. If the guys doing the port, feel like doing the port, they do the port, and we get a "testing" quality port, and if its good enough, TPtB declare it a release-quality architecture and we eventually get a "stable" quality release. There is no "Debian" borg style hive mind, or if one does exist, instead of "the three laws of robotics" or "the ferengi laws of acquisition", the hive mind has the social contract and the DFSG. There is no top down militaristic business command structure. Very few people in Debian with positions of power have the "wikipedia" attitude of "I'm not personally interested in your work, so I shall gleefully destroy it while laughing, ha ha ha".

In summary, they felt like doing it, they did it, some folks in positions of power acknowledge it. Its the same deal for all Debian packages, this port is not getting "special" treatment.

Not as insane as it sounds

[cracauer]

This sounds insane to people who approach this from the usual angle. Linux has a lot more support for all the junk and semi-junk hardware out there, but some of the GNU core Unix userland is of questionable quality. All of us cursed GNU creeping featurism in the commandline utilities and GNU libc problems at some time or another. You would think people want the Linux kernel and the FreeBSD Unix userland. So why go the other way round?

There are very specific needs being addressed by using the FreeBSD kernel inside a Debian.

FreeBSD's ports system for third-party applications only has a devhead, and that has caused an increasing number of problems. FreeBSD has stable branches and releases for kernel, for "core Unix" userland including binutils and gcc/g++, but not for third-party applications. At the time that this was created it was great, because what we wanted at the time was a stable base system to do "server stuff" with, and the ports/applications were just for accessing the things, a light desktop that didn't do much except run xterm and emacs.

Today, I see two main problems with what worked a few years back:

1) those "server style" third-party applications aren't sitting flat on a Unix anymore. They are stacks of dependencies of considerable depths. It's not an apache with mod_cgi and the base perl system anymore.

2) some third-party applications became very aggressive lately and can be unusable in their newest releases. Many people bash GNOME and/or KDE, myself my favorite target is Xorg. The Xorg server has caused the most headaches across all my Linux and FreeBSD machines in the last years.

So, here's the trick. FreeBSD only has one branch in ports, so even if you use an older -STABLE release branch of the FreeBSD core system you still get the newest releases of third-party applications via ports. That's why my *most* stable OS (FreeBSD) had caused me the most headaches lately, because it upgrades me to the newest Xorg *first*, not last like it should.

I don't want to distract too much from the point of this posting by giving reasons why people want the FreeBSD kernel, let's just say there are enough of us. But no matter how much you want the FreeBSD kernel, many see increasing problems with ports/applications for the reasons I gave.

Debian provides stable branches for all applications, and that makes some people who don't generally like Linux still go "PLING!".

In addition to all that, Debian's packaging system, and the way that it is kept working (few package screwups upgrading), the way that it integrated /etc/* file management are simply first class and blow other Linuxes out of the water, too. Debian's packaging is the best out there, I haven't seen anyone challenge that notion in a long time.

So, very suddenly you have a demand for the FreeBSD kernel in a Debian application provision system and here we are.

%%

(BTW, what blows my mind for real is that FreeBSD is now partially sold based on driver availability. Because they kept their NDIS windoze driver integration system alive and maintained when Linux didn't. That is ... something, I have to think about it)

Re:Start the RMS timer...

[TheRaven64]

It's part of the name, and it makes sense. This is a GNU userland (importantly, a GNU libc) with a FreeBSD kernel. You probably can't run FreeBSD binaries on it, because they will expect different libraries. You can run any GNU programs that don't make system calls directly and you can run most GNU/Linux binaries with the Linux kernel ABI module loaded.

This is not how a sane admin uses apt.

[brunes69]

Any sane admin has their own local apt repository, that they point all production and testing servers at. That repository has both "stable" and "testing" branches, like any apt server. All of the production servers grab off of the "stable", and all of the testing off of "testing".

The trick is, this repository SHOULD NOT be a mirror of the actual debian repository. Rather, the "testing" of your internal server should be a mirror of the "stable" debian tree. Then, weekly or daily or whenever new debian "stable" packages come out, you update your testing boxes, and TEST the packages against your local software. If something breaks, no harm no foul - you wait till the next update.

Once everything is tested OK, you sync those packages over to YOUR "stable" branch, and then that night all of your production servers will automatically get those updates. No fuss, no muss.