RHN Bind Update Brings Down RHEL Named

alexs writes "Red Hat's response to update bind through RHN, patching the DNS hole, made a fatal error which will revert all name servers to caching only servers. This meant that anyone running their own DNS service promptly lost all of their DNS records for which they were acting as primary or secondary name servers. Expect quite a few services provided by servers running RHEL to, errr, die until their system administrators can restore their named.conf. Instead of installing etc/named.conf to etc/named.rpmnew, Red Hat moved the current etc/named.conf to etc/named.conf.rpmsave and replaced etc/named.conf with the default caching only configuration. The fix is easy enough, but this is a schoolboy error which I am surprised Red Hat made. Unfortunately we were hit and our servers went down overnight while RHN dropped its bomb and I am frankly surprised there has not been more of an uproar about this."

Comments

You didn't test before deploying an update?

[Anonymous Coward]

So, you didn't test the update on a non-production server? Just install any old patch and let it take your network down? Who do you work for again? I have to make sure not to do business with that.

Re:You didn't test before deploying an update?

[suso]

Actually, I caught the error just from looking at the output of up2date/yum. It clearly said named.conf saved to named.conf.rpmsave. So all you have to do is compare what changed, implement any changes and copy named.conf.rpmsave over named.conf.

  Just as I said on the day of the release, be careful, don't just blindly update things.

Re:You didn't test before deploying an update?

[Sleepy]

>You know, not everyone has non-production servers. Every server we have IS production. And if you are paying for Red Hat Enterprise, you expect Red Hat to have tested these updates themselves. If this was a Microsoft error, Slashdot would be all over Microsoft for allowing this to happen.

You are wrong; stop whining. You're just painting yourself as misinformed.

1) The updates WERE tested.
2) The admin installed "caching-nameserver", then configured his install to act far outside the default.
3) He allows automatic updates straight into production. So do you it seems. Good luck with that! RHEL documentation says to not do this, but you're a bigshot "paying" for something different. I suggest you get a sidekick, and stick to the Windows side of your "enterprise".
4) He didn't revert his .conf file, as is usually needed when some new line is added to a server .conf. This is SO NORMAL you'd have to be a n00b to get bitten!

Your MS comparison is apples and oranges. If this guy did TEN MINUTES worth of testing he'd realize something's up, and he could revert the rpm package. How many MS updates prohibit uninstall? Quite a few!

In Windows, you can't diff the before & after config, since Windows admins would rather be blind to what they're installing, since that's the norm and it's accepted.

bug details

[tommis]

Here's the bug details: https://bugzilla.redhat.com/show_bug.cgi?id=453340

One of the bug comments says: "Latest caching-nameserver renamed my named.conf to named.conf.rpmsave in /var/named/chroot/etc" - so this should mean that you can still restore the lost conf file.

Re:bug details

[hughesjr]

it is not a bug to get a caching nameserver if you install caching-namesever ... it would be a bug to install caching-nameserver and NOT GET a caching nameserver.
A caching name server IS one that does not have any zones and only looks up zones from the DNS root servers. It is a configuration error to install the caching-nameserver package on a machine that doing anything other being a caching name server.
Stupid admins have been complaining about this for 5 years ... but the documentation and bug entries all make it clear NOT to install the caching-namesever packages on DNS servers that control zones.

You are WRONG :D

[hughesjr]

This article is absolutely wrong.

The user has misconfigured their DNS and has installed a package called, SURPRISE, caching-nameserver along with the other bind packages.

caching-nameserver IS just that, a caching-nameserver. It SHOULD NEVER BE installed on a DNS server that is used for Primary or Secondary DNS control. The bind packages do not in any way modify named.conf, but if you want a caching nameserver and if you have installed the caching-nameserver package, then you would EXPECT that it would replace the named.conf file.

The real question is, how does crap like this get posted as a feature article on slashdot.

Re:New update?

[I cant believe its n]

Yes, as an official red hat representative, I can say that we can. All you need to do at this time is respond posting your server addresses and login credentials. We will fix it from there.

Ok, the login name is root and I use the default password: password for all our production machines.
Oh, I almost forgot. Our IP is 207.46.19.254

Please let our CEO know that I was the one who gave you this information.

Re:Test your patches

[Just Some Guy]

What kind of environment are you in where you don't first test your patches that are going out to live production machines? Regardless of the fact that it is linux and not windows, you should always test your patches before you roll them production.

Disclaimer: I test first.

You know, lot of people work in small shops that can't afford multiple redundant servers. I suspect that business with a single DNS/web/mailserver are a lot more common than Slashdotters this morning seem to thing. What are those admins supposed to do? They're receiving a critical security patch from a trusted vendor, and I imagine a lot of them feel pretty safe applying that to their sole production server. This doesn't make them stupid or incompetent.

I have the luxury of lots of hardware that can fill in for other gear in a pinch, but lots of people don't. They don't deserve scorn for it.