RHN Bind Update Brings Down RHEL Named

alexs writes "Red Hat's response to update bind through RHN, patching the DNS hole, made a fatal error which will revert all name servers to caching only servers. This meant that anyone running their own DNS service promptly lost all of their DNS records for which they were acting as primary or secondary name servers. Expect quite a few services provided by servers running RHEL to, errr, die until their system administrators can restore their named.conf. Instead of installing etc/named.conf to etc/named.rpmnew, Red Hat moved the current etc/named.conf to etc/named.conf.rpmsave and replaced etc/named.conf with the default caching only configuration. The fix is easy enough, but this is a schoolboy error which I am surprised Red Hat made. Unfortunately we were hit and our servers went down overnight while RHN dropped its bomb and I am frankly surprised there has not been more of an uproar about this."

Comments

You didn't test before deploying an update?

[Anonymous Coward]

So, you didn't test the update on a non-production server? Just install any old patch and let it take your network down? Who do you work for again? I have to make sure not to do business with that.

Re:You didn't test before deploying an update?

[Anonymous Coward]

So, you didn't test the update on a non-production server?...Who do you work for again?

Possibly /., to judge by the non-command of editing on display in the writeup.
I kinda think I know what it's saying. Then again, ambiguity begets problems begets business opportunities, so maybe the lack of clarity is more feature than bug in the long run.

Re:You didn't test before deploying an update?

[suso]

Actually, I caught the error just from looking at the output of up2date/yum. It clearly said named.conf saved to named.conf.rpmsave. So all you have to do is compare what changed, implement any changes and copy named.conf.rpmsave over named.conf.

  Just as I said on the day of the release, be careful, don't just blindly update things.

Re:You didn't test before deploying an update?

[Anonymous Coward]

yeah dumb***.

sysadmin 101 Day 1: No unattended updates. Test the updates.

You weren't hit, you were dumber than the person who made a mistake w/ the package.

Re:You didn't test before deploying an update?

[Anonymous Coward]

True, but the point of the article is that this shouldn't have happened. What if they really didn't have a test server and their boss heard about it and urgently asked them to patch? There's plenty of managers out there trying to teach you how to do your job and who don't bother to pay a bit extra for a test server, even after the production server goes down because of similar mistakes.

Re:You didn't test before deploying an update?

[Anonymous Coward]

Sysadmin 101 Day 1: No unattended updates. Make sure and test your updates.

Re:You didn't test before deploying an update?

[illumin8]

So, you didn't test the update on a non-production server? Just install any old patch and let it take your network down? Who do you work for again? I have to make sure not to do business with that.

No kidding. The only "schoolboy error" as the submitter put it, was not testing the patch on a non-production server before deploying it on a production DNS server.

Re:You didn't test before deploying an update?

[djveer]

I was actually just going to mention that point and what do you know, it's the first comment! I guess i'm not the only one that realizes you should never ever deploy updates randomly to production servers without testing them first under similar conditions.

Re:You didn't test before deploying an update?

[xalorous]

People make mistakes. Even _________ (insert linux vendor here) packagers. This patch should be remembered and trotted out every time a new sysadmin is taught how to do the job. Remember the DNS bind patch of '08? That's why you test before patching production servers.

Oh, and the 'I don't have enough money for a spare of every server' excuse won't play. That's one reason to buy consistent models. Your test equipment can serve as emergency replacements or vice versa. And if all else fails, testing on a virtualized system will help catch something this heinous. As would testing on a workstation class machine with a similar configuration.

The only reason a patch is deployed without testing is that the admin is either lazy or doesn't care.

Re:You didn't test before deploying an update?

[gormanly]

also, this does not happen if you're running BIND in a chroot jail. So those falling to this are doubly dumb.

Re:You didn't test before deploying an update?

[jocknerd]

You know, not everyone has non-production servers. Every server we have IS production. And if you are paying for Red Hat Enterprise, you expect Red Hat to have tested these updates themselves. If this was a Microsoft error, Slashdot would be all over Microsoft for allowing this to happen.

Re:You didn't test before deploying an update?

[Anonymous Coward]

Then it is your bosses responsibility not yours.

Re:You didn't test before deploying an update?

[numbsafari]

And the rest of slashdot would be all over MS admins who blindly update their systems from AutoUpdate.

I find it really hard to believe you don't have at the very least a strawman test system. The fact that you don't says volumes.

Re:You didn't test before deploying an update?

[bucky0]

To be sure, red hat ballsed it up, but if you're running a service that "can't go down", you HAVE to test your patches out. If you don't have spare physical machines, test them in a virtual machine or repartion your workstation to have enough room for a server install.

If it's important enough to not go down, it's important enough to test.

Re:You didn't test before deploying an update?

[Sleepy]

>You know, not everyone has non-production servers. Every server we have IS production. And if you are paying for Red Hat Enterprise, you expect Red Hat to have tested these updates themselves. If this was a Microsoft error, Slashdot would be all over Microsoft for allowing this to happen.

You are wrong; stop whining. You're just painting yourself as misinformed.

1) The updates WERE tested.
2) The admin installed "caching-nameserver", then configured his install to act far outside the default.
3) He allows automatic updates straight into production. So do you it seems. Good luck with that! RHEL documentation says to not do this, but you're a bigshot "paying" for something different. I suggest you get a sidekick, and stick to the Windows side of your "enterprise".
4) He didn't revert his .conf file, as is usually needed when some new line is added to a server .conf. This is SO NORMAL you'd have to be a n00b to get bitten!

Your MS comparison is apples and oranges. If this guy did TEN MINUTES worth of testing he'd realize something's up, and he could revert the rpm package. How many MS updates prohibit uninstall? Quite a few!

In Windows, you can't diff the before & after config, since Windows admins would rather be blind to what they're installing, since that's the norm and it's accepted.

Re:You didn't test before deploying an update?

[GleeBot]

And contrariwise, if it's not important enough to test, then it's not important enough to not go down. So grin and bear it.

Re:You didn't test before deploying an update?

[poot_rootbeer]

You know, not everyone has non-production servers. Every server we have IS production.

Well, there's your problem right there...

Re:You didn't test before deploying an update?

[IchNiSan]

You mean to tell me you don't even have an old desktop machine sitting around with RHEL on it to "play" with? Come on, pull the other leg. Or maybe find a new line of work. Not being able to afford non production servers and test lab is one thing, but not taking the old computer you replaced on the secretaries desk and using that to do some basic testing for mission critical updates is ridiculous. Or hell, just dual boot your machine if it comes to that. You have to do SOME testing of SOME things.

Re:You didn't test before deploying an update?

[Anonymous Coward]

I would've probably done the same thing on my small cluster of servers just out of sheer laziness of letting it auto-update critical patches. I usually do that to enjoy the weekends instead of having to bite my nails to make sure a system has been patched before Monday morning, but really there is no excuse for being lazy these days with VMware and other virtualization products. You could easily recreate your entire environment (minus large data sets of course) on a workstation running VMware Server and just start the VMs you want to test patches on.

Re:You didn't test before deploying an update?

[Dr Caleb]

Perhaps it is his problem, but not his fault. Sounds like he's in the dreaded zone where IT is a necessary evil, not a department that can help leverage the business.

He gets what he needs, or just barely what he needs. When management hands you crap, you learn to make crapade.

Re:You didn't test before deploying an update?

[Anonymous Coward]

You have a pretty bad view of Windows and Windows Admins. Some of that is true, and well deserved. However, you must remember that MANY MANY mid sized business run Windows and have a limited IT dept. That is my situation. I fix printers, client computers, network, AD, Exchange, app servers, website, and most anything else that pertains to a computer. So I don't have time to test EVERY patch for EVERY server and service. It just isn't practical. So I rely on MS to make sure that Windows updates work, and that is why I pay $1000 for each server... cause it's cheaper than another admin.

Re:You didn't test before deploying an update?

[The Cisco Kid]

So run it on another machine that isnt an authoritiative nameserver?

And/or, if your company is so broke it can't afford a couple hundred bucks to put together a low-end box to run update tests on, then you are doomed anyway.

"Free Software", even when serviced and supported by a corporation such as RedHat, is about knowing WTF you are doing and being responsible for your own stuff, as opposed to being a drooling button-pusher assuming everyone else will take care of you and suing them when they don't.

And if this was a Microsoft error, the fault would STILL be on the people who didn't have the sense to test it themselves first, especially when choosing to use MS, which has a history of breaking things. In fact the only reason RedHat's mistake is news to begin with is that fact that this type of thing is so rare to begin with. To be blunt, if MS did it, I would consider it no more newsworthy than 'Sun rises again today'.

Re:You didn't test before deploying an update?

[morcego]

You mean there are people out there running RHEL with non-chrooted bind ? I'm not being sarcastic here. For me, this is simply unbelievable.

I run bind on my notebook for caching purposes (yes, I know about nscd and stuff), and even that one is chrooted.

Re:You didn't test before deploying an update?

[warsql]

My production system is a virtualized system, you insensitive clod!

Re:You didn't test before deploying an update?

[mi]

The only "schoolboy error" [...] was not testing the patch on a non-production server before deploying it on a production

Can the same line be used to defend Microsoft the next time they screw up a bug-fix or "service pack"?

Re:You didn't test before deploying an update?

[turbidostato]

"You mean to tell me you don't even have an old desktop machine sitting around with RHEL on it to "play" with?"

Neither you, it seems, or you would know "an old desktop machine" leaves out so many corners that it just doesn't matter to have it.

This very bug would be avoided by just looking at the update going on; just that. How many authoritative DNServers do you own that you can't afford this?

But then, let's go real: will your "old desktop machine" cover you from a bug regarding hardware interface? (like a kernel upgrade trashing your NIC) err... nope. Will your "old desktop machine" cover you for a bug that only shows itself after some hours/days? err... nope. There's a point were only the "real thing" will be an "equivalent enough" system for testing. And then, you can afford such kind of redundancy when the system is really thrasing money like crazy when stopped. I know we tend to think of ourself we are the masters of the company, that without us, they would crawl to a grinding halt but, while this idea has some merit, it's only a minimal percentage of the installed base that would get profit from such a so-completly redundant environment.

As it has been already stated, real admins will have real world common sense. For this kind of issue, just not allowing for automatical updates, have a look at the system after the update and hope for the best is the most profitable stanza for the vast majority of situations.

And then, if you are using Red Hat you are paying specifically for timeable and working security updates, as in you are not paying for use licenses, you are not paying for software licenses nor software copyright, you are just paying for the convenience of Red Hat itself providing you with tested binary updates. So if their updates are not confiable, what the hell are you paying them for?

Re:You didn't test before deploying an update?

[Wdomburg]

2) The admin installed "caching-nameserver", then configured his install to act far outside the default.

There's no "caching-nameserver" package - just bind, which happens to come with a default config. This was certainly a mistake on Red Hat's part, but yeah - one that any competent admin certainly should have caught. (Not having test environments is a pathetic excuse when your platform comes with virtualization out of the box and not testing immediately after an upgrade - and before upgrading secondaries - is inexcusable.)

Re:You didn't test before deploying an update?

[VGPowerlord]

I may get marked Off Topic or Troll for this, but...

"Free Software", even when serviced and supported by a corporation such as RedHat, is about knowing WTF you are doing and being responsible for your own stuff, as opposed to being a drooling button-pusher assuming everyone else will take care of you and suing them when they don't.

This is precisely why Linux isn't ready for the desktop.

Re:You didn't test before deploying an update?

[Wdomburg]

That's no excuse for not testing a change after you've put it into production. Presumably the company probably has at least one secondary nameservers a well. No way in hell should those have proceeded until after the pilot upgrade was complete.

(And seriously - virtualization is free on every major platform. If you can't get test equipment install a fargin' VM on your desktop.)

Re:You didn't test before deploying an update?

[Dr. Smoove]

Do you admin one single Linux box? Not only is this bad administrative practice (applying patches without testing on AT LEAST a VM) BUT as anyone who uses rpm at all knows, your configs are *usually* moved to .rpmsave. FUCKING COME ON, I almost suspect this is like an Ubuntu mole or something on /.... I can't even believe an admin this shitty made front page with his bullshit complaint.

Re:You didn't test before deploying an update?

[westyvw]

ROFL: I realized that I could save that $1000 and cut my support time in half, by NOT using windows. And I got to see whats REALLY going on. You could too. A test environment isnt going to set you back much (if at all) and you will find that windows forces more patches (and reboots!) then you will ever see using Linux. (I said forces. You can cut back updates to security by wisely choosing the applications you need on a server, such as removing x for example).

Re:You didn't test before deploying an update?

[turbidostato]

"Do you admin one single Linux box?"

About fifty production servers. Not such a big number but still not a short one either.

"Not only is this bad administrative practice"

I know this is not by itself an argument, but I based my administrative practices on a decade and a half of experience. Maybe -and I only say "maybe", I did think about the pros and cons of my strategies prior to put them into execution... and I've been able to test the results against reality after that.

"BUT as anyone who uses rpm at all knows, your configs are *usually* moved to .rpmsave."

Yes. That's why I...
a) Forgot about rpm-based distributions so many years ago (not that this reason is the only one, but it was one among others -clumsy update policies was another one specifically regarding Red Hat)
b) I never allowed for automatic updates on any server I admin
c) Even when I always manually updated the servers I always run find /etc/*.rpm* after it when on a rpm-based target

On a side note, do you really find defensible for a distribution to take away a locally modified configuration file? Abort or really big notice is the only reasonable policy, but it seems Red Hat hasn't learnt it in all this time.

"I almost suspect this is like an Ubuntu mole or something"

Youngsters always tend to think that the world has always been as they see now.

Re:You didn't test before deploying an update?

[Chris Mattern]

You know, not everyone has non-production servers.

Yes. The technical term for those people is "idiots". See also: "Penny wise, pound foolish."

Re:You didn't test before deploying an update?

[k8to]

I dunno.. *craaazy* debian notices if you've changed the config file and gives you a chance to decide what to do. I guess that's too... unprofessional?

Re:You didn't test before deploying an update?

[Dr. Smoove]

lol, man it doesn't matter if I am five years old as long as I maintain a good standard SA practice. 15 years doesn't mean shit if you still admin RH servers and don't know about .rpmsave files. And if you don't enable auto updates, why are you even arguing this point?!?! You actually agree with me and fail to admit it... BESIDES you should be rpm'ing to the path of your cvs/svn tree for dns servers, THEN pushing out to a test server, THEN testing the test server, THEN pushing globally. Failure? O yea that's what cvs/svn rollbacks are for...

Re:You didn't test before deploying an update?

[Anonymous Coward]

How much did it cost your company to have the down time that could have been saved by testing properly?

here are some low cost options:

vmware server is cheep(free).
a usb drive to hold your images.
can anyone say xen kernel?
amazon web services.

Re:You didn't test before deploying an update?

[Gr8Apes]

Can the same line be used to defend Microsoft the next time they screw up a bug-fix or "service pack"?

So we'll be using it tomorrow?

Re:You didn't test before deploying an update?

[Bryansix]

You morons and all the morons who modded you up seem to be living in a fairy tale world. Not all companies can afford production and test servers. Not all System Admins have a team of IT workers. Some of us actually do more then System Admin work and quite frankly considering tens of updates come out a week there is no time to even test things like this. Example: The company I work for just fought back from the brink of destruction when the market for mortgages collapsed. We went to about 25 employees and now are back at 60. We only have four servers all in production. I am the IT Department. Me, myself and I. That's it. Quite frankly I'm wasting a lot of time explaining this shit to morons like you but you pissed me off. Don't just fucking assume that everybody works in blah blah corporate environment with infinite IT budget (or a budget at all) and has a team of people working with them. Considering Small business makes up a big portion of ALL business your assumption is WRONG more then it is right.

Re:You didn't test before deploying an update?

[Bryansix]

What the fuck is wrong with you people? You think every System Admin out there had just one job to do and that's administer the servers? In my job I do everything. VOIP Phones, new employee setup, updates, backups, desktop support, fix the copier, follow up with accounting and executive assistant as to why we ran out of paper yet again etc. etc. etc. The point is the company SHOULD hire another IT person but they can't afford it and there is no freakin way I could ever test every update that comes out. Of course I monitor things and test them to make sure nothing breaks and nothing ever had from an update. But It shouldn't be assumed that everybody has the time to update a virtual machine before they update production and then monitor it so closely as to notice something so small as the problem this patch created. Just stop living in a fairy world will ya?

Re:You didn't test before deploying an update?

[arth1]

You know, not everyone has non-production servers. Every server we have IS production.

Even so, and given you don't have a single old discarded PC to use for test/dev, you still would have at least two DNS servers (there's no excuse for ever running a single DNS server).
So after the first one stopped working, what would prompt you to also install it on the second one?

No, as most others here have said, the responsibility here falls solidly on the "admin", and not on Redhat. I hope his boss reads slashdot, but wouldn't count on it.

Re:You didn't test before deploying an update?

[awrowe]

uhh...I'm sure there is a free Virtual box you can use, whats it called again? Oh yeah, thats it, Virtual Box!

It even runs linux!

Re:You didn't test before deploying an update?

[Bryansix]

Yes and Virtual PC 2007 runs linux too. However this adds nothing to the discussion because it was not my point. My point is time constraints and manpower constraints.

Re:You didn't test before deploying an update?

[Bryansix]

Oh also, the virtualbox binary is NOT free. The source is open but you still have to compile the damned thing yourself and it assumes a development environment with about 20 different programs pre-installed. I know because I tried to compile it. It's not as simple as just typing "make install".

Re:You didn't test before deploying an update?

[Metaphorically]

If I can do a diff before I patch...

Re:You didn't test before deploying an update?

[Tenareth]

You will reduce your manpower requirements if you actually plan your installs.

How much manpower does it take to recover from a failed production server?

Re:You didn't test before deploying an update?

[Bryansix]

Well considering an update never made any one of my servers fail I get a divide by zero error and you fail!

Re:You didn't test before deploying an update?

[Anonymous Coward]

You can't even set up some virtual machines? Come on! I do it to make and break stuff and it's fun also.

Re:You didn't test before deploying an update?

[ebuck]

You are right about making do with what you get, but exactly how did he lack resources in this case? He already has RHEL (and updates, so I'm guessing his support contract is up to date).

It's not like they're charging more for a non-caching domain name services server. In fact, he took a perfectly good non-caching name server, and then installed pre-packaged configuration files to make it a caching-nameserver. Then he started hacking away at the config file. Small wonder that fixes to the caching-nameserver config file will interfere with his setup. If the world worked any other way then caching-nameserver config files would never get bug fixes, ever.

He didn't know what he installed, ignored his vendor's documentation warning not to do it this way, ignored the name of the package he was installing, ignored the concept of production in the enterprise (no updates without testing), didn't bother to read RPM's log files, and restored to fire-fighting in an emergency "failure" scenario. There's half-a-dozen routine ways this could have been avoided, but he made mistakes along every step of the way.

In his favor, this sysadmin has balls. After being ignorant of his missteps, he's complaining that RPM saved a copy of his altered config file! I'll bet he won't even diff the changes into it before copying it back to it's original name.

Give this man a fish and he'll complain that you're ruining his diet. Teach him how to fish and he'll complain that you're dumping your fishing responsibilities on him. He just doesn't get it.

Re:You didn't test before deploying an update?

[pigwin32]

Having a test environment is sysadmin 101. It doesn't even have to be equivalent hardware to get the benefit. $500 will buy you a test "server" and there are free/cheap virtualisation options that will allow you to run multiple test environments on a single machine, even your own desktop machine.

There are any number of hackneyed expressions I could roll out but if you're not testing stuff at some point you will come to grief. So this is an issue of risk and if you've evaluated the risk and communicated the risk to your organisation and the business is happy to carry the risk then that is a valid approach. If you haven't evaluated the risk and are calling other admins morons because they take a different approach your argument holds no water.

Re:You didn't test before deploying an update?

[Follis]

"What the fuck is wrong with you people?" They're professionals. By your commenting style and overall whiny tone, I get the impression you're not.

Re:You didn't test before deploying an update?

[Bryansix]

No no no. I never said it was not a good practice to test updates. I think it is a great practice. The problem comes down to man hours because I'm the only IT person in the company and I have to do so much else besides sysadmin stuff. Everytime an employee starts it creates at least half an hour of work for me but probably more. We are mostly sales staff so people start all the time (and conversely are let go or leave all the time too).

So if a company has more then one IT person and that person is doing sysadmin stuff 90% of the time then they should test updates. If not then it can hardly be expected to happen. My point is these people talk like if you don't do it then you suck. Well no I don't suck; I just don't have time.

Re:You didn't test before deploying an update?

[Bryansix]

Obviously you've never worked at a company that has roots selling mortgages. People fucking cuss all the time here.

Re:You didn't test before deploying an update?

[Bryansix]

You know what I can't leave it at that. You are a jackass and you are the kind of people my comment was pointed towards. You think that because you work for a company with "sky high IT budget" and "Unlimited IT Hiring policies" that everyone is in a similar situation. Well we are not and you can't take all that for granted and then sit back and whine and bitch and moan that people are not doing their jobs. The point is you don't know shit about specific situation and you can't know why something was or was not done. Stop arm-chair quarterbacking.

Besides I like it here. Here where I fly by the seat of my pants is where the shit gets done. I don't reference billions of policy guidelines when doing my job. I MAKE policy and I MAKE decisions in a timely fashion and I GET SHIT DONE way quicker then many enterprise level IT departments can. This is where life is real and fantasy worlds fly out the window.

Re:You didn't test before deploying an update?

[pigwin32]

You have 4 servers, they need to be patched, (not) applying a patch is a risk. To mitigate the risk you first need to assign some metric to the risk. Typically this is a number from a table that assigns values to likelihood x impact. The likelihood of a patch breaking your server may be very low but the impact may be very high, resulting in a high risk. Having evaluated the risk you need to mitigate the risk appropriately. This might be as simple as maintaining a ghost image of the system prior to applying a patch. The next thing you need to do is to communicate this with the business. If a server breaks and the business is screaming for a resolution it helps if you can put some paper in front of them that shows exactly what was agreed. Not having time to do it correctly is not normally an acceptable excuse and name calling is not normally an acceptable form of argument.

Re:You didn't test before deploying an update?

[pxc]

You can download the OSE version from your distributor, and the closed-source version is free-as-in-beer.

Re:You didn't test before deploying an update?

[mysidia]

No kidding. The only "schoolboy error" as the submitter put it, was not testing the patch on a non-production server before deploying it on a production DNS server.

No, if this broke your DNS config, the schoolboy error would be: not having redundant DNS servers for your authoritative zones.

You verify redundancy before you download the update, then you download the update, then you turn off the DNS service, then you install the update.

Then you check that everything is in order.

Then you turn DNS back on. Then you verify proper server operation, IMMEDIATELY. (not 5, 10, 15 minutes later)

Most networks do not have a hive of spare servers around for the purpose of testing minor updates to DNS server software.

Even if the update works well on your test server, you can't be sure it works out the same way on the real server.

You choose a good software vendor, and you expect them to do the basic testing of their patches before they release them.

You don't waste time testing things that don't need to be tested.

If you're making major updates to critical non-redundant services, you wait for a maintenance window, and apply your update in a manner that allows a rollback, if something breaks.

Re:You didn't test before deploying an update?

[The Cisco Kid]

Actually, its why the average consumer isn't ready to see a computer as anything other than a toy or an 'applicance'.

Perhaps the concept of 'computer' needs to diverge - one fork will be for the appliance and toy crowd - the other will be for those that use it as a tool and actually might be bothered to RTFM and try to educate themselves..

The former need not ever bother with learning anything other than button-pushing, but when it blows up, they'll need to be prepared to spend some money to pay folk from the latter to 'make it all better'

In fact, I almost say the divergence has already happened. Microsoft is on one fork, Free Software is on the other. People from the ignorant fork will always slowly migrate to the other fork.

Those of us in that fork don't *WANT* a toy or an appliance where responsibility (and therefore control) is taken away from us. (BTW, having just seen WALL-E, there is part of that movie that would make an excellent analogy, feel free to watch it to see why)

Re:You didn't test before deploying an update?

[hearnz]

You don't have time and/or budget to do things properly? That's fine - but realise that doing things that way means that, sooner or later, you WILL have outages and problems. That's a risk, and you need to mitigate it somehow. If a certain number of hours a year of downtime through lack of testing is a "better" alternative for your business than the expense of the staff/resources to do things properly, that may be an acceptable choice, but don't whine when things DO go wrong.

However, if you expect untested patches and upgrades to always apply successfully to your production servers without incident, then you are simply a fool. Please don't ever come and work for me.

Re:You didn't test before deploying an update?

[Anonymous Coward]

You guys would be riding MS hard if they screwed this up. Poor MS is the victim this time a victim to a site with double standards.

Re:You didn't test before deploying an update?

[spankymm]

If your bosses won't pay for some test servers, pick your most lightly-loaded server and install a VM.

You can't keep a network running without at least one scratch box.

Yes, RH should have tested this update, but you should *NOT* trust any vendor to get it right 100% of the time.

Re:You didn't test before deploying an update?

[VGPowerlord]

Some of us prefer to not have things not work correctly and then have to search for hours on the web as to why. While this sometimes happens in Windows, I've found that it happens much more often in Linux, particularly during the setup phase. Even in newer editions, such as Ubuntu 8.04.

For example, I had to find out why my sound card didn't work (it defaulted to digital output)... or why switching to the ATI binary drivers killed desktop animations (xserver-xgl (I think) needs to be installed first)... or why changing the screen resolution made X stop working with both the default and ATI binary video drivers (well, that was in Ubuntu 7.10, I avoided changing it in 8.04 to avoid this problem).

For the record, my machines use the following operating systems:
Server box: Debian Etch
Home PC: Windows
Work PC: Windows, but I don't get a choice in the matter there.

Re:You didn't test before deploying an update?

[donaldm]

You know, not everyone has non-production servers. Every server we have IS production. And if you are paying for Red Hat Enterprise, you expect Red Hat to have tested these updates themselves. If this was a Microsoft error, Slashdot would be all over Microsoft for allowing this to happen.

Any firm that requires computers and does not provide a non-production environment is asking to have problems. To these firms I ask a simple question "What price do you place on your data and how long can you afford to have your system(s) down while your IT people do maintenance? Another question I normally ask is do you have an IT disaster recovery plan in place such as backups and have you tested them?

I can ask lots more but I suppose the best question to ask is does your firm have a "change request" system and if it does why do you blindly update all systems without testing. Just paying for as subscription does not absolve the customer from exercising professional care.

Re:You didn't test before deploying an update?

[sjames]

Neither you, it seems, or you would know "an old desktop machine" leaves out so many corners that it just doesn't matter to have it.

I don't know that I'd go that far! It leaves out corners to be sure, but it's still not a bad precaution.

As for RedHat, the whole reason this is news is that they don't typically have this problem. Yeah, it shouldn't have happened. They owe a fair number of admins an apology. But so long as this doesn't become a habit for them (I doubt it will), it's just "one of those things".

Re:You didn't test before deploying an update?

[The Cisco Kid]

You need to decice wether you want an appliance that will have a very limited set of features, and which the maker will decide what you are allowed to do with it and/or the data it contains (wether its your's or data that you 'licensed'), but will make sure everything 'just works' (well, except when it doesnt - in which case you wont be able to research and correct the problem yourself because you will have no access and everything will be closed, and will instead have to rely on the vendor to fix it, assuming they arent too busy selling more or working on their next project

or

a tool that you control, for which you expect to use (and gasp!, possibly expand) your own knowledge and skills to setup, maintain, and operate, but over which you will have complete control, and no external entity (RIAA, MPAA, Govt, hackers) will have any means to dictate otherwise.

Right now both of these scenarios overlap, to some degree - one is moving more in one direction, and one in the other.

Re:You didn't test before deploying an update?

[Anonymous Coward]

Yeah totally agree here....you don't take w/e patch and just toss it out and hope for the best. RH is operated by humans just as capable of a simple mistake as you are.

As for the uproar comment...what did your boss say about your patch practices? ;)

Re:You didn't test before deploying an update?

[turbidostato]

"man it doesn't matter if I am five years old as long as I maintain a good standard SA practice."

That's right. Experience does matter when you are to overule a sensible standard SA practice with another better fitted to current environment.

"You actually agree with me and fail to admit it..."

Not at all. It seems you forgot what the point for this thread was. I'll refresh your memory: it is about the approach that "if you don't have a test system for each and every update you must be an incompetent sysadmin". My point is that "an old PC for a test system" doesn't hold water and don't produce a better output than a sensible sysadmin doing properly his job.

"you should be rpm'ing to the path of your cvs/svn tree for dns servers, THEN pushing out to a test server, THEN testing the test server, THEN pushing globally. Failure? O yea that's what cvs/svn rollbacks are for..."

And my point is that while you are still convincing management for all this expenditure (both machinery and time), I already finished my maintenance tasks and I'm doing productive work (and yes, I maintain configuration files within a SVN-based repository).

Re:You didn't test before deploying an update?

[Bryansix]

Of course I realize the problems with not testing everything. I can fix outage quickly and it's not a big deal. I'm not advocating never testing things like updates. Don't say stupid shit like don't come work for you. I work in this environment now but if I worked for big IT dept. at some corporation then I would go by the book and do things right. I just do it this way because I can't do it any other way. My point was to show that most people were making lame assumptions about the environments people are working in.

Re:You didn't test before deploying an update?

[xalorous]

:) Hmmm. I was about to reply, but realized there was a reference in there.

Re:You didn't test before deploying an update?

[Bryansix]

You crack me up. You just don't get it. If I don't have the time to do it I don't have the time to do it. If I work for a company of 60 people (45 of whom are sales staff) then the expectations are going to be lower across the board. The point is so much is expected of me to get done that is basically data entry and managing resources that system admin stuff is relegated to my free time and when things break. Just a little history there used to be another IT consultant who worked 2 days a week plus we had a contract with a consulting agency that did work on demand when we needed it with us guaranteeing so many hours a month. Now none of that exists and it's just me so get off your high horse and stop promoting that stereotype that all IT people are jerks.

Re:You didn't test before deploying an update?

[Bryansix]

Actually NO IT IS NOT. It's free for PERSONAL USE ONLY. Read the EULA.

Re:You didn't test before deploying an update?

[Anonymous Coward]

Frankly when your arguments start and end with ad hominem attacks your credibility is pretty much non-existent. I have worked with people like you who are so busy that they don't have time to do their jobs properly but they have plenty of time to tell you about it, and they're constantly fighting fires. I have never once found a situation where much effort couldn't be saved by a little automation.

A schoolboy error?

[something_wicked_thi]

What? And isn't it an error of similar proportion to upgrade your primary DNS servers without first testing the new install?

Re:A schoolboy error?

[imipak]

Note as well that the initial release included a default conf file which specified a fixed source port, which of course breaks the fix.

[Updated 10th July 2008] We have updated the Enterprise Linux 5 packages in this advisory. The default and sample caching-nameserver configuration files have been updated so that they do not specify a fixed query-source port. Administrators wishing to take advantage of randomized UDP source ports should check their configuration file to ensure they have not specified fixed query-source ports.

Personally I'm surprised there's not been more uproar about the requirement to move internal DNS servers (yes, that means your Windows Domain Controllers in most corporate environments) outside any NAT'ing devices (eg: firewalls), as many NATs also break the fix by rewriting outbound UDP DNS queries to use the same or incremental source ports, which also breaks the fixes. Anyone here moved their AD outside the firewall?

Re:A schoolboy error?

[geekymachoman]

IMHO, rhel should have tested this. If they announce offical new version of some app, and it turns out to brake stuff, then that's theirs fault. Maybe admins should stop being lazy, and start installing 'network' services which are crucial server components, by hand, from source, which gives them more flexibility and control then rpm -i.. No ? Ok then, chrooting ? I chroot my bind's, so, if I update, the rpm package can only mess up files like /etc/named.conf which I don't use, since all of config files are in /chroot/named/ and redhat isnt aware of that. Binary file has a switch to start from chroot... you edit /etc/sysconfig/bind or something like that, and do /etc/init.d/named start - Thats it. If you can easyli chroot some service (like named), then do it. You can only gain something from it.

Re:A schoolboy error?

[something_wicked_thi]

IMHO, rhel should have tested this.

'Course they should. Nobody said otherwise.

I'm not sure what you're getting at with building from sources. Seems like overkill and doesn't solve the main problem because you can still screw it up. All anyone's saying is that you should test this on a server that you don't care about, or at least test it on one, before upgrading all of them.

Re:A schoolboy error?

[evilviper]

Personally I'm surprised there's not been more uproar about the requirement to move internal DNS servers (yes, that means your Windows Domain Controllers in most corporate environments) outside any NAT'ing devices (eg: firewalls),

NAT is not a firewall.
A firewall is not NAT.

I wouldn't think that practically any major sites are running their public-facing DNS servers from behind a NAT (though I expect most are behind a firewall).

Re:A schoolboy error?

[db32]

Personally I'm surprised that this is getting modded Informative. I suppose the NAT piece is informative, but I think "Anyone here moved their AD outside the firewall?" qualifies as either -1 Job or +5 Funny.

Re:A schoolboy error?

[igb]

Hand off DNS queries emerging from AD servers inside your firewall to caching-only servers in your DMZ. I have all my AD servers on RFC1918 IP numbers with no NAT, because they strike me as devices I'd prefer to keep as far away from the big bad Internet as possible.

ian

Re:A schoolboy error?

[Anonymous Coward]

I run 2 domains from my house and have 1 DNS server while a friend serves secondary for me. I ran the patch and was hit as well. It was so devastating. I had to actually type in the the 2 zone records into named.conf. OMG the horror. Pity I can't have test servers at home.... Luckily at work where there are MANY more domains I have test machines.....

Re:A schoolboy error?

[oyenstikker]

What is wrong with running behind a NAT?

w.x.y.z -> heavy duty router -> 10.x.y.z

Re:A schoolboy error?

[Anonymous Coward]

Personally I'm surprised there's not been more uproar about the requirement to move internal DNS servers (yes, that means your Windows Domain Controllers in most corporate environments) outside any NAT'ing devices (eg: firewalls), as many NATs also break the fix by rewriting outbound UDP DNS queries to use the same or incremental source ports, which also breaks the fixes. Anyone here moved their AD outside the firewall?

Actually, DNS doesn't need to be running on a domain controller, but that's beside the point.

It's generally risky to put a domain controller on the internet. In the event of a compromise you've lost quite a lot.

A better idea would be to get a better NAT device - like OpenBSD. OpenBSD will assign a new random UDP port for each transaction. Unpatched DNS servers behind NAT devices that behave like this aren't vulnerable.

Unfortunately, as everyone knows, OpenBSD is dying. Netcraft confirms it.

Re:A schoolboy error?

[imipak]

It's not just public-facing DNS servers that are vulnerable. Think about it. Dr Evil spams your entire company with a web-bug infested mail, or just a link to the domain he wants to poison. Ten seconds later he starts flinging spoofed poison DNS responses at the outside of your NAT (if you have one.) Result: misery!

Re:A schoolboy error?

[imipak]

You are right, of course, (another alternative is a patched BIND outside your broken f/w), but somehow I doubt many Windows-only shops are going to get far building an OpenBSD-based NATing pf config from scratch in the next week or two.

Re:A schoolboy error?

[imipak]

I don't know why you find it so funny, that's the best advice going for a workaround, until the vendors have a patch. See the Vixie story linked above. Unless you'd prefer to push hosts files to all your users every morning?

Re:A schoolboy error?

[db32]

It seems to me that moving your AD outside the firewall would be monumentally stupid. Moving your AD outside the firewall would imply putting a domain controller outside the firewall, which pretty much defeats the whole point of the firewall being there. Sidewinder firewalls do something called split DNS. It has a DNS server running on the inside interface and on the outside interface. The DNS server on the inside interface forwards all requests to the DNS server on the outside interface. You configure your AD to forward to the inside interface DNS server. The inside interface slaves to the AD DNS and the outside can really be master/slave to your external DNS. (Of course, this type of setup requires that you don't do something stupid like making your entire internal DNS structure externally resolvable). So the only DNS server talking to anything on the outside is the outside interface DNS server on the sidewinder. Patch that one, and don't expose your AD structure to the world. This also has the added benefit of stopping any of that DNS tunneling stupidity that results from allowing your internal computers to directly query any external nameserver. You shouldn't be allowing DNS traffic through your firewall like that.

So no, putting your AD outside the firewall is incredibly bad advice. Engineering your DNS in a secure manner is good advice.

Re:A schoolboy error?

[evilviper]

What is wrong with running behind a NAT?

Overhead...
Additional point of failure...
Pointlessness and Futility...

It's not a crime against humanity, but there's little or no good reasons to do it, and a few reasons not to.

Re:A schoolboy error?

[the_B0fh]

Ah. Cisco weenie. Luckily, other firewalls and other DMZ structures do not require stupid natting crap.

MS

[FozE_Bear]

If it was a Microsoft product, we'd all be carrying pitchforks and torches....

Re:MS

[sleekware]

Unless you are one of the slaves who is forced to use Microsoft, and therefore your protest abilities are rather slim to none.

Re:MS

[Anonymous Coward]

But nobody is using RH in production anyway.

Re:MS

[Anonymous Coward]

Yes... because businesses use Microsoft products. There's only one person complaining since nobody uses RHEL.

Re:MS

[GNUALMAFUERTE]

Well, one thing is for sure: nobody should.

Long live Slackware! ( And long live old-school sysadmins that compile their own software )

We have proved to be right, once again.

Re:MS

[Minwee]

If it was a Microsoft product, we'd all be carrying pitchforks and torches....

If it was a Microsoft product, we'd still be waiting for the patch.

Re:MS

[hughesjr]

sure, if it was really a bug ... in this case, it is totally user error and the installing of a package called caching-nameserver.

Re:MS

[prandal]

MS08-037 was released on the same day, and was much loved by ZoneAlarm users :-)

Re:MS

[altamira]

Ehm, no. The patch for this precise issue has been released by MS on July 8 (953230 MS08-037), more than a week ago.

Re:MS

[Eighty7]

If it was a Microsoft product, we'd all be carrying pitchforks and torches....

To put this in /. terms, Red Hat's got good karma. Contributing to the community for over a decade kinda does that.

Re:MS

[Anonymous Coward]

Other security products worked just fine with that update. Zonealarm is a piece of shit.

Re:MS

[unity100]

thousands of web hosting companies use RH to serve millions of websites. you are clueless about the market pal. rh one of the most popular web host platform choices.

Re:MS

[debatem1]

looks like you fell into the sarchasm.

Re:MS

[debatem1]

Yup. I've got all the torches, can you grab an extra pitchfork? I'll pay you back when we get to the castle.

Seems like the general opinion is that no admin worthy of avoiding the boiling oil treatment wouldn't have applied the patch blindly to a production environment, but it still doesn't let RedHat off the hook.

Re:MS

[Sleepy]

I'm sorry, this isn't a bug. You just don't understand servers I guess. Let me explain:

When you customize a server's configuration file, you save the .conf file somewhere safe.
You might even copy it to another system.

When you roll out updates, it is ROUTINE for the new software to backup the old conf file and install a new one.
This is completely standard.

If you've done ANY customizing in your conf file, you don't want to lose it: you diff the .rpmsave vs the new, and copy in the old settings (or copy the old file over the new, if there are no major additions to the conf). In UNIX, you keep your config just like in Windows you reboot for everything. It's part of the process.

Even a 1st year novice admin knows this! And my statements here fall WELL SHORT of what some people are suggesting (a pair of upgrade-test servers that soak the release before you go live*).

I'm sorry, but "alexs" is a DUMMY. He does not know "best practices". He compounds his mistake by complaining and pointing blams on /.
If alexs /. handle can be associated with his real name, someday he will be embarrassed by this when a job interviewer brings up this episode.

(BTW, if it were a Microsoft product, you'd have NO WAY of auditing the changes.. so you could never get by without more testing than I outlined here).

Re:MS

[Anonymous Coward]

MS08-037 was released on the same day, and was much loved by ZoneAlarm users :-)

Considering the only application that had a problem was Zone Alarm, I would consider that a success and place fault with Zone Alarm programming, and not anything with MS. With the many numbers of firewall applications, only one breaks? Hurray!

Re:MS

[alien_life_form]

When you roll out updates, it is ROUTINE for the new software to backup the old conf file and install a new one.
This is completely standard.

Not quite. What I'd expect is a named.conf.rpmnew with the old config in place. My feeling is that the .rpmsave thing should never be done unless if explicitely accepted.

Example of why: fubar.2.0 makes a few incompatible config changes (a no-no in itself) over 1.99. Yum pulls the .rpmsave trick as part of a 25+ packages upgrade. You copy .rpmsave over. You lose. (Dovecot did this a few times).

As for having a test server where you roll out the updates I'd say that the shops that can afford to:

i) mirror all their servers to an exact replica (it has to be current and exact, or whatever testing yo do will be void).

ii) have the time and resources to run the update/test/deploy

are not in the majority. This said I'd say that

yum update bind

rather than

yum -y update

would have been a lot wiser.

Cheers,
alf

Re:MS

[Anonymous Coward]

"When you roll out updates, it is ROUTINE for the new software to backup the old conf file and install a new one.
This is completely standard."

Sorry, but you are wrong.

The standard routine from properly functioning software is to have a look at the MD5sum of every configuration file; if it's the same than that of the previous version package, then you can just overwrite it with the new one. If it is not, then assumen it has been manually managed by the sysadmin and stop, offer him a diff (preferably a three way diff) and ask for his input.

Re:MS

[Draek]

But since it's Linux we're all happy that Red Hat fixed an important SECURITY bug, while only introducing a minor (AKA, non-security) one so fast, since at least we won't get rooted.

Re:MS

[Draek]

Example of why: fubar.2.0 makes a few incompatible config changes (a no-no in itself) over 1.99. Yum pulls the .rpmsave trick as part of a 25+ packages upgrade. You copy .rpmsave over. You lose. (Dovecot did this a few times).

That's precisely the reason why .rpmsave is used instead of .rpmnew, so that if the config file's syntax suddenly changes you've at least got the thing working in a default config. And you're not supposed to copy the .rpmsave over, you're supposed to diff both files, see what's changed, and copy the relevant lines over, it's just that if there hasn't been any syntax change, copying the file over amounts to pretty much the same thing, but that's a pretty big *if*, specially for a production server.

Re:MS

[drsmithy]

When you roll out updates, it is ROUTINE for the new software to backup the old conf file and install a new one.
This is completely standard.

No, it's completely crazy. As this bug demonstrates.

What should happen (and what does in RH systems, AFAIK) is that the new .conf file is created with a different extension (.conf.rpmnew) and then you diff *that* with your existing, customised config file.

Default behaviour should be as non-intrusive and - especially - as non-destructive as possible. This is referred to as "the principle of least amazement". Overwriting existing, customised files completely fails it.

The admin is not without fault for not testing and backing up his config, but Red Hat are much more at fault for *destroying end user data*.

(BTW, if it were a Microsoft product, you'd have NO WAY of auditing the changes.. so you could never get by without more testing than I outlined here).

FUD. Of course you could.

Re:MS

[prandal]

If you'd had selinux enabled, a

yum update bind

would have resulted in a broken setup.

Why do you think they released updated selinux policies at the same time?

Re:MS

[donaldm]

Actually when using Redhat yum is only standard when you have RHEL5 although if you like you can put yum on RHEL4 (I have not tried this on 2 or 3) and it does work. As for auto updates this may be fine for the home user (like Microsoft update) but definately not for any business where any update must be done by hand only after the requites approval is given. Just updating without any formal approval such as a change request or even an email from the appropriate person to do this is just plain stupid.

It is very easy to setup and configure a private "yum" repository that can be customised for the business and by doing this you can have the latest "base" releases and the latest updates. In fact it is easy to lock an update and over a few days or even months update all appropriate machines to exactly the same update. The only downside is you may need a few hundred GB of disk space (mirrored or at least Raid 5). Any firm that will quibble over a few hundred dollars for the extra disk space and a possible thousand dollars for the yum server which can be used for other things as well is too cheap to even be in business.

Re:MS

[alien_life_form]

Yet another reason to run with SELINUX=0, if I ever needed more.

Cheers,
alf

Re:MS

[alien_life_form]

[...] And you're not supposed to copy the .rpmsave over, you're supposed to diff both files, [...]

Yes I know that, but that's not my point. Which is: upgrades involving icompatible config changes a.k.a .rpmsave should never happen without explicit approval.

Which would give you the chance to deny it, read up on what's happened, mnerge7test the new config, and THEN deploy it. The way things are, a "slip of yum" leaves you with a disabled system until you figure out the needed changes - which may be hours.

Re:MS

[sjames]

Not having a long history of releasing updates that screw things up also helps. Anyone can make a mistake once in a while. When it becomes the rule rather than the exception, the pitchforks and torches come out.

What kind of an idiot would...?

[Anonymous Coward]

What kind of an idiot would do something like that? You'd think they're running RHEL or something.

Re:What kind of an idiot would...?

[ThePhilips]

On most (all?) other distros it works perfectly. I had Debian for ages in production (supporting piles of services) with apt-get update/upgrade running regularly. SuSE and Gentoo also do good job keeping you informed about changes in updates and if post-update human interaction is needed.

The crucial difference here is mindset of RH. It didn't changed the damm yota in the decade. The very same problem why I threw away RH6/7 in past from production, the very same stupidity of RH, is still there.

RH is only distro I have ever tried - and I tried many of them - would silently without any warning or prompt replace your config files with shipped version. It took them ages to learn that files can be renamed - yet it didn't went thru completely it seems.

This is not a single mistake. This is happening now for more than a decade now: RH during maintenance can and does override your configuration. The RH folks simply have no trivial respect to their users...

[/rants]

Re:What kind of an idiot would...?

[nabsltd]

RH is only distro I have ever tried - and I tried many of them - would silently without any warning or prompt replace your config files with shipped version.

First, it doesn't do this without any warning...the output of rpm (which does the actual install) is forward to yum, or rhn, or whatever is running the "figure out everything I need and get it" process, and that is displayed to you when you are applying the patch. It clearly states in that output what happened with the file.

Second, for some updates (particularly security updates like this one), it is appropriate to save the old config file and load a default one, especially if that default one helps provide more security. Then, the admin can figure out what parts of the new default should be applied to their config, merge everything together, and restart the service.

These are the kinds of procedures that good admins do when they make changes to the system in any way.

Re:What kind of an idiot would...?

[Anonymous Coward]

I agree with the parent.

RH quality has been slipping for some time now. Heck, I don't even know what RH brings to the table as far as distributions are concerned.

Their GUI system-config-* programs are so poorly written that a lot of them don't even work. And those that do can't repaint themselves properly on the screen. Imagine GUI programs that take up to 30 seconds to (re)paint themselves after being covered/exposed. And that's on a beefy dual-core PC!

I don't know so much about their enterprise stuff (thankfully I only have to deal with a single RH server) but Fedora certainly sucks ass. I don't think RH does *any* testing on it - that's for suckers ^H^H^H er...the community to do. Every time there is update, I say to myself "Okay...what did they break this time?"

Re:What kind of an idiot would...?

[eh2o]

On Debian and its derivatives the upgrade process is suspended when a config file difference is detected--the admin is then given an interactive prompt where they can inspect the difference, accept or reject the new version, drop to a console, or cancel the upgrade entirely. Since these are the kind of procedures that good admins do... why not make it easy and quick for them to do it, when they need to do it? The RH system is a poor implementation from a usability perspective.

Re:What kind of an idiot would...?

[the_B0fh]

Remember, this is the company that agreed that rpm has a bug that can cause corruption, and yet closed the case out with a "WILL NOT FIX".

Respect their users? Bwahahahahaha. They want to be the Microsoft of linux, and don't you forget that.

Who is the last major distro to join any standardization efforts? Bah.

Re:What kind of an idiot would...?

[donaldm]

Redhat 6/7? How long ago was that? See the following release dates I think you will find Redhat has improved since 1999/2000.

Professional Redhat releases RHEL2.1 (26 Mar 02), RH3 (22 Oct 03) through RHEL3.U9, RH4 (15 Feb 05) through RHEL4.U6 and now RHEL5 (14 Mar 07) through RHEL5.U2 although the above link does provide more information.

If people want a Linux distribution that has full software support they either go to Redhat or Novel (SuSE). There are firms that will provide support but the Redhat followed by Novel have the lion's share of the market.

But, you plan on making one!

[C_Kode]

I am frankly surprised there has not been more of an uproar about this. ...but you plan on making one! OOH-RA!

Re:But, you plan on making one!

[MikeDawg]

If you don't check out how neat the RHN satellite server, or the new spacewalk server is, you're really missing out. It is really nice in the enterprise environment.

New update?

[KasperMeerts]

Why do we have to fix this ourselves. Can't RHT just issue a new update fixing it?
Also, don't they test their updates?

Re:New update?

[CrackerJackz]

Because the named.conf file gets stomped, the 'backup' RPMSAVE file it creates is the caching-only file, not the original named.conf file.

I caught this a couple of weeks ago on a test server (where *all* patches should be tested first, Microsoft or otherwise) best way to fix? cp /etc/named.conf /root/named.conf.backup ; up2date-nox -u ; cp /root/named.conf.backup /etc/named.conf ; /etc/init.d/named restart

Little to no downtime on the prod servers :)

Re:New update?

[Anonymous Coward]

Yes, as an official red hat representative, I can say that we can. All you need to do at this time is respond posting your server addresses and login credentials. We will fix it from there.

Re:New update?

[I cant believe its n]

Yes, as an official red hat representative, I can say that we can. All you need to do at this time is respond posting your server addresses and login credentials. We will fix it from there.

Ok, the login name is root and I use the default password: password for all our production machines.
Oh, I almost forgot. Our IP is 207.46.19.254

Please let our CEO know that I was the one who gave you this information.

Re:New update?

[Anonymous Coward]

server: batman.slashdot.org
ueser: cmdrtaco
password: sex69
Thanks!

Re:New update?

[Anonymous Coward]

Our IP is 207.46.19.254

OrgName: Microsoft Corp
OrgID: MSFT
Address: One Microsoft Way
City: Redmond
StateProv: WA
PostalCode: 98052
Country: US

NetRange: 207.46.0.0 - 207.46.255.255
CIDR: 207.46.0.0/16
NetName: MICROSOFT-GLOBAL-NET
NetHandle: NET-207-46-0-0-1
Parent: NET-207-0-0-0-0
NetType: Direct Assignment
NameServer: NS1.MSFT.NET
NameServer: NS5.MSFT.NET
NameServer: NS2.MSFT.NET
NameServer: NS3.MSFT.NET
NameServer: NS4.MSFT.NET

Re:New update?

[HiThere]

Thanks, I'd been wondering whose IP he'd chosen.
(But I'd probably have been too lazy to check.)

Re:New update?

[jdogalt]

If this comment is accurate, please mod it up. It adds needed clarification to the understanding of the issue one gets reading the comments. I.e. this makes it sound like there is a scenario where you lose your named.conf, as opposed to the story and the comments above that all make it sound like the only negative is that your conf gets moved to .rpmsave. (yes, I understand the whole caching-nameserver aspect. I however don't think it's counterintuitive for an inexperienced admin to think that they could have that package installed, and still run their own domains. Now, back in the day or distro when the package was named caching-*only*-nameserver... maybe... But even still, I think RH should support the scenario where someone installs caching-nameserver, then reads docs, then extends their config files so that their system is providing a caching-nameserver AND authoritative or secondary for things, and then when they update to errata, their config should not get blown away. $0.02...)

They are all busy ...

[PolygamousRanchKid]

I am frankly surprised there has not been more of an uproar about this.

... maybe they are all still busy fixing their servers, and don't have time to post now?

Re:They are all busy ...

[Anonymous Coward]

unknown host slashdot.org

applying patches without testing them?

[BenLutgens]

So unless I miss my guess, these patches took down your production DNS servers. This leads me to believe you are applying patches blind, without testing them.

Serves you right. Submit a bug report and quit whining.

Re:applying patches without testing them?

[sleekware]

Guess he didn't have a lab (or decided not to use it). I've been guilty of that before... :P

what, it's not tagged with "haha"

[Anonymous Coward]

Linux fanbois strike again

TCO

[toxygen01]

I wonder if this is included in Total Cost of Ownership. i.e. I'm really interested in estimates how big loss this mistake generated to big companies.

Re:TCO

[msuarezalvarez]

The accountants doing the computation just called and they need a little more time before they have a good estimate on the cost of this: they are waiting for their computers to finish defragmenting their disks and for their antivirus apps to scan every word file for vba worms.

Re:TCO

[Anonymous Coward]

I work for a big company, and the initial patch cost us about 10 seconds of downtime per server. Because we didnt patch all the DNS servers at the same time, there were no interuptions in service. Because we do not install the caching-nameserver package on our production DNS servers, this "issue" did not effect us.

I would have to guess that it cost us $0, since patching servers is part of our normal duties ...

Re:TCO

[arth1]

I wonder if this is included in Total Cost of Ownership. i.e. I'm really interested in estimates how big loss this mistake generated to big companies.

Absolutely no loss. Because big companies would test this out in a test environment (and possibly also a staging environment) before installing the patch on production servers.

Even smaller businesses who are foolish enough to install directly in production would do a simple test or two after install to see whether it was working, before going home. They would discover it's not, and it should take no more than 5-10 minutes to identify the problem and fix it.

In other words, it is a non-issue that would only be a problem for admins who would be better off flipping burgers.

bug details

[tommis]

Here's the bug details: https://bugzilla.redhat.com/show_bug.cgi?id=453340

One of the bug comments says: "Latest caching-nameserver renamed my named.conf to named.conf.rpmsave in /var/named/chroot/etc" - so this should mean that you can still restore the lost conf file.

Re:bug details

[Chris Mattern]

In other words, as somebody else posted, he installed the "caching-nameserver" package and got, surprise, a caching nameserver. Shocking.

Re:bug details

[hughesjr]

it is not a bug to get a caching nameserver if you install caching-namesever ... it would be a bug to install caching-nameserver and NOT GET a caching nameserver.
A caching name server IS one that does not have any zones and only looks up zones from the DNS root servers. It is a configuration error to install the caching-nameserver package on a machine that doing anything other being a caching name server.
Stupid admins have been complaining about this for 5 years ... but the documentation and bug entries all make it clear NOT to install the caching-namesever packages on DNS servers that control zones.

Re:bug details

[DaveAtFraud]

Yes but a caching name server is (or at least was for a long time) the Red Hat default. Go figure. That's all most people want or need. Any bets that lots of the people who got bit with this built the machine internally with the caching name server RPM installed and then just edited or copied over the production named.conf file to turn it into their "real" name server when the box went into production?

Cheers,
Dave

Re:bug details

[ilumits]

You have a point. You also need to get real. It is a bug when an update overwrites your configuration file.

Re:bug details

[_Sprocket_]

It is a bug when an update overwrites your configuration file.

Normally I'd say you've got a valid point. The problem here is that the config file seems to be part of the intent of the package (please correct me if I'm wrong).

A rough example would be if someone replaced a packaged binary with a custom-compiled version and then complained when the package update overwrote that modified binary.

Re:bug details

[ilumits]

I see your point. But the fact is that BIND and caching-nameserver can coexist on the same box (obviously, because caching-nameserver is dependent on BIND). Red Hat would save themselves some trouble by just correcting this on their end (perhaps something in system-config-bind for toggling between serving records and just caching...similar to the "switchmail" app for toggling between Sendmail and Postfix). If the flag for serving records is checked, then the system will protect named.conf regardless of whether caching-nameserver is installed. I suspect this would serve Red Hat better than just hoping sysadmins miraculously become smarter.

Re:bug details

[_Sprocket_]

Red Hat would save themselves some trouble by just correcting this on their end (perhaps something in system-config-bind for toggling between serving records and just caching...similar to the "switchmail" app for toggling between Sendmail and Postfix). If the flag for serving records is checked, then the system will protect named.conf regardless of whether caching-nameserver is installed.

Or the sysadmin could simply remove the caching-nameserver package. I might be missing something but it would seem to be just as easy as what you're describing.

Re:bug details

[ilumits]

What you're missing is that I was talking about what Red Hat should do to prevent this problem in the future, not what sysadmins should do to wise up to it. I know that talk of idiot-proofing the OS isn't too popular around here, at least not with the hardcore Linux geeks. But I've never heard Red Hat Linux be accused of being for hardcore Linux geeks.

Re:bug details

[Anonymous Coward]

It is a bug when an update overwrites your configuration file.

Normally I'd say you've got a valid point. The problem here is that the config file seems to be part of the intent of the package (please correct me if I'm wrong).

A rough example would be if someone replaced a packaged binary with a custom-compiled version and then complained when the package update overwrote that modified binary.

If the checksum in the RPM database does not match the actual file, it mostly likely means that the file has been modified.

If this is the case the installer should not overwrite or remove the file, but rather save the new one under a different name (e.g., ...rpmnew).

Same with the 'caching-namesever' package: if the admin has edited /etc/named.conf file (as determined by the checksum), the upgrade should not touch the file, but rather save the updated config to a different file name and print out a message.

Touching on-disk file when you know it's been modified is a bug.

Re:bug details

[Anonymous Coward]

Config files, by definition, are meant to be configured/modified.

Package managers like apt and portage will usually warn you if they detect that they're overwriting a user-modified config file.

Re:bug details

[_Sprocket_]

But wouldn't having this switch require the aforementioned clueless admin to know about it and make use of it? And if so, isn't it the same issue as having the knowledge to know whether you should have the caching DNS package installed or not?

Re:bug details

[ilumits]

The difference is that it would be an option in the configuration gui for the daemon, as opposed to having to have the knowledge that you need to remove this package that was installed on the system by default in order to prevent your BIND configuration from getting automatically borked in the future.

Re:bug details

[Anonymous Coward]

Maybe you are right or maybe you are just arrogant, but as I recall from the 70 or so Centos 4 servers I have installed, the default when you install named is both caching and chrooted. Are you saying that you should uncheck caching if your server is authoritative or slave for a zone?

The renaming original config files .rpmsave has been a Redhat default for years. A rational default I think.

Finally virtually all the Redhat/Centos servers I have installed in the last eight years have been set to automatically update. They are for small customers who do not have sysadmins or the technical knowledge to deal with this. They are not interested in paying me to test updates before they get them. That would be nuts. When occasionally an update breaks something, they call and I fix it. Not a big deal.

Re:bug details

[_Sprocket_]

Is the caching-nameserver package installed by default?

Re:bug details

[hughesjr]

well ... I built the OS for the 70 or so CentOS servers you are running ... and I am arrogant. What does that have to do with caching-nameserver :-D

Re:bug details

[Anonymous Coward]

Maybe the idea of having offering a full nameserver as a caching nameserver is silly. They should provide the caching configuration as the default install configuration, or as an install option (you know, the kind of questions dpkg does all the time).
If I know I installed BIND I will want to use it and configurate it anyway I want, as long as I know this kind of shit can happen in advance I don't expect to uninstall it and install it again the same bloody package just in order to configurate it differently.
What comes next? Four mysql packages to accomodate the four example configurations provided by upstream?

Um...

[wellingtonsteve]

"I am frankly surprised there has not been more of an uproar about this"

That's because the entire Internets are now broken!

Re:Um...

[I cant believe its n]

You mean my internet

:3.

Re:Um...

[exi1ed0ne]

That's because the entire Internets are now broken!

It was as if millions of p0rn sites cried out in terror, and were suddenly silenced.

Re:Um...

[wellingtonsteve]

Careful, don't give the government ideas... you might find servers breaking more often..

argh

[__aardcx5948]

I guess the syadmins could put in an option in a configuration file somewhere on what files to "keep untouched" when doing package upgrades, no? So that the configuration file wouldn't be overwritten. I think I've seen something similar in Debian distros. Anyway when I install a new (custom) kernel in Ubuntu for example, synaptic asks me if I want to overwrite GRUB's menu.lst with the newly generated one, view the differences or keep my old one etc. Surely there's something similar in Redhat?

Re:argh

[turbidostato]

"I guess the syadmins could put in an option in a configuration file somewhere on what files to "keep untouched""

chattr +i /path/to/file

Hey, it works even on non-configuration files!

Re:argh

[HTH NE1]

chattr +i /path/to/file

Ah yes, the immutable bit. I remember when it was a way to defeat TiVo's boot image protection, until they got the idea to reboot if the overwrite failed.

That why they get paid

[nicolas.kassis]

Half of whole point of a subscription to RHEL is to ensure that patches they put out are properly QAed. The other side is support, but I never had a chance to test that part out.

Re:That why they get paid

[MikeDawg]

Umm. . . I disagree completely. The only way I would consider a patch "put out properly" if it was tested in my exact, or near exact environment. I can only assume that I'm not important enough for that.

Re:That why they get paid

[Anonymous Coward]

I had an incident where I was, umm, forced to call Red Hat support over a broken LVM issue. I had already researched the issue, and came up with a solution, but the powers that be wanted to make sure that my solution was the correct one. It took RedHat a little over 1 hour to tell me "I'm not sure it will work," and when I pushed for the next level of support, the first words out of the techs mouth were, "You have backups, right?"

I just need the OS and patches. I feel that between the team that I have now and the resources that I have, that I can support the damn thing myself.

Re:That why they get paid

[the_B0fh]

I have. Support sucks. I especially love the "we found the problem. Can you reproduce the problem on your box". Yes, that is why I submitted a bug report, damnit. Then, "we think we have a solution. Can you test it on your box?" No, *YOU* test it on your box, and once you verified it works in your test environment, then I will test it, damnit. I am not your fscking QA department.

Even better. I was following a thread in opensolaris about nfs. They found the bug (something involving acls and so on). RedCrap refused to fix it.

--- wonder why that particular sucker^H^H^H^Hbscriber continues to pay for the subscription

No worries

[FlyingBishop]

I don't need to worry about that, I run Debian

Also, I don't run my own DNS. But if I were paying someone to make sure my patches weren't idiotic, I'd be pretty pissed, whether the patch was for something I used or not.

Re:No worries

[ettlz]

I don't need to worry about that, I run Debian

Ah, well then, you just keep on rolling them dice, OK?

Re:No worries

[larry bagina]

Idiotic.... like Debian's openssl "enhancements" that made the random number generator not so random?

Re:No worries

[Anonymous Coward]

Idiotic.... like Debian's openssl "enhancements" that made the random number generator not so random?

He didn't say that Debian was flawless. Just that it didn't have this specific flaw and that when this kind of flaws happen, people should be mad. He also didn't say that he wasn't mad when Debian had it's bug.

(note that I don't comment on if this is a bug to be mad about as I don't know the subject. /. consensus seems to be this was user error...)

Re:No worries

[HiThere]

Just remember that Debian has had it's own problems. Something recent about openssl being weakened for several years, possibly?

I run Debian, too. THIS problem may not exist on Debian, but enough do that one shouldn't be smug...it's unwarranted. (Granted, I usually run testing...but I *EXPECT* that at least once during the testing cycle my system will get so hosed that I need to reinstall [or wait for a few days, and then update/upgrade again].)

The other option is to run Debian stable...and that's so stable that many consider it moribund. (Which is, of course, true. But it generally takes a few years to actually die. I think Potato is finally dead...but I'm not altogether sure. Sarge seems to be no longer available.)

FWIW, I just received an order of Pink Tie 6.1 that I ordered from CheapBytes, because there are some old games that won't run on newer systems. Of course, this will need to be run in emulation, and sealed from the internet, but it's still reasonable...and Pink Tie == Red Hat from a 3rd party (CheapBytes repackaged Red Hat 6.1 as Pink Tie 6.1 ... Red Hat no longer ships the older system, however.)

Each system/distro has it's own advantages and costs. It's appropriate that Red Hat be dinged over this mistake, but don't go overboard about it.

You are WRONG :D

[hughesjr]

This article is absolutely wrong.

The user has misconfigured their DNS and has installed a package called, SURPRISE, caching-nameserver along with the other bind packages.

caching-nameserver IS just that, a caching-nameserver. It SHOULD NEVER BE installed on a DNS server that is used for Primary or Secondary DNS control. The bind packages do not in any way modify named.conf, but if you want a caching nameserver and if you have installed the caching-nameserver package, then you would EXPECT that it would replace the named.conf file.

The real question is, how does crap like this get posted as a feature article on slashdot.

Re:You are WRONG :D

[mrbluze]

The real question is, how does crap like this get posted as a feature article on slashdot.

And the obvious non-answer is "you must be new here".

Re:You are WRONG :D

[MicktheMech]

/. QA is worse than the poster's?

Re:You are WRONG :D

[Stevious]

Sounds like user error to me. I applied the recent update to BIND and still have the same named.conf I had before the update on both my primary and backup machines.

Re:You are WRONG :D

[Kamokazi]

There is a reason a lot of articles used to get tagged 'kdawsonfud'

Re:You are WRONG :D

[Anonymous Coward]

I've installed the patches on a number of RHEL systems and had no such problems.

This post, especially the title, is misiniformation.

Re:You are WRONG :D

[vegiVamp]

Point, but even then, it's still a config file, which should never ever ever be touched by a package update if it's been user-modified.

Re:You are WRONG :D

[Anonymous Coward]

The real question is, how does crap like this get posted as a feature article on slashdot.

Because suckers like us keep reading and commenting regardless. Slashdot can continue to post whatever crap they like with impunity because they know that the value of their site comes from the comments.

Slashdot is never going to improve.

Re:You are WRONG :D

[hughesjr]

BUT ... how can you create a caching-nameserver without changing that file???
If you do not change that file, you do NOT have a caching-namesever ... which was the whole point of installing that package.

Re:You are WRONG :D

[vegiVamp]

Packages should never even touch configfiles without asking for permission in triplicate.

I'm not familiar with the package in question, but I assume it also installed some binaries. If it found that there already was a configfile of that name, it should have asked what to do.

If setting up the caching-nameserver was a matter of changing config options, you don't need a package for that, you need a HOWTO.

Re:You are WRONG :D

[Anonymous Coward]

We can improve the comments.

Re:You are WRONG :D

[Anonymous Coward]

I hope my sarcasm detector is nonfunctional. Really.

Please read your post again and than look up what you need for a simple caching nameserver..

Re:You are WRONG :D

[_Sprocket_]

I'm not familiar with the package in question, but I assume it also installed some binaries. If it found that there already was a configfile of that name, it should have asked what to do.

If setting up the caching-nameserver was a matter of changing config options, you don't need a package for that, you need a HOWTO.

I would hazard to guess that unfamiliarity with the package is the real root cause of this. From the package description for caching-nameserver-7.3-3 (which could be a very old version):

The caching-nameserver package includes the configuration files which
will make BIND, the DNS name server, act as a simple caching nameserver.
Many users on dialup connections use this package along with BIND for
such a purpose.

If you would like to set up a caching name server, you'll need to install
the caching-nameserver package; you'll also need to install bind.

The file contents show:

Copyright.caching-nameserver
caching-nameserver.spec
localdomain.zone
localhost.zone
named.broadcast
named.conf
named.ip6.local
named.local
named.root
named.zero
rfc1912.txt

And so there we have it - a package designed to install and maintain the very generic files needed to configure a caching DNS server. DNS server not included.

And sure - this could be a HOWTO. But making a package allows for quick-and-simple configuration. And since this kind of thing is so generic, it really lends itself to packaging. I disagree that it should only be a HOWTO.

Re:You are WRONG :D

[Anonymous Coward]

"I'm not familiar with the package in question,"

Then you don't belong in this thread.

Re:You are WRONG :D

[baileydau]

Sounds like user error to me. I applied the recent update to BIND and still have the same named.conf I had before the update on both my primary and backup machines.

So did I, and got the same result.

I *think* the problem is that as well as the BIND package, there is the CACHING-NAMESERVER package (which according to the package documentation is actually bind, but with a caching .conf file)

If you have caching-nameserver installed and then make it into a *real* nameserver, then you could have problems when you update. But if you have bind installed it works fine.

YMMV

David

Re:You are WRONG :D

[vegiVamp]

Such a package is indeed a matter of convenience, so I'm willing to leave that to personal preference; but I will keep insisting that no package should touch conffiles unasked.

They're doing it wrong! :-)

Re:You are WRONG :D

[_Sprocket_]

Such a package is indeed a matter of convenience, so I'm willing to leave that to personal preference; but I will keep insisting that no package should touch conffiles unasked.

The more I think about it, the more I've come to decide that the other key part to this is what you've touched on. There's an expected behavior that isn't followed in this situation. Most of the time, config. are handled in the manner you noted. But because in this case the package IS config files, we get unexpected behavior that is obviously likely to take some by surprise. Because of this, the normal behavior of only replacing unmodified configs should be followed.

The advantage here is that the admin who installs this package without understanding what it is (or picks up responsibility of the box w/out realizing the package is installed) won't be suprised come update time. Likewise, the admin who wants to maintain a cache proxy with little fuss can still automagically have their configs updated as long as they don't touch them (and really - the intent of this package is so you don't HAVE to do any configs). Seems to be the best of both worlds. Or at least a suitable compromise.

I'm not keen to bash RedHat about for the way things are. But I do agree that there's an opportunity for them to improve the current situation.

Test your patches

[MikeDawg]

What kind of environment are you in where you don't first test your patches that are going out to live production machines? Regardless of the fact that it is linux and not windows, you should always test your patches before you roll them production.

Re:Test your patches

[mrbluze]

...you should always test your patches before you roll them production.

But that would cost time and money.

Re:Test your patches

[MikeDawg]

. . . and DNS servers not responding doesn't?

Re:Test your patches

[jeiler]

Not to mention that it would involve actually doing things correctly.

Re:Test your patches

[mrbluze]

. . . and DNS servers not responding doesn't?

That was my point (with sarcasm quotes missing)

Re:Test your patches

[Just Some Guy]

What kind of environment are you in where you don't first test your patches that are going out to live production machines? Regardless of the fact that it is linux and not windows, you should always test your patches before you roll them production.

Disclaimer: I test first.

You know, lot of people work in small shops that can't afford multiple redundant servers. I suspect that business with a single DNS/web/mailserver are a lot more common than Slashdotters this morning seem to thing. What are those admins supposed to do? They're receiving a critical security patch from a trusted vendor, and I imagine a lot of them feel pretty safe applying that to their sole production server. This doesn't make them stupid or incompetent.

I have the luxury of lots of hardware that can fill in for other gear in a pinch, but lots of people don't. They don't deserve scorn for it.

Re:Test your patches

[radish]

Couple of points. Firstly, the smart but under resourced admin should use this incident as evidence of what happens when management won't cough up for required equipment. A test environment is not a luxury, it's a necessity for a reliable system. Secondly, something like vmware can let you set up and test an environment on whatever hardware you do have lying around - you could even run it on the prod box at a pinch.

Re:Test your patches

[Anonymous Coward]

Gimmie a break. Get yourself a copy free vmware server, xen, uml, or whatever else. You don't need to run on hardware.

Re:Test your patches

[Sleepy]

You nailed the demographic, but these are EXACTLY the group that should not be running their OWN exposed servers.
This would be true for any server, but "double" so for DNS.

DNS hosting is cheap, and the expenses are unpredictable (unlike the commotion raised when you have "Windows IT" people whining about "what they pay Redhat for". UNIX does what you ask. If you want 10 meters of rope, tie it to a beam and stick your head in... it will let you.

Automatic updates are FINE for the desktop, in shops like you describe.

Re:Test your patches

[Anonymous Coward]

When you can buy a used laptop or desktop for $50 or $100 (lots of people will give away old PCs for free just to get rid of them) what excuse is there for not testing again? How long did the patch install take, a minute? It's called LAZY - and if it's not stupid AND incompetent it is VERY poor judgement. Stop making excuses for click happy admins who don't think before they act.

Re:Test your patches

[IchNiSan]

I'll put this more politely than I did in an above post, but, at the very least people should be able to test by dual booting their own machine with their production OS and testing things like, what happens to my zone files when I run this patch. If they can't be bothered to do that, they shouldn't whine about getting screwed over.

Re:Test your patches

[GleeBot]

First, this problem takes like a few minutes to correct. If you're applying the patch on command, instead of just blindly patching automatically, then you should be around long enough to do a smoke test and make sure you didn't bork anything.

Second, if you only have one server, it's eventually going to go down for one reason or another, whether it's a botched upgrade, misconfiguration, or simple hardware/power failure. A little downtime isn't ideal, but it's the expected situation, and for a small business, usually not fatal.

If reliability is that important to you, then you simply need a test server. If you can't or won't, you have to live with less than reliable service. That's all there is to it.

There will always be bugs with updates, because you simply can't test every configuration out there, especially for a time-sensitive security update.

Re:Test your patches

[Imagix]

I have the luxury of lots of hardware that can fill in for other gear in a pinch, but lots of people don't. They don't deserve scorn for it.

Then these admins really need to learn about stuff like VMware Server. Free, and doesn't require additional hardware. The admin's PC is probably large enough to host a VM alongside their normal tasks. They could run their staging server in a VM (and not even have it running 24/7, only have it run to test things).

Re:Test your patches

[citylivin]

Never worked for a non profit or charity have you...

Re:Test your patches

[VGPowerlord]

Second, if you only have one server...

If you only have one server, you should get a clue and realize that DNS requires a minimum of two nameserver IPs for a given domain for a very good reason, and that those IPs should be different machines.

Re:Test your patches

[Anonymous Coward]

>can't afford multiple redundant servers

Virtualization on the low end is free. Test on a 1-CPU virtual server on your desktop PC.

Re:Test your patches

[Anonymous Coward]

1. If a single machine provides the services, get a single machine to test installs. In a bind, one could just ensure configs are good and daemons run.

2. The guy installed the wrong package for the task and the upgrade reverted the config to the default config.

Something needs to change if someone doesn't have enough time to verify that the configs are okay after an upgrade or someone uses the wrong package for the task. The company probably didn't have enough money for decent pay or proper hardware.

The real question is how much we should berate a person for announcing a bug on Slashdot (made worse by an administration error) when it could have simply gone to Redhat bugtrack.

Re:Test your patches

[Anonymous Coward]

How a small support shop with several hundred Mom and Pop clients who barely understand how to turn their systems on? There is no way every patch can be tested for every client with Quickbooks, Office, OSAS, Peachtree, etc. The options are to leave automatic updates off or to turn them on and trust the manufacturer to know what they are doing.

Re:Test your patches

[Anonymous Coward]

What kind of environment are you in where you don't first test your patches that are going out to live production machines? Regardless of the fact that it is linux and not windows, you should always test your patches before you roll them production.

Disclaimer: I test first.

You know, lot of people work in small shops that can't afford multiple redundant servers. I suspect that business with a single DNS/web/mailserver are a lot more common than Slashdotters this morning seem to thing. What are those admins supposed to do? They're receiving a critical security patch from a trusted vendor, and I imagine a lot of them feel pretty safe applying that to their sole production server. This doesn't make them stupid or incompetent.

I have the luxury of lots of hardware that can fill in for other gear in a pinch, but lots of people don't. They don't deserve scorn for it.

UMMM actually yes it does make them stupid, there is this thing called vmware, even a free one called vmware server, that will let you create a test environment. hell if you don't have extra hardware you can install it on your pc and setup test machines even if you can only run and test patches on one server at a time

Re:Test your patches

[hxnwix]

a lot of them feel pretty safe applying that to their sole production server. This doesn't make them stupid or incompetent.

OK, obviously misusing a computer this badly can not make someone stupid.

That is probably a result of all the paint chips, lead solder, and tasty, leaky, salty batteries...

*sigh* At least it's easier to clean drool off an LCD screen.

Old News

[lib3rtarian]

This update was released via RHN more than two weeks ago.

Red Hat's been kind of iffy lately

[propanol]

A few months prior to the release of RHEL 5.2, they released a kernel update (2.6.18-53.1.6.el5) in which they had added a patch for an issue that could make a system oops upon when files with names of a certain character were present on NFS shares. However, this patch also contained a bug which broke NFS lookup caching and subsequently crippled NFS performance to the point of NFS being completely unusable when working with multiple smaller files. They released a patch for it, but it would only apply cleanly to their testing kernel (which would later become the kernel shipped with 5.2) and they refused to backport it to their then-stable kernel. Shortly after, the vmsplice flaw was found forcing people to update and bring this bug upon them. For us it wasn't that big a problem since we're using CentOS and don't have anything requiring us to use standard RHEL packages (so we backported the patch and built our own kernel package), but a large amount of corporate RHEL users are required to use only standard RHEL system packages because of service contracts with hardware vendors and hence they could do little to remedy this bug. As we were among the first to report this and post about it on mailing lists, we received a lot of communication from corporate RHEL users/sysadmins asking us for help on this, further proving that this was a major issue that should have been addressed right away and not post-poned to the next major release.

Experienced Monkeys...

[spankymm]

...check for rpm mouse droppings by running find.

RH may have made a small coding mistake - you made an even bigger one.

caching-nameserver

[novadragoon]

Did the OP have the package caching-nameserver installed? If so, that packages whole point is to change the bind configuration into doing just caching.

Re:caching-nameserver

[Sleepy]

Did the OP have the package caching-nameserver installed? If so, that packages whole point is to change the bind configuration into doing just caching.

I PAY REDHAT GOOD MONEY FOR THIS!

I don't need you implying that they can't prevent my mistakes, or read my mind.

(joking, but look at all the "if this were Microsoft.." people skimming right OVER this fact. You saw it, I saw it, and that should be enough to shut them up... and never mind the bad practices by the submitter. He installed the WRONG PACKAGE, folks!).

Common Red Hat Mistake

[Spazmania]

Red Hat makes this mistake a LOT. It makes the update process very unreliable. SuSE isn't as bad but they still have problems if you customize a piece of software's configuration in an unexpected way.

Debian is king here. The incremental patches almost never break a configuration and the major release upgrades tend to work; they often change package names if the new "version" has a major incompatible change in the configuration.

Re:Common Red Hat Mistake

[houghi]

openSUSE has warnings in those files NOT to use them and tell you which one you need to use. e.g. for Apaches http.conf it says:
# If possible, avoid changes to this file.
It then goes on to exlain a lot about what to place where.

There are many other files like that. Another example:
# PLEASE DO NOT CHANGE /etc/bash.bashrc There are chances that your changes
# will be lost during system upgrades. Instead use /etc/bash.bashrc.local
# for your local settings, favourite global aliases, VISUAL and EDITOR
# variables, etc ...

So if you decide to not read what is in those files and just change them and then do an upgrade, don't be surprised if the system works as intended.

I like also how you use 'almost never', 'tend to work' and 'often'.

Re:Common Red Hat Mistake

[Anonymous Coward]

Negative claims without any examples for proofs is just trolling.

Re:Common Red Hat Mistake

[Anonymous Coward]

Give me a break. He has caching-nameserver installed which is supposed to do that. This is user error, pure and simple. It's time for them to hire an RHCE.

Re:Common Red Hat Mistake

[sdsucks]

Not that I doubt you, but I would appreciate if you would expand on "Red Hat makes this mistake a LOT" ?

It's never happened to me and I've been administering some RHEL boxes for about 5 years now.

I always do check my updates properly though, and even backup any configuration files again before updates are installed. (In addition to the nightly backup which has a special run for config files)

I will also add that this patch applied fine for me -- I do primary DNS for about 350 domains on one box and the patch worked 100% when applied with up2date.

Re:Common Red Hat Mistake

[Spazmania]

openSUSE has warnings in those files NOT to use them and tell you which one you need to use.

True enough. Doesn't help when I want the application to do something it is capable of but which wasn't envisioned by the SuSE packagers. Like binding sendmail to a non-privileged port as a non-privileged user and then using iptables to redirect port 25 up to that port.

Debian won't automatically overwrite my modified init.d script during an upgrade. It'll ask permission with the default set to "no." SuSE and Red Hat will overwrite it.

Then there's my sendmail.cf that I've built up over the course of 15 years. I'm not about to replace it with a suse-originated sendmail.mc. With Debian, I can prevent the upgrade process from trying to regenerate the .cf files from the default .mc. Not so with SuSE or Red Hat.

I like also how you use 'almost never', 'tend to work' and 'often'.

Almost never: I had a Debian upgrade in 2002 that had a bug in it which removed one of the inetd.conf listeners because it started with the same string as another listener that it was supposed to remove and replace. With scores of servers over better than a decade, that's the only incremental upgrade I recall hosing me. Close enough to never for you?

Tend to work: I built one of my home servers on Debian in 1997. Since then I've upgraded it in place all the way to the current Debian stable version by sshing in and using the package management system. For hardware upgrades I use tar and grub-install, preserving the exact system image. Try that with SuSE. Oh, that's right, you can't. Despite how difficult it is to automate that sort of task, my most recent major version upgrade from sarge to etch required manual cleanup to only three packages afterwards.

I suppose I will have to finally rebuild that one Debian server now that the hardware is moving from 32-bit to 64-bit. But who knows -- the clever Debian maintainers may find a way around that too.

Nope.

[juiceg]

[root@struct etc]# rpm -q bind
bind-9.3.4-6.0.2.P1.el5_2

[root@struct etc]# grep zone named.conf | wc -l
49

Freshly updated and restarted the service. Still have all my zones. Sounds like someone didn't do too well on the RHCE?

Re:Nope.

[betterunixthanunix]

There is a good reason this was tagged, "kdawsonfud." The person who reported the problem had caching-nameserver installed, not just bind, which explains why we aren't seeing widespread outages; most people don't install caching-nameserver when they don't want a caching nameserver.

Re:Nope.

[Anonymous Coward]

this comming from someone who hasn't learned to not run a shell as root...

Re:Nope.

[sdsucks]

Yep same here. Patch applied fine.

Well

[ledow]

Yeah, it's a silly mistake.

But you should be testing things like this first, and whenever you upgrade you should really be looking at/for all .rpmsave or equivalent files first to make sure nothing has changed in the meantime. Otherwise, you're just removing your config and replacing it with the default whatever happens. You should also be checking .rpmnew (or equivalent) each time to check that it hasn't changed in terms of syntax, defaults etc. (which, let's be honest, is quite likely for such an important update - especially given that we hardly know what the exact problem is yet). I wouldn't go so far as to suggest intimate analysis of packages while they are still packed unless the systems you are running are quite critical to the operation of a business.

Part human-error on RH's part (it happens). Part incompetence in not testing the updates yourself first. Chances are that if I were affected by this, I would catch it as part of "right, what did that package change?", or notice as part of usual testing later, and then just move the file. I probably wouldn't even bother to send RH a note.

If you have a DNS server, that suggests that there are reliant computers. As courtesy to all those reliant computers you HAVE to test changes and check carefully what they are doing first. If you were "stung" by it, it suggests you hit this problem on ALL your DNS servers and/or that you only have one DNS server anyway. To deploy packages like this on such a setup is just asking for trouble.

Re:Well

[LeneJ]

I just want to clarify a bit about rpmnew vs rpmsave.

Red Hat will create an rpmsave file when we make a significant change to the configuration file, or a mandatory change. Other than that, we keep the original config file, and store the rpm-config as rpmnew.

Don't forget!

[prandal]

Don't forget to check your named.conf on RHEL 5.x (and CentOS 5.x).

Make sure that any lines like

query-source port 53;
query-source-v6 port 53;

are commented out or deleted so that forwarded DNS queries come from random ports.

Restart BIND if necessary.

Re:Don't forget!

[Anonymous Coward]

Uhh.... You might want to check with your FW admins before you do that. Maybe they've only punched holes for port 53...

caching-nameserver vs. bind-chroot

[uberlinuxguy]

The file I actually had replaced was

/var/named/chroot/etc/named.conf

That file is not owned by caching-nameserver, but is owned by the bind-chroot package.

Re:caching-nameserver vs. bind-chroot

[hughesjr]

I just pulled up the SRPM and looked, and bind-chroot has:
%ghost %config(noreplace) %prefix/etc/named.conf
%ghost %config(noreplace) %prefix/etc/named.caching-nameserver.conf
%ghost %config(noreplace) %prefix/etc/rndc.key
It should not replace that file with an .rpmsave file

Configuration management

[cluening]

Have you considered using a configuration management tool such as Bcfg2 or cfengine to make sure your own config files are restored after package updates are made? You can never really trust those package maintainers...

Re:Configuration management

[Just Some Guy]

Or even version control. As much love as Git and Subversion get around here, I'm surprised you don't hear more people advocating using them for config files.

Re:Configuration management

[Anonymous Coward]

Gentoo is fantastic at this. If a package includes a file in /etc, first it will try to merge trivial changes, and then if that's too difficult it will leave your copy alone. Then it prompts you after package install that X-many files in /etc need to be updated with "etc-update". This tool lets you view the new version of the file and decide whether to keep the old, replace with the new, or do some sort of interactive merge.

Very slick, IMO.

Re:Configuration management

[cluening]

Ha! Take that one step further and you've got the winning combination: on my rather large system, I use an SVN-managed Bcfg2 repository to hold all of my config files. That way I've got disaster recovery and rollback in the version control system and good change control and monitoring with the configuration management system. It makes sleeping at night that much easier.

Automatic Updates for Fools

[Anonymous Coward]

Anyone who allows automatic updates is a fool. Do you really trust the updater to not install a backdoor somewhere in your system while you aren't watching?

In any case, Redhat is guilty of sloppy authoring (and ultimately poor design) of rpm scripts which is a significant contributor to configuration problems.

Try eradicating SELinux from your system and see the mayhem caused by various rpms erroneously depending on what should be an optional module.

Re:Automatic Updates for Fools

[X0563511]

Because I really have the time or want to work with a hand-installed package system, and build all the packages myself.

The idea is that you TRUST the vendor. If you don't trust them, why the hell would you use their software at all?

Re:Automatic Updates for Fools

[DavidTC]

Do you really trust the updater to not install a backdoor somewhere in your system while you aren't watching?

Um, yes?

If we cannot trust Red Hat not to install a backdoor in our system, we'd hardly be able to use Red Hat, would we?

Re:Automatic Updates for Fools

[the_B0fh]

Anyone who allows automatic updates is a fool. Do you really trust the updater to not install a backdoor somewhere in your system while you aren't watching?

Oh my, another person who read Trusting Trust, and writes his own compilers.

Which distro?

[Anonymous Coward]

If I just needed to setup a DNS server, which distro would anyone recommend? I have an old BSD 4.8 with bind 8.somethin and I know it all needs to be canned.

Re:Which distro?

[MichaelSmith]

I use netbsd but a current openbsd or freebsd would be fine I am sure.

not on EL3 it didn't

[Anonymous Coward]

my RHEL3 nameservers were upgraded by up2date without any problem whatsoever.

Suprised...

[kenh]

I must say that I am very suprised that this patch acted one way in the posters test environment and another when it was installed on their production machine... That's very odd.

What, he didn't test it before placing it in production? Never mind, move along - nothing to see here.

If the poster made an error (as suggested by a previous post), or if he installed a patch without testing it, bad on the original poster - but if the patch truely was bad (a possibility), then bad on RHN for letting something bad out of QA and into production. But RHN's possible mistake doesn't absolve the system admin for not testing the patch before using it.

The only way this isn't the original poster's error is if the patch worked different in production than in test, but no one is claiming that AFAIK.

No matter what you pay for support to RHN, you are ultimately responsible for your systems, not RHN or any other vendor...

Re:Suprised...

[merlinokos]

What, he didn't test it before placing it in production? Never mind, move along - nothing to see here.

...The only way this isn't the original poster's error is if the patch worked different in production than in test...

It occurs to me that you have entirely missed the point. The point is not that the user screwed up. Everybody knows that. Had the user done the same thing in hist test environment and noticed the problem, it still would have been a valid post on /. and a valid problem for RedHat.

The only effective difference between what he did and the right way to do it is the amount of ridicule he's going to face from people who have either A) a proper test environment to test it in (not every company thinks that sort of investment is a good way to spend money), or B) want to act pedantic about a point that really should be a footnote to the article, rather than the focus.

Re:Suprised...

[kenh]

I don't think I've missed the point. In this day of cheap hardware and free Virtual Machine packages, there really is no excuse for not having some sort of simple test enviroment (even it is a VM "world" on your personal desktop). I give RHN no free ride for (possibly) issuing a bad patch, but the original poster didn't say "I made a mistake and put a RHN patch in my production environment without testing it", no he heaped responsibilty on RHN for the bad patch.

Your "Had the user done the same thing in his test environment" comment is interesting - the real answer isn't to post bug reports on slashdot, post it with RHN - that is the correct response.

The difference between what he did and the right way to do it is not the amount of ridicule he will face, it would have left his production environemnt functional while he awaited RHNs corrected patch. That is a big, big difference.

As far as being pedantic, how many System Admin "wannabes" read Slashdot? Far to many "home schooled" future system admins think this is a proper way to run a production environment, it is not, and it should be pointed out.

While I don't simply accept the position that RHN released a flawed patch, his poor admin skills (patching a production environment) put his assertion in doubt.

Don't like the OpenSUSE way

[Anonymous Coward]

I've been using SuSE for ages but I never quite liked its way of shoo-shooing me away from config files (like XF86config, which I had to edit to get my video card to work).

Some time ago I got so fed up with Flash constantly crashing Firefox I renamed the plugin file. So now OpenSUSE is constantly bickering about 5 new updates but refuses to install them because it's gotten confused about the missing .so file.

It's fine to have graphical config utilities but they should be smart enough to cope with manual editing. If that's not possible, how about just sticking with manual editing.

Re:Don't like the OpenSUSE way

[andrewd18]

If that's not possible, how about just sticking with manual editing.

Because 98% of all users hate having to edit a configuration file just to get the maximum resolution out of their video card.

Re:Don't like the OpenSUSE way

[X0563511]

Then suse is probably not for you. You should try something like Slackware (slamd64 is a good amd64 port) if you need that kind of flexibility/control.

Hell, if you tried hard enough, I'm sure you could port YaST to Slackware!

Mod parent up!

[Chrisq]

I am sure that many people do not realise that going through a NAT device usually means that predictable port numbers will be allocated.

Of course until we get details of the hole and fix we cannot be 100% sure but it is very likely that exposing predictable port numbers (which the fix randomised) reintroduces the hole.

If DNS software vendors had a year's notice then why didn't the NAT firewall vendors. They could have introduced a patch at the same time.

Re:Mod parent up!

[afidel]

Apparently they did, either that or Checkpoint's protection feature for the general class of DNS poisoning attacks just happened to protect against this one too. However, even if it did protect against it I doubt they could have release a same day press release stating it did if they hadn't been notified of the vulnerability ahead of time.

Re:Mod parent up!

[something_wicked_thi]

Judging by the CERN details, it sounds like there are two things you need to do. You need to be able to predict the 16-bit random number, and the 16-bit random port. My reading (and this was very brief, so someone *please* correct me if I'm wrong here) is that the older DNS servers had two flaws: a flaw in the RNG for the 16-bit transaction number, and they used fixed or predictable ports.

A NAT will reintroduce only the second problem because it gives you predictable ports, but obviously, relying solely on the unpredictability of a 16-bit transaction id is a little scary. Because of the birthday paradox, (assuming the attacker has perfect knowledge about which port you're choosing) an attacker would need to send only something on the order of 2^8 packets to poison the cache.

Re:Mod parent up!

[Anonymous Coward]

Judging by the CERN details, it sounds like there are two things you need to do. You need to be able to predict the 16-bit random number, and the 16-bit random port. My reading (and this was very brief, so someone *please* correct me if I'm wrong here) is that the older DNS servers had two flaws: a flaw in the RNG for the 16-bit transaction number, and they used fixed or predictable ports.

A NAT will reintroduce only the second problem because it gives you predictable ports, but obviously, relying solely on the unpredictability of a 16-bit transaction id is a little scary. Because of the birthday paradox, (assuming the attacker has perfect knowledge about which port you're choosing) an attacker would need to send only something on the order of 2^8 packets to poison the cache.

No, the birthday problem doesn't apply when you are trying to match a specific person's birthday.

Re:Mod parent up!

[Effugas]

[This is Dan Kaminsky]

The NAT vendors didn't get as much notice because we didn't realize so many of them were doing this.

If we had, they'd have been brought in from the start.

Now they're scrambling, to their credit. It's a bit of a facepalm for me.

Re:Mod parent up!

[Ash Vince]

People like you should not be allowed to post to slashdot. Why confuse all the pointless speculation and mindless insults of people doing a damn good job some actual facts :)

For the record I am very glad this has been handled the way it has. Although I as a sysadmin would like more information regarding what this exploit is I understand this is a double edged sword and that if I had that then so would a great many people I would not like to have it.

I will watch your webcast with some eagerness but in the meantime have patched all my systems as best I can with the information available.

Re:Mod parent up!

[something_wicked_thi]

You're assuming just one DNS lookup. Odds are, the DNS server is sending out lookups all the time, and you need to match only one of them.

Re:Mod parent up!

[imipak]

That only works if you use SmartDefence, which (for various reasons) many sites do not want to do. They have not patched the underlying bug in VPN-1 NAT. According to Vixie (linked from the "related stories" at the top of the page) CERT are apparently working on a bulletin. I do wonder if it and the associated patches will be out before BlackHat...

Re:Mod parent up!

[imipak]

That's good to know; after I reported it to CERT, Checkpoint and Cisco, following Tom Cross' XForce blog posting, the only replies I've had were an auto-responder from CERT and an automated "mail loop detected, aborting processing" alert from the bowels of Cisco. (I've also seen Vixie say CERT are working on a bulletin.) Mebbe someone else had reported it to them first. *shrug*

FWIW we're slaving internal DNS off patched external servers for external lookups. Apparently Windows 2000 DNS/DCs can't do that.

Re:Mod parent up!

[petermgreen]

IIRC linux based nats (which seems to be most home routers nowadays) tend to use a port preservative strategy, that is they only modify the port if not modifying it would conflict with an existing mapping.

Re:Mod parent up!

[Effugas]

BTW, do you have a real name I can credit you by? Or should I just refer to you as Imipak?

Check out the new slaving info at www.doxpara.com. Does it work for you?

You Should Be FIRED.

[Anonymous Coward]

Let's just blindly patch overnight and hope all patches work, then piss/moan when something breaks. Really smart, BTW, on a production DNS server!

JR : "I have a gun in my room, I'll just go shoot him, and. . . "
Dr. Evil: "We'll just close the door and assume everything went according to our plan."

Parent should be modded up

[dstech]

I wish I had mod points with which to mod you up. This is NOT a bug, and a few RHEL test machines I have here updated just fine, keeping their zone files as expected.

What about Debian's OpenSSL bug

[dfcamara]

Recent Debian's OpenSSL bug was orders of magnitude worse...

At least...

[xalorous]

...the file was backed up and not deleted.

yea smartypants

[unity100]

the thing youre forgetting is that microsoft REGULARLY does that, and even with irrelevant minor updates. thats why people are too worked up because of microsoft. they will gonna let this red hat incident slip by, because red hat doesnt have a track record of messing it up.

Re:yea smartypants

[Anonymous Coward]

the thing youre forgetting is that microsoft REGULARLY does that, and even with irrelevant minor updates.

[citation needed]

I can think of two that have been reported on slashdot in the years I've been reading it and one of those was mentioned by MS in their patch release note ahead of time.

Re:yea smartypants

[unity100]

one has been reported just in the last 1.5 months span. many more issues with updates had been reported in the last 2 years. which slashdot you were reading, i wonder. its certainly not the one i read.

you were hit ?

[rs232]

"Unfortunately we were hit and our servers went down overnight while RHN dropped its bomb and I am frankly surprised there has not been more of an uproar about this"

You mean you installed a patch that you didn't know was faulty and didn't have a rollback option in place, shame on *you* Mr. Sysadmin .. :)

question

[bucky0]

Does the red hat version of apt-get (yum? I've used debian exclusively for a long time, so I forget what command it was) not prompt you when it wants to overwrite a config file? On any debian (or debian derived) machine I've used, apt-get always asks what you want to do if your config file is different than the package's.

If yum(?) blows away configs without prompting, that's pretty bad.

What are those admins supposed to do ..

[rs232]

"lot of people work in small shops that can't afford multiple redundant servers .. What are those admins supposed to do?"

Keep two harddrives in the same machine and keep a clone of the DNS server on the second HD, if an upgrade borks, just swap the drives.

Re:What are those admins supposed to do ..

[Just Some Guy]

Summary: keep backups. :-)

RHN satellite server

[Swampcritter]

If you don't check out how neat the RHN satellite server, or the new spacewalk server is, you're really missing out. It is really nice in the enterprise environment.

Here are the problems with the RHN satellite server:
1) It only runs on RHEL4. RHEL5 is not supported.
2) It costs a lot of money ($13,500 annually)

Reference: http://www.redhat.com/red_hat_network/

As for the Spacewalk server:
1) Requires Oracle Database (9i or 10g)
2) Only supports Fedora and CentOS. Cannot manage RHEL releases.
3) Requires RHEL5 as the base machine for the initial installation/web server platform.

Reference: http://www.redhat.com/spacewalk/faq.html#compare

Re:RHN satellite server

[Anonymous Coward]

yes, looks promising, but back to cfengine I go...

Re:RHN satellite server

[donaldm]

You can now get the source code for RHN satellite server, however like you said it does cost although if you are managing thousands of machines this is basically nothing compared to what it can do. RHEL5 is not supported because RHEL5 uses yum and if you care to look you can find a yum for RHEL4 which works fine, especially if you want to create your own private "yum" server which if you do your homework may only cost between US$1000 and US$5000 (could even do it cheaper but you do get what you pay for).

In a enterprise situation a few thousand dollars is small change so if this can save time and provide greater reliability then the money is well spent or in some case wasted.

Those incompetent fools!

[prisoner-of-enigma]

I swear, Microsoft finds a way to mess up even the simplest of tasks! Can't those fools in Redmond even push out a simple patch for... ...wait, did you say a patch for named? But...named isn't a Microsoft product.

Surely you're not suggesting that a cherished and prestigious vendor like Red Hat can make a mistake, right? Something is amiss! Must. Find. Way. To. Blame. Microsoft!

Welcome to third party packaging...

[Venotar]

This is news? Redhat (like every OS vendor I've ever dealt with) have been pushing out updates with broken assumptions for years.

In fact, this isn't even the first time they've done something similar when updating bind:
back in 2004 they released RHEL 3 update 4 and many people had precisely the same experience. Additionally, when applied, Update 4 removed the /etc/rc*.d/S*named and /etc/rc*.d/K*named and then shut named off.

As a quick glance at redhat's bugzilla shows, the first problem (the same one you experienced in this release) wasn't a schoolboy mistake on the packagers part, or a bug. It was the result of a poorly understood choice on the part of the person who originally provisioned the machine.

Rather than installing just the original bind-9.2.4, the people who had their named.conf overwritten had installed bind plus a package called caching-nameserver. It's that package that, when updated, backed up and overwrote their bind config. The "caching-nameserver" package should only be installed if you want to run a caching nameserver, because the caching-nameserver package isn't an application at all - it's simply a named.conf file.

The real bug (back in 2004) wasn't actually in Update 4's bind package. As it turns out, the package it replaced incorrectly contained a `chkconfig --del named` in its uninstall script.

Anyone without proper alerting and a good QA process found that one out the hard way. I had customers who'd gotten so blasè about performing nighttime maintenances without proper reversion testing that they scheduled nightly cronjobs that ran up2date at midnight and rebooted the production machine, Naturally, they woke up in the morning to find they'd just suffered 8 hours of downtime.

Lesson? Don't trust the vendor's QC work, don't install unnecessary packages, and make sure to QC your own work! Ask any experienced Windows admin about unintended consequences from "trusted" vendor patches...

On Blindly Updating - OS X Server 10.3.x

[not_hylas( )]

GUILTY.
Seems the person that prepared the patch is a new hire at Red Hat.

Beware Latest 10.3.x security update - it replaces /etc/named.conf:

http://discussions.apple.com/message.jspa?messageID=5876624

This is why I don't run Windows...

[Blimey85]

Crap like this is why I lost faith in Microsoft and quit running Windows years ago. Thankfully my RHEL box isn't affected by this sort of... oh... wait... really? Shit.

Or you will just lose the file unless you backup

[Roberto]

Check here:

http://lateral.netmanagers.com.ar/weblog/2008/07/16.html#BB701

In at least one very common config, named.conf is a symlink, so copying it doesn'tavoid it being overwritten.

The named update script "copies" symlinks by making another symlink, not by copying the underlying file.

Not a bug, expected behavior

[Todd Knarr]

This sounds like how RPM's behaved as long as I can remember. It looks at three versions of a config file: #1 the one from the old package, #2 the one currently on disk and #3 the one in the new package. If the config file hasn't been customized (1 and 2 are identical), it moves the old file to .rpmold (if 1 and 3 differ) and puts #3 into place. If the config file has been customized, it checks whether 1 and 3 differ. If they haven't then nothing's chanced, the customized config file's still valid and it drops #3 in with the .rpmnew extension. But if 1 and 3 differ, then something in the config file may have changed and the customized config file may no longer be valid. But it's got customizations in it that the admin may need to refer to. So it outputs a warning message about what it's doing, moves the customized config file to .rpmsave and installs #3, and the admin's expected to have seen the warning and to merge their customizations into the new config file. You do watch for warnings and errors during the update, right?

In this case RPM is right, old named.conf files aren't valid. If they're based off RH's old stock config files, they have the source port locked and that disables much of the security fix. So the admins do have to check and modify their customized files before the system's finally ready (or at least RPM has to assume they do, since it can't know exactly what their changes were). That's exacerbated by probably having caching-nameserver installed, but I think a stock BIND install has a similar named.conf until you add your own zones to it.

I'd chalk this one up to admins who a) don't understand an inherent limitation of package-management systems (namely, it doesn't know why you changed something, only that you changed it), b) didn't watch the update process for errors, and c) didn't check the systems for functionality after the update.

Re:Not a bug, expected behavior

[DraconPern]

You just summrized why I hate Linux even though I use it. #3 gets changed often because comments are clarified, which means this rpmsave thing happens very very often. Since the package manager has access to all three files, it should have migrated the old settings over instead.

Re:Not a bug, expected behavior

[Todd Knarr]

In an ideal world, yes. But look at all the different syntaxes for all the different config files. How's the package manager supposed to know that it's just a comment that changed in the config file, as opposed to a critical setting or something major like a change from "name = value" to an XML syntax? As far as automatic migration, you're asking for CVS merge functionality. And being a professional software developer who's worked with version control systems and done merges, I can tell you I never completely trust automatic merges and they're never automatic. You always end up with conflicts that have to be manually resolved, and you always have cases where the merge result isn't right even though there aren't any conflicts. The closest you can get is a system like Debian's, where the configuration's stored by dpkg outside the config files. If there's a change, dpkg recreates the config from the stored answers to the configuration questions. And that's never fully automatic, the admin has to manually answer any new or changed config questions before dpkg can finish it's work and dpkg can't update fully-local config files that aren't under it's control (like the authoritative zones config for BIND).

Remember that, while DWIM sounds good in theory, in practice it results in things like "No results for 'rm *~', trying 'rm *' instead.".

You'd Think the Slashdot Editors Would Know...

[afabbro]

...that neither 'bind' nor 'named' should be capitalized. Then again, they're not very technical people.

Re:You'd Think the Slashdot Editors Would Know...

[prandal]

BIND = Berkeley Internet Name Domain

Looks like it should be capitalised to me.

unrelated question

[mzs]

I've been using Thiobor HyperWRT for a while now but that has not been maintained for a while now. I use dnsmasq on the WRT54G (an old one more like today's GL version) and I see that it has been patched to the newer stuff. It looks like I will have to move to OpenWRT, does anyone know which versions are new enough so that they have this fix? I took a look but could not find a Chnage Log and the versions seem older. Or alternatively is HyperWRT in fact still being maintained somewhere and do you know the new link?

Thanks in advance.

This is a software design issue

[Skapare]

This is software design issue, too. A good software package using config files would have the ability to parse and understand separate files for a default configuration and a locally customized configuration. When such software is distributed, whether in source form that you compile and install, or a binary package you simply install, it will install a default config file that never needs to be updated by the admin/user. To customize such software, a local config file will be written and placed in a different location that the software looks for. The local config will override the default config, when the local is present. The installation of the software will never touch the local config file.

Oh so LATE

[alexborges]

Thanks ./, ive known about this for TWO WEEKS.

And no one died.

So there.

I gasped and the realized the OP is a moron.

[Anonymous Coward]

I have several dns servers internal and external. I slapped the patches on without testing feeling cozy that 1) I have redundant dns servers 2) I took a tar of /var/named 3) I watched the output of the install 4) I stopped/started named and did a dig on the server and from an external site to make sure it was still resolving everything. This article is dumb.

Microsoft

[Frankie70]

Would your response have been if this was done by Microsoft?

Just asking.

Re:Microsoft

[something_wicked_thi]

I love how any time there's a Linux bug posted here and not everyone screams, "Damned Linux, let's switch to Windows", then all the weenies assume it's because there's a double standard.

Please go back and read my post. Notice the bit about "an error of similar proportion"? If you pay really close attention, you might notice there's an implication that RedHat screwed up just as much as the poster.

Microsoft breaks stuff all the time with WindowsUpdate. It's one of the reasons (and, for the subtlety-challenged, please note that I didn't say it was the only or even the main reason) for patch Tuesday. Everyone can schedule their testing and deployment around a known release schedule because simply turning on automatic installation and letting Windows do it on its own never really worked all that well.

Anyone, on any OS, who installs updates on a production server without first testing them, gets what's coming to them.

Don't put all of your eggs...

[mi]

Don't entrust the function like DNS to a single vendor. With some services it is hard, as authors support a limited range of OSes/hardware or charge too high a price for each installation to make redundancy affordable.

But not DNS. Free solutions abound, and the commercial ones are quite cheap too. They are available for all imaginable "server-grade" OS/hardware combination. If you use more than one servers for DNS in your enterprise, and both of them use the same platform, you aren't doing your job.

Mind you, I don't blame the victims here — Red Hat screwed up royally, and that's that. Just advising on how to avoid being hit by such (inevitable) mistakes — from any vendor — in the future.

chroot is not a security measure

[Anonymous Coward]

Why do you run BIND in a chroot 'jail?' It doesn't do squat for security.

Re:chroot is not a security measure

[Anonymous Coward]

Sigh. The old myth of "chroot doesn't work so don't bother" again.

Chroot in conjunction with dropping privileges eliminates a huge amount of risk associated with someone rooting your nameserver.

Is it perfect? Maybe not. Someone might have a way to break out of the chroot jail even without superuser privileges. But if it's possible, it's only known to a small percentage of potential hackers, and only ephemerally because whatever kernel bug allows it will be quickly patched.

Multiple levels of security is a good thing. Chroot'ing is worth doing, even if it's not perfect.

If you want perfect network security, then see Marcus Ranum's "perfect firewall" icon from several years back. It's a wirecutter. Disconnect yourself from the network.

Re:chroot is not a security measure

[mysidia]

These arguments come up all the time. So it is with chroot.

The Linux kernel lost 'securelevel'. ("A hacker can turn it off by mucking around with /dev/mem anyways, or use $kernel_bug_of_the_day to flip the bit")

Python lost 'restricted' mode. (There are some ways to get code out of the restricted jail..)

PHP6 is losing features like safe_mode, open_basedir (Custom extensions may be able to open files despite the open_basedir restriction)

I wouldn't be surprised if chroot itself gets removed eventually, and ext3 'immutable' bit, or gets a fat disclaimer not to use it. It probably only stays because it is used for some build environments.

Why? Because these security measures aren't perfect They don't guarantee 100% security against a skilled attacker. They don't satisfy everyone.

Apparently for some folks, security measures aren't acceptable unless they're effective in 100% of situations and against 100% of the possible attackers.

Even if the measures had some very practical uses... the very danger that 'people might think this is a security measure', is worth removing useful features that make life harder for crackers.

Re:chroot is not a security measure

[sjames]

Why do you run BIND in a chroot 'jail?' It doesn't do squat for security.

If bind is run as a non-root user, chroot has some value. It's just not the ultimate weapon some think it is.

modifited files should not be touched!

[Anonymous Coward]

The user has misconfigured their DNS and has installed a package called, SURPRISE, caching-nameserver along with the other bind packages.

If the RPM utility saw the configuration file was modified (from a mis-matched checksum against its database), it should not have touched the file.

What would have happened if the user was BIND in a caching-only mode, but he modified the file to have a 'listen-on' directive for security purposes? He would have been using the package "in the correct" way, but it still could have borked his configuration.

While he probably should have tested, not having your update system modify your configuration behind your back a pretty good prerequisite to have in an updating system.

Isn't this one of the advantages of having configuration stored in text files? Being able to see changes in a granular fashion? Imagine if it was a binary blob that was updated. Copying back a file (or restoring it from backup) is fairly easy, but if you had some obscure bag of bits it might be harder.

it only breaks things if you are using the caching

[LukeCrawford]

nameserver RHN package, caching-nameserver.arch to serve authoritative zones. it's a caching nameserver, it's not supposed to serve authoritative zones! if you are using the regular nameserver package, bind.arch, it breaks nothing. it keeps the old config and copies the new config to .rpmnew.

And we have another loser

[bruce_the_loon]

RHEL - 5.2 - caching-nameserver-9.3.4-6.P1.el5.i386.rpm

RHEL - 5.1 - caching-nameserver-9.3.3-10.el5.i386.rpm

RHEL - 4.6 - caching-nameserver-7.3-3.noarch.rpm

RHEL - 3.9 - caching-nameserver-7.3-3_EL3.noarch.rpm

it was a linux thing

[Anonymous Coward]

...and you want to see whines about a linux F up on /. ??

Now if this was an issue for MS DNS servers...you sure would have seen the haters trolling this thread...

Dupe :(

[Anonymous Coward]

I don't know how deeply asleep the editors are, but this non-problem has affected RedHat-ish distributions for eons. The proper way of installing this is as follows (on a new system): install bind and caching-nameserver. THEN install bind-chroot. Then UNINSTALL caching-nameserver. Voila.

I can't understand why people expect a caching nameserver not to be one -- after all, caching-nameserver is a convenience package. If you're running anything *but* a cache, you have to configure it yourself, thus *don't* install caching-nameserver.

This story goes into my "lamest stories collection" bucket. Aren't we supposed to be geeks who at least know what we're talking about?!

Does anyone test anything any more?

[Master of Transhuman]

I don't think testing is done by anyone - commercial company or OSS - on anything any more.

I updated to Firefox 3 from the openSUSE repositories, and have been using it for the last week. I had to uninstall it yesterday when the frustration got too great.

Firefox 3 has a serious right-click menu bug, wherein any right click you do may select any item on the right click context menu without bringing up the menu. Hard to believe this wouldn't have been caught in testing.

Also Firefox 3 locks up or crashes on about every third Web site, including sites I have no trouble with in Firefox 2. Can you say "LOUSY JavaScript support"?

For all the hype over Firefox 3, I knew there would be issues when I saw how compressed the time frame was from beta to release candidate to final release.

They simply didn't test the thing. I mean, some Windows users are complaining they can't open Gmail! Gmail! Probably the most heavily used Webmail service - and Mozilla couldn't test Firefox 3 with it?

Pathetic.

Since I had upgraded Firefox 2 to 3, I read where somebody suggested wiping the old profile and reinstalling. I did that - no change.

Firefox 3 is not read for prime time. I will not reinstall it until they get to Firefox 3.1 at least.

Never install a point 0 release of anything. The entire industry simply is incapable of producing a solid product on a zero point release.

No, you are wrong.

[ToasterMonkey]

The DNS server might be one of ten, fifty, hundreds, maybe more different servers that an admin has to care about. The person deploying a machine might not have ANY clue whatsoever about the exact package configuration on a given machine. It might even be a repurposed machine, or one providing general network service to which DNS is added later on. It might have even BEEN a caching DNS server previously!

The whole idea of a package that merely overwrites another package's primary configuration file is absolutely flawed. The two packages should be mutually exclusive.

I'm sick and tired of the Linux crowd berating other systems such as Solaris and Windows because updates often require reboots, or because they sometimes break things (Solaris/sendmail.cf ALWAYS gets brought up), or because single user mode is recommended. You'd get the impression that Linux uses some magical pixie dust, and never needs to be restarted after any update other than a new kernel, and anything other than the kernel can be hot swapped out with a million users logged in using it. Look at the responses here. Something that is obviously poorly engineered by the vendor gets brushed under the rug and labeled a user error. Every time an update screws up some running processes, it's "oh you should have had everyone log off first". When a configuration file gets overwritten it's "you should have tested it outside of production first". Does anyone realize that when you start taking all of these precautions, Linux starts to be just as much a pain in the ass as any other OS? Linux is easy when admins are lazy, and not doing their job.

I fucking HATE "Linux people" now. People pushing it where it's not ready to go, and without understanding why other systems are different first. You're all just as bad as the same ugly windows crowd that was always repressing you a few years back. Now Linux is more accepted, but the fanboys just switched sides, that's all. A new generation of young idiots. Given OS X, Windows, and Solaris, Linux is totally fucking useless. Now, I appreciate and support free software, but let's not get the ideological, philosophical, and technical aspects of Linux mixed up here. I do not want open or free software to go away, but I do want the supporters to try to understand what others systems do differently and not assume the Linux way is best.

Let me calm down... please appreciate open source software because of the openness, not some imagined technical superiority. Linux is far, far, far away from that.

Re:No, you are wrong.

[hughesjr]

we linux people are not very fond of you either ... hopefully you are ready for retirement soon (if you work in the IT industry any way) since you will not be able to work in a datacenter or for a major company anymore.

Re:No, you are wrong.

[buchanmilne]

The DNS server might be one of ten, fifty, hundreds, maybe more different servers that an admin has to care about.

We have over 200 servers. We patched the servers that have a running 'named' first.

The person deploying a machine might not have ANY clue whatsoever about the exact package configuration on a given machine.

They should not be doing a priority patch deployment on that machine then.

It might have even BEEN a caching DNS server previously!

Then the caching-nameserver package should have been removed.

The whole idea of a package that merely overwrites another package's primary configuration file is absolutely flawed.

It doesn't:
$ rpm -qf /etc/named.conf
file /etc/named.conf is not owned by any package
$ rpm -q bind
bind-9.2.4-28.0.1.el4

The two packages should be mutually exclusive.

That would require shipping the same binaries twice.

When a configuration file gets overwritten it's "you should have tested it outside of production first".

We applied the patches on production authoritive DNS servers first, but using up2date (not from RHN). All other servers were patched later using normal procedures. If we had had caching-nameserver installed anywhere, we would have seen the warning from rpm.

Now, I agree that sometimes RedHat should mark some files as %config(noreplace) instead of %config (/etc/issue is one that we have to deal with with a custom rpm that has a trigger on redhat-release just to put our security-policy-mandated banner back), but anyone who was caught by this really needs to review the process they use to deploy priority updates (for all platforms).

Heres a solution.

[cryptodan]

Why not just copy your DNS Configurations to another directory apply the update then move the files back over after the up then service bind start. Along with having a redundant DNS Server.

Gentoo users just laugh at you :P

[ewanm89]

I would like to point out that this is impossible on gentoo, it doesn't update config files automatically, replacing them or otherwise. It includes a nice interface that you can choose the update, the original, diff them, select bits from both files, etc... But it *never* overwrites anything in /etc/ without asking you first.

djbdns ftw!

[thedonvaughn]

djbdns ftw!

Comment removed

[account_deleted]

Comment removed based on user account deletion