Slashdot Mirror
HP Server Killer Firmware Update On the Loose
OffTheLip (636691) writes "According to a Customer Advisory released by HP and reported on by the Channel Register website, a recently released firmware update for the ubiquitous HP Proliant server line could disable the network capability of affected systems. Broadcom NICs in G2-G7 servers are identified as potentially vulnerable. The release date for the firmware was April 18 so expect the number of systems affected to go up. HP has not released the number of systems vulnerable to the update."
And this is why I wait at least 2 months before installing firmware updates (unless it's a major security issue). It's not uncommon for a firmware update to be pulled shortly after being published. The 2 month window delay is generally more then enough time to ensure it's a proper update is solid.
Life is not for the lazy.
We pushed a firmware update this morning to the firewall and its been smooth sai#*($$#[NO CARRIER]
Good people go to bed earlier.
...don't flash it.
Do admins routinely flash firmware updates in the absence of some identified need? I could see flashing an update if I was suffering from a known problem, or if the vendor identified a security flaw in a previous release. I could see flashing it if necessary to install new hardware.
I just don't see why a server admin would flash a firmware update as if it were Patch Tuesday. In the absence of a security vulnerability or production issue there is no reason to treat a firmware change as an expedited change and not perform full testing before deploying it. That isn't to say that doing some testing of security patches/etc isn't wise - but I can see why it would get rushed.
You don't flash firmware unless it is for an important issue.
Or at least not until it has been out quite some time so that other people have done your testing for you.
Aren’t they also the ones who limit their firmware updates only to customers who have support contracts? I guess you get what you pay for..
http://slashdot.org/story/14/02/05/0258244/hp-to-charge-for-service-packs-and-firmware-for-out-of-warranty-customers
Known for reliable oscillators and calculators, and then they made a line of laser printers that lasted for a while; great engineers behind all that stuff too. Yes, I remember them. How are they doing now post-Carley? (HP's calculators put Rockwell's to shame. I can still remember the Rockwell jingle from over the radio, "big green numbers, and little rubber feet.").
You can't be ahead of the curve, if you're stuck in a loop.
didn't brick my server but it screwed up the device list in Windows and caused a cluster not to see the one node where i upgraded the drivers/firmware. put a null device driver into the device manager and i had to delete it and all was OK. just $250 to MS to figure this out since didn't think it was a HP issue
on the server the network worked and all but the NIC's weren't "seen" by Windows and so the clustering was screwed up
It's probably a mercy killing. Some of those poor servers were probably forced to run HP/UX. I'd want to die, if I had to run HP/UX...
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
That was available for a week.
There are two types of people in the world: Those who crave closure
Unless the executives don't give you 'non-critical boxes' for every piece of infrastructure to test updates.
"Why do you need an additional SAN at $100k? We'll deal with that if it happens. It happened? It's all your fault!"
There are two types of people in the world: Those who crave closure
It's why you should always have your resume' up to date and another potential job in your pocket. Easier said then done, I know. Sysadmin positions are always inherently sacrificial lambs for when shit hits the fan. Deal with it, or find another industry to work in. Just some friendly sage advice to offer you.
Life is not for the lazy.
As a network engineer, I can see being involved in arguments between the server platform support teams (read: off-shore) and the network engineering teams (read: on-shore). It'll be like this; "we need network support on a call" "Hello. what's wrong?" "The entire network is down for everyone!!!!! You need to fix this!!!!! The support we get from you is horrible!!!! AAHHHHhhhhhhh!!!!!!" "OK. What changed? What was being done at the time the entire network disappeared for everyone?" "we (15 people on the call - it apparently takes that many) were doing nothing (to do nothing)." "OK, well, I'm on the cores and I can see a lot of traffic, other servers, the outside world etc. you need to define the "everything being down" part." "well, we were in the middle of doing a firmware update on server xxx01 and...." "OK, so, you lied to me about doing nothing. what did you update?" "the NIC card to improve performance for..." "And now you're wondering why the network is down..." It'll go this way for some time until the next couple of layers of management get involved....... lots of yelling, me sending pictures of the network working I should write a script for this call. I know it'll be coming.
You know, if that's how your company is being ran, you should already be looking for another job.
Where I work, we've got proper test equipment, a CAB to review the proposed changes, and an expectation that you will test before deploying. When we schedule outages, we have to have a backout plan, and we're expected to have applied the updates in either the lab or a test environment.
The admins aren't considered sacrificial lambs, but they are expected to apply due diligence, test, and identify any risks. But once you've done that and made sure people know what you're doing and why, what the results of your testing is, and what you've done to mitigate any risks ... a bunch of senior people in IT have signed off on it and people have had a chance to voice their concerns. The people overseeing this tend to be department heads with a lot of industry experience, so they understand there is always risk, but they also understand what you need to do to minimize it.
If your company refuses to give you what you need to do your job without being able to do these things, your company is sailing straight towards a major disaster with or without you.
If your company is treating it as "stop talking and do it" combined with "but if you do it wrong you're SOL" ... your company is being managed by people who don't understand what is involved in your job, and will always have unrealistic expectations.
Companies which don't plan for these things, don't build a proper process around it, and don't fund being able to ensure things get tested are just being penny wise and pound foolish.
And, from a certain perspective ... I would never even consider applying a patch to a production environment which had only just been released by the vendor. At least a month, maybe as much as two. If someone wanted to put a firmware update on my production systems which was only just released from the vendor, the answer would be a firm "no bloody way". And my manager, and his manager, and all of the other people at that level would also be saying the same thing and back me on that position.
You have to have a company culture which owns the process, takes responsibility for it, and actually takes the time to understand the impact of it and plan for it.
Now, if a system admin does any of these things without going through all of the process, and things go wrong ... then you likely will be neck deep in crap pretty quickly. But if you have followed the process, and something goes wrong, the process shifts to remediating what went wrong, and understanding what can be done better next time. It has to be a continuous process, and it has to actually have some institutional memory, and companies have to take the process seriously.
Lost at C:>. Found at C.
I agree with you. But...
If your company refuses to give you what you need to do your job without being able to do these things, your company is sailing straight towards a major disaster with or without you.
Which is the majority of SMB companies out there. They also make up the majority of the work force.
Must be nice to work for a fortune 500 company to have the resources available...
Life is not for the lazy.
Don't manufacturers test their updates? it's not like they couldn't keep some of the stuff they sell for said testing...
I've got better things to do tonight than die.
You don't need to be a Fortune 500 company to apply this level of rigor. I'm quite sure we're not one.
Yes, you need resources to do it. Yes, you need corporate will to do it. And you also need to have a company whose culture includes actively assessing risk against their needs, as well as understanding how the risks translate into business risk. If the systems affect the actual production of your business, you need to treat it as Very Important.
If you stand to lose millions of dollars per hour in the event of an outage, the cost of screwing up gets pretty high. Which means the expense is absorbed. If you have much less exposure due to an outage, your tolerance to risk is going to be much higher.
My wife does outsourced/leveraged IT ... and some of her clients, if some environments are down, basically have to halt all production, shut down equipment, and go through an expensive restart process.
Even at the SMB scale, you need to understand your risks, and have management be partly responsible for the decision making process, as well as having people who can provide the information needed to make decisions. These shops may not have the resources to test and deploy everything to a lab, which means, if anything, they should be staying away from applying a brand new patch as soon as it's released.
Lost at C:>. Found at C.
Is the article suggesting that the Broadcom NICs that HP used in the old Proliants actually _did_ work before this update?
That goes against years of experience in the field with those things.
With HP only one problem persists. Namely, it is what the money cant do.
This is one of the first rules of administering servers -- unless it's an absolute necessity, let someone else find these firmware bugs.
This is especially true now that firmware controls so much in modern hardware. I've had business PCs that have gone through more than 10 EFI revisions in their 18 month lifecycle, and all the release notes show that they fix surprisingly low level things.
The unfortunate trend is that these firmware bugs are more and more prevalent. It seems like manufacturers are skimping on QA and testing. I'm not surprised that HP is affected -- their maintenance applications and documentation look like it's now written by an offshore team. So, I wouldn't be surprised if the EEs and SEs sitting in Houston have to write specs and have their offshore counterparts hack up the firmware changes. Worse, since they're getting the NICs from Broadcom, it's engineers --> offshore team --> Broadcom --> Broadcom's offshore team, making it even more likely that confusion will be introduced.
Which is exactly why I do not do ITIL.
Borne from a bunch of technocrats with nothing to do all day long but twiddle their thumbs, ITIL gives any spy organization an exact behaviour profile of your organizations entire operations.
You implement ITIL and your already half way to having your infrastructure compromised.
Got Geometrodynamics? Awe, too hard to figure out? Too bad.
I deployed a DL380p Gen8 last year, and it gave me heart failure.
Under Red Hat, I needed to change the IP address, so I modified the file /etc/sysconfig/network-scripts/ifcfg-eth0 then did a "service network restart"
Alas, the box did not come up on the new IP. Got to the console which was blank and unresponsive. Power cycled, and the RAID array was GONE (and let's just say this was EXTREMELY inconvenient timing).
Support was able to walk us through some BIOS disk recovery that (thankfully) worked. But I'll never change the IP address on a Proliant without a full reboot.
Their last update bricked broadcom blades of the G1 variety.
If an SMB does not have the resources to operate then they will not be successful. It doesn't matter what the resource is, office space, electricity, phone lines, etc. People who run SMBs seem to think electricity is important, but IT is not.
I see this all the time, an SMB customer buys telephone or internet service from the company I work for and then the customer thinks we have become their IT department. They are desperate for IT help, but not desperate enough to actually hire someone to take care of it. I have little sympathy for such companies.
I really shouldn't have used someone else's email address for this account.
We use ITIL, and the CM part definitely has saved us from major issues.
And some of the problems we see would have been avoided if the teams responsible had followed the CM process.
The more complex the organization is (we are a truly global company), the more you need structure.
Structure is fine.
But a canned structure isn't, and no organization benefits from a canned structure it uses that either its competitors use, or its enemies.
Really this has been demonstrated so many times I am quite frankly puzzled why people/organizations feel the need to copy a plan to organize infrastructure, when more than likely they had exactly ZERO input to the process.
Google realized this when it created its infrastructure and came up with thier own plan.
Everyone should do the same, it isn't that hard really. Are people such simpletons they can't figure out what the best plan is for your organization to structure security and software updates that you have to refer to a bunch of non experienced, government bureaucrats who wrote ITIL?
I think not.
Got Geometrodynamics? Awe, too hard to figure out? Too bad.
ITIL is a recommendation. As you mention, implemeting ITIL by the letter is simply wrong. But inventing a system from scratch is beyond the capability of most organizations (believe me, our company tried and failed miserably befoe adopting ITIL). Why should everybody reinvent the wheel?
Pardon me, but I've never flashed firmware, but ... "Always have a reasonably quick way to revert if stuff like this happens."? How is this possible on a firmware upgrade?
I think we've pushed this "anyone can grow up to be president" thing too far.