Slashdot Mirror
McAfee Retracts Lowball Bug Damage Estimate
bennyboy64 writes "McAfee has changed its official response [warning: interstitial] on how many enterprise customers were affected by a bug that caused havoc on computers globally. It originally stated the bug affected 'less than half of 1 per cent' of enterprise customers. Now McAfee's blog states it was a 'small percentage' of enterprise customers. ZDNet is running a poll and opinion piece on whether McAfee should compensate customers. ZDNet notes a supermarket giant in Australia that had to close down its stores as they were affected by the bug, causing a loss of thousands of dollars."
I thought this affected anyone running XP SP3, which I expect would be a majority of enterprise desktops, not less than half of one percent.
http://blindscribblings.com - Tasty pop-culture in conceptual fashion.
A chain of supermarkets close down, and they only lose thousands
of dollars? Really? I would expect that figure to be a lot higher than that for a single store... Think about all the fresh produce that'll go bad (that have daily deliveries). Think of the power usage (lights, refrigerators). And that's assuming that they aren't paying any of their employees while the store is closed. I'd imagine the loss would be on the order of tens of thousands of dollars per store. Not thousands of dollars across all of the stores...
If a man isn't willing to take some risk for his opinions, either his opinions are no good or he's no good
... why they didn't test the new dat file against Windows system files.
Seriously, we pay them a LOT of money for their product licenses and they cannot even test against known system files?
...If McAfee has a clause in their EULA somewhere that limits their responsibility, and should that be the case, if it is legally enforcable.
Maybe someone with access to said EULA could look it up?
Microsoft once pushed their accountability as a selling point for the Windows Server platform against Linux, if I recall well -- however their maximum responsibility was something like 50$. I wonder what is McAfee's stance in this regard.
I don't know which one anymore I dislike more, McAfee or Symantec. I stopped using both several years ago, I not run Avast Home on my gaming system at home.
"The two most abundant elements in the universe are hydrogen and stupidity." -Harlan Ellison
In Chile, this bug affected operations of the judiciary systems. They had to suspend hearings and other proceedings for the day.
I would bet that the reason the affected numbers are so low is because a large number of corporations know to delay the application of patches for at least a day. This isn't the first time McAfee has done this, and it definitely won't be the last. It's the same concept with Microsoft/Apple/other OS patches. Every organization needs a patch strategy and the good ones include some kind of lab environment to make sure stuff isn't going to break before it's rolled out.
The whole moon and the entire sky are reflected in one dewdrop on the grass. - Dogen
Well, one condition - that the v8.7 McAfee app scanned the svchost.exe file of a WinXPsp3 machine.
Which could happen under three situations:
1. You manually launched a scan.
or
2. A scheduled scan launched.
or
3. A setting in your policy said "scan processes on enable".
Everything here is windows xp sp3 with McAfee installed.
Fortunately for us, all software updates are filtered through and managed by an internal server due to security restrictions on some of the work we do for the government.
It could only effect that few if the policies were set up update infrequently (ever few days or so). My policies are set to check for updates and push them frequently, so I got bitten. I have less than 100 desktops but am a 1 person shop. 4 hours of sneaker net repairs and corporate downtime. Thanks McAfee. There was at least 1 hospital in the area that had to resort to turning non-critical patients away. Don't these things get testing before release? These products are a necessary evil... they don't need to be more evil than the purpose they are attempting to provide.
Heck I was at a small IT security trade event yesterday and like a quarter of the attendees had to cancel because they were dealing with the aftermath...
McAfee had almost a 50% corporate AV market share, and nearly all of those companies still run many XP SP3 boxes. If 10% pulled the DAT before it was yanked, that's a metric buttload of machines...
Is that it would only take 1 oil and gas company who usually handles Million Dollar deals. Lets see.
International Corporation... Lets say 3000+ Employees... lets say just half the company goes down. Rule of thumb is 1 IT guy for every 100 computers (but we all know thats in a perfect world).
So, the simplest way to get out of downtime is to go into safe mode and disable the Antivirus, right? Lets say it takes on average 5 minutes to walk to each machine and preform the steps. 500 minutes, or 8.3repeating hours.
MCaffee basically put you out of business for the day.
People still use McAfee for support, that's laughable.
I imagine this impact was much higher than they stated. One of the small operations I support on the side called a couple of days ago about this issue when it cropped up. The Windows XP computer would not even allow him to do a system restore let aloneuse his computer. Luckily, we found out about the fix yesterday or it could have cost them a couple hundred dollars to fix. Along with the lost productivity time, this isn't a good thing for McAfee. While we use other solutions for our systems, this highlights how much testing needs to take place before a patch is deployed. It's amazing these types of 'issues' occur in today's world. Time for McAfee to step up QA.
I've read a few interviewed accounts where the story was much like this:
We applied the updates, and rebooted, then I went on to kick off the others. When I went back to the first couple of servers, I noticed they had rebooted again... then I knew something was wrong.
I know things can't be 100% perfect in an IT world, and yes, virus definitions can be touchy when sometimes zero-day shit can really cause havoc, but I, myself, have of test boxen on my network that I test all patches/updates/virus definitions on for *NIX and Windows boxen. It's not perfect, because to test and interrogate everything is impossible, but I don't apply things blindly. And yes, I've had a few fallout where the package/patch/update applied fine, but there was a bug in it that affected something. But at least you had some comforting notion that you prepared as best as you could. It just is mind numbing that 1) things still get deployed blindly at the enterprise level and 2) for the amount we all in an IT organization fork out for trust and support from these companies for services and big fallouts are happening.
It is "real time protection" even if that setting is set to "off".
McAfee's documentation specifically mentions turning it off because there is a high processor utilization bug still in it. Although you'd need to read the "read me" file that came with the patches.
Other than that, unless you choose the highest security setting, it is off by default in a BRAND NEW VANILLA install. But not if you had upgraded from a previous version where it was set to "on" by default.
This is 100% McAfee's fault on so many levels.
I feel sorry for that super market chain but: wtf is AV doing on a POS computer?
POS should be a dedicated computer, running one and only one application (the POS software), on a thoroughly shielded LAN, talking to only a centralised server (or small network of servers if one is not enough) that collects the sales data and distributes prices etc. That server should itself be connected only to the POS network and a corporate LAN. In other words: no direct access out of the Internet, no web browsing, no local storage of any data files, no downloading, nothing that could have the most remote risk of a virus.
Or am I missing something here?
McAfee or being part of a botnet?
First, McAfee blew this big time, that such a bug made it to production shows a complete breakdown in their internal processes. XP with SP3 is the number one OS combination in enterprise environments, and should have been the first thing that they tested on. Without doubt McAfee has liability on this and needs to get aggressive about damage control with clients.
That being said, every one of these clients that was hit by this is just as guilty as McAfee is! They are in no better shape and those responsible need to be going management review for their failure. Enterprise Management 101 - nothing goes into production that has not been tested in a lab for pre-pilot and a small group of production computers for pilot! This is as basic as enterprise management gets. Every single environment that was taken down by this shows professional incompetence by their requisite IT departments.
The only question is if it is the fault of management for failing to allow the budget and support needed for a lab for testing or if it is the fault of the IT staffer who never tested things as they should. This is without doubt one of the most public examples of IT incompetence to make the news in years. This is a case of sheer and utter incompetence by every affected party and no pity should be given. If pity were to be given, give it to the poor desktop techs that have to go around making apologies and manual fixes for everything.
A buddy of mine is in IT at a college in the area. This affected almost all of their computers. Although it's harder to put a dollar figure on, the students and professors were NOT happy when all of the computer labs on campus went down, along with a "server" or two. Ever seen professors gets mad ? Now imagine your an IT guy and the professors can't access their online grade books that you pushed them into using. I really think McAfee is going to have a big problem on it's hands come contract renewal time. Pissed off IT people have long memories!
We use Sonicwall's security services, their anti-virus is a crippled version of Mcafee business. And we've been hit hard: Machine where going down but WITHOUT any explanation or any warning messages (this version is silent to the user) and since svchost was killed, no chance of getting in the event monitor or using any tools, it took me couple of hour to figure it was the AV. I am sure they "forgot" to add all those third party security solution who rebrand Mcafee solutions. What is making me mad is the way they try to play with "numbers" (a small percentage, half of a percent...) and the way they hide everything and to act like it didn't happen(go navigate on their website and try to find any information about this bug, they even closed their support form in the peak of the crisis). C'mon if you screwed up, at least PLAY FAIR and be sorry, we might forgive you.Pplaying the ostrich game will make us angrier.
This was hardly the fault of Windows.
100% third-party problem, here... troll.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
Quite apt, even though not POS: http://xkcd.com/463/.
At a certain large semi-conductor manufacturer this false positive wreaked havoc. Most of our IT-supported laptops are running XP. Fortunately I figured out what was going on pretty quickly and knew how to fix it. Other people here weren't so lucky and it took the IT department at least half the day to figure out a solution. Most people were down the entire day.
McAfee must have had a really good sales guy to convince a Project manager that the POS machines needed AV, either that or who ever developed the POS machines didn't decide to secure them with Enhanced Write Filter, SteadyState, DeepFreeze or some other disk write protection so every time the machine is rebooted it loses all its write cache.
Even though it is Windows, there is absolutely no need for AV when the application is so limited.
i work at a Fortune 25 company that was CLOBBERED by the antivirus virus. because we span all timezones, the impact was greatest on the east coast while the west coast was minimal [due to halt in DAT push]--except for those early-risers who connected to the network before business hours.
yes, mcafee really dropped the ball. but it's equally careless not to have it internally tested before allowing ANY updates; moreover, because our corporate image is XP-SP2, our Q&A team could've easily--but didn't/dont--test the DAT on SP2 & SP3.
"McAfee Interwebs Secrutiny has detected that your outgoing mail to customerservices@mcafee.com, subject "You f**king idiotic t**tballs of a son of a ****** in the ******** with a hatstand!!!!" has been detected as Offensive Spam and will be deleted. Thank you for Trusting in McAfee! [TM]"
On a more serious note, I ran into a few small shops that were badly hit, but most of the people I know who work in the enterprise have a time delay before the updates hit the machines, which is usually a hangover from the last time $av_vendor bollocksed up an update.
Personally, I'm still a believer in most AV's being worse that the viruses themselves, and don't run any on my windows boxes - I don't think I've used a single one that hasn't fucked up at some point. Most of my colleagues feel the same way (and, IMHO, by the time it's hit your filesystem and you have that 20% chance of the AV detecting it, it's already too late anyway) and the only reason we run it at work is because of compliance issues... that and the majority of machines being a poorly patched IE6. Yay!
Moderation Total: -1 Troll, +3 Goat
screw corporatespeak
"ZDNet is running a poll and opinion piece on whether McAfee should compensate customers."
Poll? Opinion piece??? This is fucking America. Spare me the nonsense, show me the lawyers.
we have 11K computers
only XP SP3 computers were impacted
whether running Virus Scan 8.7 or 8.5
but in fact less than 100 computers were impacted,
1% compared to our total
one thing that helped
was employees had started to leave after work when update propagated
and they shutdown computer when they leave
it could have been a nightmare
we were very lucky
The world belongs to those who get up early. - I'm far from being the king of Earth then
From my perspective the IT departments that had issues should be to blame. The patch or dat file for an anti-virus program should be treated like any software update. Update one system only, test that nothing serious goes wrong, then deploy the patch to production machinces. Do these guys just allow the anti-virus application to update itself? That seems seriously wrong, and I only blame the IT group for that.
about a day and a half of productivity time at work. Granted some of that was because of how slowly information was passed out. It wasn't til the next day that I found the solution on my own using my own personal notebook and internet connection.
Regardless it was a massive disruption and when you work for a company that has 50,000 customers world wide the task of fixing the problem is massive and the effects of downtime can be disastrous as it spans entire divisions, etc.
On a correctly designed OS:
a) there's no need to run an anti-virus
b) a third-party party software does NOT need to know the admin/root password to do its job
c) a software running without admin/root priviledges CANNOT break havoc in anything but the user account
Tech-savvy companies who switched tens of thousands of XP machines to Linux and were
criticized for doing so by MS fanbois/astroturfers (don't forget to add *that* to your CTO reports
if they were running Mc Afee) are now laughing all the way to the bank.
But, I know dear MS fanbois/astroturfers: nothing to see here, move along, Windows has
nothing to do with this issue right!? Because the Windows family are the most well-designed
OSes on earth right!? It's of course the fault of McAfee (nonetheless on *my* OS there's
no third-party software that can render my system unusable)... And all the paid "reporters"
that make a living by ever only talking about the Microsoft ecosystem would be silly to
cut the grass under their feet by pointing out the *real* guilty one here.
But, no, dear paid MS astroturfer/fanboi, I won't find your answer compelling.
( Title after the VirtualDUB developer's excellent post entitled "Just because it is not your fault does not mean it is not your problem"; http://www.virtualdub.org/blog/pivot/entry.php?id=245 )
Here's the thing.. it's not Windows' fault that some random program deletes svchost.exe , just as it isn't Windows' fault that any app or user can delete ntldr (e.g. a badly designed uninstaller).
But it -is- a Windows problem because without those, it won't start up. So why is Windows even allowing these files to be deleted?
I can't delete by hiberfil.sys even though all it is, is pre-allocated space for the hibernation functionality. If I deleted it, nothing would be lost, and upon hibernation it could re-allocate the required space or tell the user the drive is too full and they're SOL. But no - I simply can't delete it. But I -can- delete vital system files.
So, no.. it's not Windows' fault that McAfee's virus scanner deleted the file. It -is- Windows' problem that they -can- in the first place.
I realize that sometimes there may be a need for a 3rd party application to modify a system file - however rare - but then provide this through a proper mechanism that backs up the original and deletes/replaces on reboot only, with the option to deny the change on boot-up. ( System Restore points only go so far as you'll need the Windows CD/DVD in order to get to the restore utility if you can't boot into Windows anymore. It's also an overly complex solution to the simple problem of renaming files on bootup. )
Read the EULA people, your software, written by "Software Engineers" comes with:
NO WARRANTY
NO FITNESS FOR A PARTICULAR PURPOSE
CONTAINS KNOWN DEFECTS
You paid your money, now you take your chances.
Unlike real engineers, you can't sue a software engineer, report them to some sort of professional body, or seek any type of remedy, besides a possible refund of the money paid for the software.
Aren't you glad you paid the full retail price of windows, the most secure OS ever?
Enjoy the FREE*:
-Digital-Restrictions-Managements
-Viruses, Trojans, Etc
-Internet Explorer
-Shiny New Icons(TM)
*: some restrictions apply, co
There's no magic here. They have a signature that matches a specific version of svchost.exe.
They did not test the scan engine with that dat against that version of the file.
That's all it is.
Don't businesses run their own update server and categorize, verify, and deploy those updates based on what software THEY have running?
If you're telling me that a hospital IT system is setup to take any and all updates directly from vendors( McAfee, Microsoft, etc ) all I can say is they get what they deserve for doing that and it's nobodies fault but their own. Let me guess, this is how most Windows shops are run these days and that is why Windows admins cost much less than *nix admins. IMO
so 4 hours of corporate downtime for this one issue. And why do you not have a few machines configured to represent your standard corporate computers and run the updates on them before expecting some other company to have tested their update with _your_ software configuration? Does Microsoft Windows not give you the power to push out updates locally? The very first time I setup a classroom configuration using Linux it dawned on me that I did not want every computer doing auto updates so I mirrored the Ubuntu repo, setup a cron to keep that updated, and configured all the lab computers to pull from a secondary local mirror where I'd move updates over as they got tested. dah.
LoB
When was the last virus outbreak that caused this much damage?
if companies dont claim large amounts of damage - wait till a virus hits them and they sue the virus writers... and claim... this mcafee incident should be interesting for FUTURE lawsuits... either against mcafee or virus writers...
Even though it is Windows, there is absolutely no technical need for AV when the application is so limited.
Fixed that. I am afraid that the Payment Card Industry (PCI) differs from your opinion.* In their infinite wisdom**, PCI has decreed that ALL computers need to be running AV. After, all, if it is good for the desktop, it must be good for the servers, right? And since a virus can be spread from anywhere to anywhere, all computers need to have their own protection.
I know it seems silly, but many of the PCI Audit Drones actually believe this. I spent hours trying to convince an auditor that we did not need AV on a Linux server that cannot accept email and has no internet connection. If the PCI Audit Drone finds a computer without AV, you fail the PCI Audit. If you fail the Audit, you get marked as failing on a public web site. If you fail enough times, you lose your ability to accept credit cards. So the need to have AV on a POS is there, it is just not a technical need.
*Reality
**For very, very small values of infinite
Great civilizations have lived and died on false theories. Don't mess up mine with a few facts.
a) Windows has serious flaws that exacerbate the problem (only recently did they get something roughly sudo like that is still laughably trivial to bypass, and even then poor third-party implementations that haven't grown out of the Win9x days further torture things), nothing short of disciplined users can do anything to get rid of anti-virus market. So long as a user is actually allowed to execute what they want on a system, some stupid thing will convince them to execute it, and damage/manipulate any data that user has access to.
b) ok, that seems fair enough
c) I concur, but back to point a, most users have all the stuff they care about under their account and aren't mollified that the system files are ok when all their personal documents have been corrupted.
I will say the ability to, in the worst case scenario, boot a system single or log into an existing alternate account to 'clean' the afflicted user account is perfect. I've spent a lot of time trying to rescue a windows system that was malware infected because I couldn't clean it from within the afflicted system (the malware already had control, and did an effective job blocking attempts).
XML is like violence. If it doesn't solve the problem, use more.
Well, low detection rates and excessive hardware utilization didn't get anybody to kick McAfee out of companies, but massive downtime like this certainly get the ball rolling. We are replacing McAfee on corporate computers :-)
Seriously
1) McAfee did not stop a single virus outbreak we had in the past couple of months - i had to send a couple of files to their research labs. Most other anti-viruses already detected the virus if I submitted it to VirusTotal.
2) Hardware utilization for McAfee is INSANE - especially memory utilization. Older PC's just grind to a halt.
Why does McAfee have so many corporate customers? Their software simply does not work!
Just because it's a hospital doesn't mean it has an IT department much more elaborate than a server admin and 2 techs, or that it has budget for the kind of stuff you're talking about.
If you can't see the value in jet powered ants you should turn in your nerd card. - Dunbal (464142)
There is just no way that Mcafee can make whole everyone that they screwed up,the only ones that will make the money are the lawyers. What good is that going to do? I say make Mcafee fire everyone that way a part of the mistake,and 1 year of free virus updates to the affected company's. Just keep the dam lawyers out of it.
Jack of all trades,master of none
We managed to keep our labs up during this debacle but the staff and the profs were hit pretty bad. Apparently UVU had to close down all of their labs. The funny thing is that just few days before, our windows guys were talking about finding an alternate solution to McAfee. This pretty much made their minds up for them.
Be gone from my sight or prepare to feel my flaming wraith!
All that time the computer weren't running windows. I tought that at the end of the day the economic balance should have been positive.
I can see it now. Mesothelioma, YAZ and now McAfee lawsuit ads trolling for money.
Conservative, mod down for violating
I'm going out on a limb here and guessing that you aren't a native English speaker. It looks like you're well on your way to fluency, but you should really take some more writing classes. Your post contains a lot of easily-correctable flaws.
I am a native French Canadian, and yes my English need improvement, thanks for your understanding.
..if a certain option, "Scan Process on Enable", was turned on. That option is disabled by default. We run XPSP3 and McAfee 8.7i and machines loaded the bad DAT but we were unaffected because when I configure the policy for VirusScan in EPO I did not turn the "Scan Process on Enable" option on.
I don't always use unix-like operating systems; but when I do, I prefer FreeBSD.
I was working with a group of employees from a MAJOR computer systems company, and suddenly no one could use their PCs - we were about to set up a Web conference call, and luckily one of the personnel had been traveling and had not hooked up to the corporate network for a day or two. He was the only one with a functioning PC. It was pretty embarrassing...
(I have a MacBook myself.)
The corporate IT departments that are using McAfee should just take this as a lesson and pull McAfee off of their computers. It's not as if McAfee AV is held-up by users as this amazing piece of software. See. See.
Corporate IT departments need to get the message that McAfee is a piece of junk -- in that sense, McAfee kind of did them a favor...
I spent hours trying to explain we don't need AV on old mainframe system, which use an operating system only known from a few grey beard. In the end she only accepted it if we signed a document that we take responsability or something, with the management. *sigh*
In a corporate environment, what you say comes close to flying (practically speaking, it's horribly expensive to have enough IT to cover all the edge cases to get enough productive work), but even then the user is able to spawn executables of some sort. I guess if /tmp, /var/tmp, and /home in a *nix env are mounted noexec, you're pretty much where you describe, and I suppose Windows is disadvantaged from that perspective.
That, however, ignores the home market where most everyone is their own administrator.
XML is like violence. If it doesn't solve the problem, use more.
The corporate IT departments that are using AV should just take this as a lesson and test signature updates. It's not as if AVs don't do this sort of thing now and again.
Corporate IT departments need to get the message that AV vendors don't test against all hardware and software configurations - in that sense, McAfee kind of did them a favor...