Former Equifax CEO Blames Breach On One Individual Who Failed To Deploy Patch (techcrunch.com)

Equifax's recently departed CEO is blaming the largest data breach in history on a single person who failed to deploy a patch. TechCrunch reports: Hackers exposed the Social Security numbers, drivers licenses and other sensitive info of 143 million Americans earlier this summer by exploiting a vulnerability in Apache's Struts software, according to testimony heard today from former CEO Richard Smith. However, a patch for that vulnerability had been available for months before the breach occurred. Now several top Equifax execs are being taken to task for failing to protect the information of millions of U.S. citizens. In a live stream before the Digital Commerce and Consumer Protection subcommittee of the House Energy and Commerce committee, Smith testified the Struts vulnerability had been discussed when it was first announced by CERT on March 8th.

Smith said when he started with Equifax 12 years ago there was no one in cybersecurity. The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team. However, Smith had an interesting explainer for how this easy fix slipped by 225 people's notice -- one person didn't do their job. "The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not," Smith, who did not name this individual, told the committee.

Comments

Ob

[Hognoxious]

He's Spartacus!

Nice to have Cyber Security Team

[avandesande]

Sucks that you don't do configuration management.

Re:Nice to have Cyber Security Team

[Tokolosh]

What do the other 224 do?

Re:Nice to have Cyber Security Team

[houstonbofh]

What do the other 224 do?

Apparently, not watch that one guy...

Re:Nice to have Cyber Security Team

[Anonymous Coward]

Meetings to plan the schedule of that one guy.

I smell bullshit.

[Hylandr]

If .25Bn has been invested then there's sure as hell no process that could have allowed a single critical patch go unchecked as described. There's teams, or should be teams of people watching these things.

I smell a really shitty cop-out excuse.

Re:I smell bullshit.

[rahvin112]

You missed the best part, 3 years ago, they didn't even have a security department. At least according to his throw the wage slave under the bus testimony. He's distracting you with this tale of rouge employee while dropping a bombshell you didn't even notice.

3 years ago the company responsible for approving credit for all americans had NO information security department. According to the CEO's testimony they had zero budget and not a single employee dedicated to security of their IT networks. That's grounds for jailing him IMO.

Re:I smell bullshit.

[Hylandr]

I caught that part but was much more incensed by the lame attempt to parry liability.

Re:I smell bullshit.

[fahrbot-bot]

3 years ago the company responsible for approving credit for all Americans ...

Technically, Equifax and the other credit bureaus don't approve credit to anyone, they simply provide a centralized source for credit information. Individual lenders make approval decisions based on this information - which is available to, and can be challenged by, the borrower.

Re:I smell bullshit.

[dszd0g]

It's either utter incompetence or bullshit.

At the enterprise level and especially for PCI compliance there should be 3 independent levels where this could have been caught: 1) applying the patch, 2) monitoring patch compliance, 3) vulnerability scanning. Organizations that really care about security also have a Web Application Firewall (WAF) or other Intrusion Detection System (IDS) or Intrusion Prevention System (IPS) which would have been a fourth level that could have prevented this attack.

Blaming this attack on one person when there should have at least been 3 levels of prevention with at least 3 different teams involved is stupid.

1) Patch Management Solution: In the enterprise, this should be a software solution (like Quest KACE or IBM BigFix type solutions) that monitor the patches on each endpoint and apply patches on a schedule after they are tested. Most organizations have a 30 day patch cycle although critical remote vulnerabilities like this should have been escalated sooner.

What would have been reasonably possible is for the person responsible for escalating the patch to apply sooner than 30 days could have missed escalating it. However, the normal 30 day cycle then should have caught it.

a) Patch application
b) Patch monitoring

In some organizations there is one team that applies the patches (and is usually involved in testing the upcoming patches) another team that monitors the patch levels. In other organizations they are the same team although there should still be independent checks for application and monitoring.

2) Vulnerability Scanning: Especially anything that is visible to the Internet should get vulnerability scanned at least every 30 days. A decent remote vulnerability scanning software should have picked this up. Tenable's Nessus which is one of the industry standard vulnerability scanners tests for CVE-2017-5638 which is the vulnerability that effected Equifax. Nessus started testing for it on March 14th.

3) Web Application Firewall: Web Application Firewalls will block known attacks before they hit the application. A decent WAF should block known vulnerabilities such as the one that hit Equifax as long as it was up to date. That said a lot of companies I have worked with tend to run WAFs in intrusion detection mode instead of intrusion prevention mode due to false positives and not wanting to block legitimate traffic. Some companies I have worked with are much better than others at going through the alarms, how quickly they respond to alarms, and filtering out the false positives so that the alarms are easier to manage. Usually for Web applications you will have a WAF rather than a general purpose IDS/IPS as the WAF will have access to the unencrypted traffic although there are ways to have IDS/IPS products have access to the Web server private certificates to decrypt the traffic.

Re:I smell bullshit.

[Hylandr]

After reading this it occurs to me that it's much more likely someone sold the info rather than had it hacked.

Re:I smell bullshit.

[shaitand]

"If .25Bn has been invested then there's sure as hell no process that could have allowed a single critical patch go unchecked as described. There's teams, or should be teams of people watching these things."

The .25Bn and teams of people sounds like the problem to me. Like in every other large enterprise environment that leads to a whole lot of procedures and massive dysfunctional security theater. The security people want everything silo'd, disconnected, and to tie the hands of ops in every way possible and then want everything fully patched on a schedule that simply isn't feasible at that scale operationally. Generally there is something like a 90 day requirement on critical patches... in a large enterprise environment you probably won't even find out all the applications that are running in 1-5yrs, let alone keep up with every patch for every application that doesn't get bundled with OS patches. And of course half their environment is probably ancient EOL or EOL for everybody who couldn't get the vendor to make a super special exception that isn't actually backed by any resources and therefore on some kind of exception list that makes them compliant.

the tree must be watered

[retchdog]

To quote Thomas Jefferson, "The Tree of Bare Fucking Minimum Standards of Responsibility and Decorum must be refreshed from time to time with blood."

Human Error???

[Moblaster]

Anyone who has worked with sensitive processes (esp computer security processes) knows that relying on one person for a mission-critical function is not a "human error" - it's a process failure. If this person's communication job was that essential, they should have had a team-based process in place with multiple individuals charged with making sure the process got executed, backed up by computerized records and nag alerts if not done. Seems like this "human error" would have happened if the person had gone on vacation, gotten fired, or went off their meds. That's not a human error. That's execs failing to make sure they build a resilient security process. Quarter billion in expenditure won't buy common sense, it seems.

Re: Human Error???

[Mr D from 63]

There's a thing called independent verification that might have helped. Guess its that one guys fault that they didn't practice that.

Re:Human Error???

[msauve]

"Anyone who has worked with sensitive processes (esp computer security processes) knows that relying on one person for a mission-critical function is not a "human error" - it's a process failure."

Absolutely. Human redundancy is just as important as network/system redundancy. If the organization isn't set up to continue working even if someone gets hit by a bus, that's a management failure. It's not a single individual. Who was responsible for checking that the work was done as required?

Re:Human Error???

[Anonymous Coward]

Even Mormons are deployed with full redundancy.

Re:Human Error???

[Bert64]

Exactly... You should have defense in depth, not only to counter someone who fails to apply a patch but also to try and mitigate against attacks against vulnerabilities for which there is no patch.

Re: Human Error???

[ausekilis]

At the retirement party for one of my best managers I asked him one question: "What's your secret to being a great manager?" his answer: "My only job is to enable my people to do theirs."

Sure, one person may not have applied the patch. But it wasn't that persons fault that a process wasn't followed, that the appropriate funding was available, that assorted checks and balances were in place, and that IV&V didn't happen. That falls on management.

This is the same scapegoating VW tried to get away with in their "Rogue Engineer" nonsense.

Ah yes, the blame game

[quonset]

"It was his fault. That's why I sold my company stock when I found out about the breach rather than inform anyone except the other folks in the executive suite."

Wow, that's scummy

[JohnFen]

"The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not,"

What a scummy thing to say, and he doesn't even realize that the statement makes Equifax look even worse.

With a couple of hundred people on the security team, the idea that it's a single person's responsibility to tell everyone to apply a patch is ludicrous. If it's true, then that's institutional incompetence.

I've been working in computer security for years, and do you know what I and all of my coworkers do? We keep up on computer security developments, particularly newly discovered vulnerabilities. And we discuss them. And send emails about them.

Even if the one team (not individual) who is responsible for ensuring that our own systems are patched for some reason fails to do that job, there is exactly zero chance that this would go unnoticed.

If that's not how it works at Equifax, that's the fault of Equifax, not some single individual.

Re:Wow, that's scummy

[JohnFen]

I'm not so sure how effective that would be. In most of the places that I've worked, nobody takes vacation time unless management forces them to.

huh?

[kefalonia]

bollocks. Yes, that.

Any security organization which relies on a single individual's action or inaction to remain in good standing is simply fairytale.
Every good process which involves a human in the loop, should always ensure that at least one more is present to enforce check-and-balance objectives.
There is a good reason why all commercial flights have two pilots as a default.

Let me state this: when you see management pointing one single downstream individual for such an event, there are at least TWO levels of management at fault.

Such BS

[gordona]

The buck stops with the CEO! If the CEO knew about vulnerability that needed patching, he should have been expecting a report regarding the application of the patch. If he didn't get that he should have come down on the admin or system owner for not installing it. Unless of course that wasn't in the security policy in which case it still falls on the back of the CEO. DUE CARE and DUE DILIGENCE! Non existent.

$225 million isn't much

[phalse phace]

The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team.

Spending $225 million over 3 years isn't really that much when you consider the type and amount of personal data Equifax has on us.

JP Morgan Chase spent $500 million in 2016 alone, Bank of America spent $400 million on cyber security in 2016 although they have an unlimited cyber security budget, Citibank's cyber security budget topped $400 million and Wells Fargo spends roughly $250 million per year.

Re:$225 million isn't much

[mentil]

All of those you cite are banks with numerous branches, subject to robbery and internal theft. They have security cameras which send their video over the internet, all branches are connected to multiple financial networks including their own, and lots of mundane paperwork is computerized. Securing all of these things counts as 'cybersecurity' and goes beyond what Equifax has to deal with, for the most part. If someone breaches/hacks Equifax, and they can ignore it/cover it up, then it's business as usual, so why spend money on it? It's only once the mandatory disclosure laws went into effect they took cybersecurity seriously.

Failure of way more than one person

[Todd Knarr]

Failing to apply the patch would be the failure of that one person to order the patch applied, plus the failure of his superior to notice that an action item hadn't been handled, plus a failure of the security team to notice that a ticket hadn't been completed, plus the failure of the head of the security team to notice his subordinates had uncompleted tickets sitting there. All this stuff should be tracked, and where I work it is and we have daily status meetings where stuff like this gets asked about, and development team managers and product managers have weekly status meetings where lack of progress on tickets and what needs done about it is a standard agenda item.

Accountability means managers and executives are just as accountable for work getting done or not getting done as low-level employees are expected to be.

Re:Failure of way more than one person

[dgatwood]

Failing to apply the patch would be the failure of that one person to order the patch applied, plus the failure of his superior to notice that an action item hadn't been handled, plus a failure of the security team to notice that a ticket hadn't been completed, plus the failure of the head of the security team to notice his subordinates had uncompleted tickets sitting there. All this stuff should be tracked, and where I work it is and we have daily status meetings where stuff like this gets asked about, and development team managers and product managers have weekly status meetings where lack of progress on tickets and what needs done about it is a standard agenda item.

Plus a failure of their regular security auditing process to detect that a machine was running a version of software below the minimum allowed version. All this stuff should be detected programmatically in a company that size. This was not a failure of one person. This was a complete failure of the entire security organization at every level, which usually points to either a complete lack of leadership, inadequate budget to hire sufficient qualified staff, or (more likely) all of the above.

Re:Failure of way more than one person

[avandesande]

It's complete garbage excuse. It's not like somebody reads sit on Slashdot and gets an idea to apply a patch. There are bulletins like this https://www.us-cert.gov/ncas/b... that you compare with your inventory. When you find something you open a ticket.... there are probably ways automate this. (I am not a cyber guy)

Re:Failure of way more than one person

[Hylandr]

By this time it's very obvious it's nothing to do with a security team.

This is just smoke from the CEO trying to protect his ass.

Re:Failure of way more than one person

[dgatwood]

This is just smoke from the former CEO, trying to avoid prosecution.

FTFY.

Re: Failure of way more than one person

[Monster_user]

I think the comment you were replying to, the one which said "weasel words", is expressing frustration that the technology in our jobs is becoming increasingly over utilized, requiring unrealistic levels of up time for the lowest level of funding. Meaning no downtime.

To put it in laymens terms, the servers are suffering from an alternative to "burn-out", where patches are not applied, infrastructure isn't replaced, and/or eventually internal support has to be provided for products which are no longer viable for vendors or consultants to commercially support.

There are only so many corners to cut, and some can only be cut for so long, even in the name of profit. And those making the decisions don't have the acumen to understand the yin/yang of it.

Struts being an application framework...

[hattig]

Struts is an application framework, which means it is an application dependency. That means that every Struts-using application within Equifax would have needed to be upgraded, to be tested at least on the new version. That is the job of more than one person!

It is possible that Equifax's application servers (Tomcat, JBoss, etc) were configured with Struts being provided at the container level, but even that would be a full upgrade of multiple application servers within the company - a platforms team responsibility. However I suspect Struts would have been incorporated into the application itself at build time (as a dependency library).

I do not know how many applications Equifax's systems are made up of, but certainly the company I work for has dozens or hundreds to build up a trading platform (or two or three!). I imagine it is similar at Equifax.

I also cannot imagine a security team of 225 people having just one person be responsible to notifying and reminding of critical library vulnerabilities and updates for the entire business.

This smells of "VW Single Rogue Engineer" to me. Clearly bullshit.

"The buck stops somewhere else"

[david.emery]

Sign on the desk of CxO's everywhere

(contrast this with the US Navy, where the captain of the Fitzgerald was relieved, even though he was not on deck when the collision occurred and in fact was almost killed by the accident. Subsequently, the Navy relieved several higher ranking officers, including Flag officers, for supervisory failures.)

So what you're saying is

[rsilvergun]

Your entire operation is one under paid and overworked sys admin away from disaster? Did I get that right?

Yep, one single bad it guy....

[burtosis]

Reminds me of the time 'a couple of rogue engineers for the whole VW emissions fiasco. I think handsome bonuses are in the works due to management for uncovering this subterfuge.

Re:Horseshoe nail

[MountainLogic]

Unless you have a buggy.

Hope the scapegoat got paid off too!

[ErichTheRed]

Expecting the CEO to know _anything_ about what goes on in the IT department is expecting too much. Executives have no clue what's going on outside of the boardroom, and the only time they ever get any sort of information is from management consultants or the odd 'red alert' that bubbles up to the CFO/CIO/COO/CSO. There is absolutely zero chance that the CEO of Equifax has any idea what patch level of Apache Struts is running on their Internet-facing services.

I wonder if he just went to the CIO and said, "give me a name, anyone remotely responsible for patching, so I can say I fired someone over this." I've never had it happen to me, but I have worked with people who were scapegoats in a major incident. Sucks when you're the one holding the bag...

what a bs.

[kiviQr]

blame one person for no security. Company with that data should assume their webserver will get hacked and act accordingly by implementing multiple layers of security. Web server should have been in DMZ with limited view to data (and no access to sensitive data). That is 101 security. $225m/3y where did that go? To an audit that showed nothing?

Miserable little fuck

[satan666]

What a miserable, no good, lying, sniveling, double crossing, douchebag, fuckface, fucktard, dickwad lying little bitch.
From his resignation letter:
"I'm outta here suckers! Let me throw a few of you worms under the bus on my way out. Not my fault. Fuck you and goodnight."
Love, dickwad in charge, Ret.
P.S. Bitch better have my moneyyyy!

Entire Team/Policy FAILURE

[Anonymous Coward]

This wasn't a single person failure. If it was, that means the policy that setup that single person to be able to fail a mission critical issue is at fault. Also at fault is the actual PROTECTION OF DATA! How in 2016 and 2017 does ANY COMPANY have UNENCRYPTED PERSONAL INFORMATION on ANY COMPUTER/DATABASE which is attached to the INTERNET?!??! And this is in a company that is touting that it has spent billions on cybersecurity. Sure you may have spent the money on cybersecurity, but you certainly didn't take their advise and spend the money to change your processes which relied on using unencrypted data!!!!

It's even worse

[PatientZero]

Any number of reasonable things could have caused the patch to be missed, but you'd expect $250M spent over three years to provide a few more security processes beyond, "Fred forgot to apply the patch." The attackers were spreading through their systems over several months without detection.

Also, way to lead from behind. Every corporate officer I've met has shared one tenet with all others: they are responsible for everything that their team does, good and bad. If some employee several rungs down the corporate ladder fails, it's because the leadership above them failed to hire or train them correctly or put in the right processes.

Engineers Should Pay Attention to This

[Anonymous Coward]

If you work in engineering, you need to see the writing on the wall. No longer are you going to be indemnified for mistakes you make at work, even if you are forced to make them by bad management policy or lack of basic resources. No longer will the penalty for grievous error be a simple firing.

Face the music. If you make a mistake that causes what ends up being a tortious harm, you are going to jail.

Like any CEO

[stabiesoft]

Any good stuff that happens I did, give me a big bonus. Any bad stuff that happens, blame. The old saying *hit rolls downhill has never been truer.