Former Equifax CEO Blames Breach On One Individual Who Failed To Deploy Patch (techcrunch.com)

Equifax's recently departed CEO is blaming the largest data breach in history on a single person who failed to deploy a patch. TechCrunch reports: Hackers exposed the Social Security numbers, drivers licenses and other sensitive info of 143 million Americans earlier this summer by exploiting a vulnerability in Apache's Struts software, according to testimony heard today from former CEO Richard Smith. However, a patch for that vulnerability had been available for months before the breach occurred. Now several top Equifax execs are being taken to task for failing to protect the information of millions of U.S. citizens. In a live stream before the Digital Commerce and Consumer Protection subcommittee of the House Energy and Commerce committee, Smith testified the Struts vulnerability had been discussed when it was first announced by CERT on March 8th.

Smith said when he started with Equifax 12 years ago there was no one in cybersecurity. The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team. However, Smith had an interesting explainer for how this easy fix slipped by 225 people's notice -- one person didn't do their job. "The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not," Smith, who did not name this individual, told the committee.

Comments

Ob

[Hognoxious]

He's Spartacus!

Nice to have Cyber Security Team

[avandesande]

Sucks that you don't do configuration management.

Re:Nice to have Cyber Security Team

[Tokolosh]

What do the other 224 do?

Re:Nice to have Cyber Security Team

[houstonbofh]

What do the other 224 do?

Apparently, not watch that one guy...

I smell bullshit.

[Hylandr]

If .25Bn has been invested then there's sure as hell no process that could have allowed a single critical patch go unchecked as described. There's teams, or should be teams of people watching these things.

I smell a really shitty cop-out excuse.

Re:I smell bullshit.

[rahvin112]

You missed the best part, 3 years ago, they didn't even have a security department. At least according to his throw the wage slave under the bus testimony. He's distracting you with this tale of rouge employee while dropping a bombshell you didn't even notice.

3 years ago the company responsible for approving credit for all americans had NO information security department. According to the CEO's testimony they had zero budget and not a single employee dedicated to security of their IT networks. That's grounds for jailing him IMO.

Re:I smell bullshit.

[Hylandr]

I caught that part but was much more incensed by the lame attempt to parry liability.

Re:I smell bullshit.

[dszd0g]

It's either utter incompetence or bullshit.

At the enterprise level and especially for PCI compliance there should be 3 independent levels where this could have been caught: 1) applying the patch, 2) monitoring patch compliance, 3) vulnerability scanning. Organizations that really care about security also have a Web Application Firewall (WAF) or other Intrusion Detection System (IDS) or Intrusion Prevention System (IPS) which would have been a fourth level that could have prevented this attack.

Blaming this attack on one person when there should have at least been 3 levels of prevention with at least 3 different teams involved is stupid.

1) Patch Management Solution: In the enterprise, this should be a software solution (like Quest KACE or IBM BigFix type solutions) that monitor the patches on each endpoint and apply patches on a schedule after they are tested. Most organizations have a 30 day patch cycle although critical remote vulnerabilities like this should have been escalated sooner.

What would have been reasonably possible is for the person responsible for escalating the patch to apply sooner than 30 days could have missed escalating it. However, the normal 30 day cycle then should have caught it.

a) Patch application
b) Patch monitoring

In some organizations there is one team that applies the patches (and is usually involved in testing the upcoming patches) another team that monitors the patch levels. In other organizations they are the same team although there should still be independent checks for application and monitoring.

2) Vulnerability Scanning: Especially anything that is visible to the Internet should get vulnerability scanned at least every 30 days. A decent remote vulnerability scanning software should have picked this up. Tenable's Nessus which is one of the industry standard vulnerability scanners tests for CVE-2017-5638 which is the vulnerability that effected Equifax. Nessus started testing for it on March 14th.

3) Web Application Firewall: Web Application Firewalls will block known attacks before they hit the application. A decent WAF should block known vulnerabilities such as the one that hit Equifax as long as it was up to date. That said a lot of companies I have worked with tend to run WAFs in intrusion detection mode instead of intrusion prevention mode due to false positives and not wanting to block legitimate traffic. Some companies I have worked with are much better than others at going through the alarms, how quickly they respond to alarms, and filtering out the false positives so that the alarms are easier to manage. Usually for Web applications you will have a WAF rather than a general purpose IDS/IPS as the WAF will have access to the unencrypted traffic although there are ways to have IDS/IPS products have access to the Web server private certificates to decrypt the traffic.

Re:I smell bullshit.

[Hylandr]

After reading this it occurs to me that it's much more likely someone sold the info rather than had it hacked.

Human Error???

[Moblaster]

Anyone who has worked with sensitive processes (esp computer security processes) knows that relying on one person for a mission-critical function is not a "human error" - it's a process failure. If this person's communication job was that essential, they should have had a team-based process in place with multiple individuals charged with making sure the process got executed, backed up by computerized records and nag alerts if not done. Seems like this "human error" would have happened if the person had gone on vacation, gotten fired, or went off their meds. That's not a human error. That's execs failing to make sure they build a resilient security process. Quarter billion in expenditure won't buy common sense, it seems.

Re: Human Error???

[Mr D from 63]

There's a thing called independent verification that might have helped. Guess its that one guys fault that they didn't practice that.

Re:Human Error???

[msauve]

"Anyone who has worked with sensitive processes (esp computer security processes) knows that relying on one person for a mission-critical function is not a "human error" - it's a process failure."

Absolutely. Human redundancy is just as important as network/system redundancy. If the organization isn't set up to continue working even if someone gets hit by a bus, that's a management failure. It's not a single individual. Who was responsible for checking that the work was done as required?

Ah yes, the blame game

[quonset]

"It was his fault. That's why I sold my company stock when I found out about the breach rather than inform anyone except the other folks in the executive suite."

Wow, that's scummy

[JohnFen]

"The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not,"

What a scummy thing to say, and he doesn't even realize that the statement makes Equifax look even worse.

With a couple of hundred people on the security team, the idea that it's a single person's responsibility to tell everyone to apply a patch is ludicrous. If it's true, then that's institutional incompetence.

I've been working in computer security for years, and do you know what I and all of my coworkers do? We keep up on computer security developments, particularly newly discovered vulnerabilities. And we discuss them. And send emails about them.

Even if the one team (not individual) who is responsible for ensuring that our own systems are patched for some reason fails to do that job, there is exactly zero chance that this would go unnoticed.

If that's not how it works at Equifax, that's the fault of Equifax, not some single individual.

huh?

[kefalonia]

bollocks. Yes, that.

Any security organization which relies on a single individual's action or inaction to remain in good standing is simply fairytale.
Every good process which involves a human in the loop, should always ensure that at least one more is present to enforce check-and-balance objectives.
There is a good reason why all commercial flights have two pilots as a default.

Let me state this: when you see management pointing one single downstream individual for such an event, there are at least TWO levels of management at fault.

Such BS

[gordona]

The buck stops with the CEO! If the CEO knew about vulnerability that needed patching, he should have been expecting a report regarding the application of the patch. If he didn't get that he should have come down on the admin or system owner for not installing it. Unless of course that wasn't in the security policy in which case it still falls on the back of the CEO. DUE CARE and DUE DILIGENCE! Non existent.

$225 million isn't much

[phalse phace]

The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team.

Spending $225 million over 3 years isn't really that much when you consider the type and amount of personal data Equifax has on us.

JP Morgan Chase spent $500 million in 2016 alone, Bank of America spent $400 million on cyber security in 2016 although they have an unlimited cyber security budget, Citibank's cyber security budget topped $400 million and Wells Fargo spends roughly $250 million per year.

Failure of way more than one person

[Todd Knarr]

Failing to apply the patch would be the failure of that one person to order the patch applied, plus the failure of his superior to notice that an action item hadn't been handled, plus a failure of the security team to notice that a ticket hadn't been completed, plus the failure of the head of the security team to notice his subordinates had uncompleted tickets sitting there. All this stuff should be tracked, and where I work it is and we have daily status meetings where stuff like this gets asked about, and development team managers and product managers have weekly status meetings where lack of progress on tickets and what needs done about it is a standard agenda item.

Accountability means managers and executives are just as accountable for work getting done or not getting done as low-level employees are expected to be.

Struts being an application framework...

[hattig]

Struts is an application framework, which means it is an application dependency. That means that every Struts-using application within Equifax would have needed to be upgraded, to be tested at least on the new version. That is the job of more than one person!

It is possible that Equifax's application servers (Tomcat, JBoss, etc) were configured with Struts being provided at the container level, but even that would be a full upgrade of multiple application servers within the company - a platforms team responsibility. However I suspect Struts would have been incorporated into the application itself at build time (as a dependency library).

I do not know how many applications Equifax's systems are made up of, but certainly the company I work for has dozens or hundreds to build up a trading platform (or two or three!). I imagine it is similar at Equifax.

I also cannot imagine a security team of 225 people having just one person be responsible to notifying and reminding of critical library vulnerabilities and updates for the entire business.

This smells of "VW Single Rogue Engineer" to me. Clearly bullshit.

So what you're saying is

[rsilvergun]

Your entire operation is one under paid and overworked sys admin away from disaster? Did I get that right?

Miserable little fuck

[satan666]

What a miserable, no good, lying, sniveling, double crossing, douchebag, fuckface, fucktard, dickwad lying little bitch.
From his resignation letter:
"I'm outta here suckers! Let me throw a few of you worms under the bus on my way out. Not my fault. Fuck you and goodnight."
Love, dickwad in charge, Ret.
P.S. Bitch better have my moneyyyy!

It's even worse

[PatientZero]

Any number of reasonable things could have caused the patch to be missed, but you'd expect $250M spent over three years to provide a few more security processes beyond, "Fred forgot to apply the patch." The attackers were spreading through their systems over several months without detection.

Also, way to lead from behind. Every corporate officer I've met has shared one tenet with all others: they are responsible for everything that their team does, good and bad. If some employee several rungs down the corporate ladder fails, it's because the leadership above them failed to hire or train them correctly or put in the right processes.