Slashdot Mirror
Former Equifax CEO Blames Breach On One Individual Who Failed To Deploy Patch (techcrunch.com)
Equifax's recently departed CEO is blaming the largest data breach in history on a single person who failed to deploy a patch. TechCrunch reports: Hackers exposed the Social Security numbers, drivers licenses and other sensitive info of 143 million Americans earlier this summer by exploiting a vulnerability in Apache's Struts software, according to testimony heard today from former CEO Richard Smith. However, a patch for that vulnerability had been available for months before the breach occurred. Now several top Equifax execs are being taken to task for failing to protect the information of millions of U.S. citizens. In a live stream before the Digital Commerce and Consumer Protection subcommittee of the House Energy and Commerce committee, Smith testified the Struts vulnerability had been discussed when it was first announced by CERT on March 8th.
Smith said when he started with Equifax 12 years ago there was no one in cybersecurity. The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team. However, Smith had an interesting explainer for how this easy fix slipped by 225 people's notice -- one person didn't do their job. "The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not," Smith, who did not name this individual, told the committee.
Smith said when he started with Equifax 12 years ago there was no one in cybersecurity. The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team. However, Smith had an interesting explainer for how this easy fix slipped by 225 people's notice -- one person didn't do their job. "The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not," Smith, who did not name this individual, told the committee.
He's Spartacus!
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
Sucks that you don't do configuration management.
love is just extroverted narcissism
If .25Bn has been invested then there's sure as hell no process that could have allowed a single critical patch go unchecked as described. There's teams, or should be teams of people watching these things.
I smell a really shitty cop-out excuse.
~ People that think they are better than anyone else for any reason are the cause of all the strife in the world.
To quote Thomas Jefferson, "The Tree of Bare Fucking Minimum Standards of Responsibility and Decorum must be refreshed from time to time with blood."
"They were pure niggers." – Noam Chomsky
Anyone who has worked with sensitive processes (esp computer security processes) knows that relying on one person for a mission-critical function is not a "human error" - it's a process failure. If this person's communication job was that essential, they should have had a team-based process in place with multiple individuals charged with making sure the process got executed, backed up by computerized records and nag alerts if not done. Seems like this "human error" would have happened if the person had gone on vacation, gotten fired, or went off their meds. That's not a human error. That's execs failing to make sure they build a resilient security process. Quarter billion in expenditure won't buy common sense, it seems.
Buggy whips are gone, but the need for horsehoe nails remains.
Prove anything by multiplying Huge Number times Tiny Number
"It was his fault. That's why I sold my company stock when I found out about the breach rather than inform anyone except the other folks in the executive suite."
"The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not,"
What a scummy thing to say, and he doesn't even realize that the statement makes Equifax look even worse.
With a couple of hundred people on the security team, the idea that it's a single person's responsibility to tell everyone to apply a patch is ludicrous. If it's true, then that's institutional incompetence.
I've been working in computer security for years, and do you know what I and all of my coworkers do? We keep up on computer security developments, particularly newly discovered vulnerabilities. And we discuss them. And send emails about them.
Even if the one team (not individual) who is responsible for ensuring that our own systems are patched for some reason fails to do that job, there is exactly zero chance that this would go unnoticed.
If that's not how it works at Equifax, that's the fault of Equifax, not some single individual.
bollocks. Yes, that.
Any security organization which relies on a single individual's action or inaction to remain in good standing is simply fairytale.
Every good process which involves a human in the loop, should always ensure that at least one more is present to enforce check-and-balance objectives.
There is a good reason why all commercial flights have two pilots as a default.
Let me state this: when you see management pointing one single downstream individual for such an event, there are at least TWO levels of management at fault.
The buck stops with the CEO! If the CEO knew about vulnerability that needed patching, he should have been expecting a report regarding the application of the patch. If he didn't get that he should have come down on the admin or system owner for not installing it. Unless of course that wasn't in the security policy in which case it still falls on the back of the CEO. DUE CARE and DUE DILIGENCE! Non existent.
"Gentlemen, you can't fight in here! This is the War Room!" -- Dr. Strangelove
The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team.
Spending $225 million over 3 years isn't really that much when you consider the type and amount of personal data Equifax has on us.
JP Morgan Chase spent $500 million in 2016 alone, Bank of America spent $400 million on cyber security in 2016 although they have an unlimited cyber security budget, Citibank's cyber security budget topped $400 million and Wells Fargo spends roughly $250 million per year.
Failing to apply the patch would be the failure of that one person to order the patch applied, plus the failure of his superior to notice that an action item hadn't been handled, plus a failure of the security team to notice that a ticket hadn't been completed, plus the failure of the head of the security team to notice his subordinates had uncompleted tickets sitting there. All this stuff should be tracked, and where I work it is and we have daily status meetings where stuff like this gets asked about, and development team managers and product managers have weekly status meetings where lack of progress on tickets and what needs done about it is a standard agenda item.
Accountability means managers and executives are just as accountable for work getting done or not getting done as low-level employees are expected to be.
Struts is an application framework, which means it is an application dependency. That means that every Struts-using application within Equifax would have needed to be upgraded, to be tested at least on the new version. That is the job of more than one person!
It is possible that Equifax's application servers (Tomcat, JBoss, etc) were configured with Struts being provided at the container level, but even that would be a full upgrade of multiple application servers within the company - a platforms team responsibility. However I suspect Struts would have been incorporated into the application itself at build time (as a dependency library).
I do not know how many applications Equifax's systems are made up of, but certainly the company I work for has dozens or hundreds to build up a trading platform (or two or three!). I imagine it is similar at Equifax.
I also cannot imagine a security team of 225 people having just one person be responsible to notifying and reminding of critical library vulnerabilities and updates for the entire business.
This smells of "VW Single Rogue Engineer" to me. Clearly bullshit.
Forgot to turn on auto-update for the Flash player? PLAY AND YOU PAY!
who was too busy posting on /.
Sign on the desk of CxO's everywhere
(contrast this with the US Navy, where the captain of the Fitzgerald was relieved, even though he was not on deck when the collision occurred and in fact was almost killed by the accident. Subsequently, the Navy relieved several higher ranking officers, including Flag officers, for supervisory failures.)
Your entire operation is one under paid and overworked sys admin away from disaster? Did I get that right?
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
Somebody in Management decided to hire a totally incompetent and unqualified CSO. Nice omission there Mr. BS CEO.
Reminds me of the time 'a couple of rogue engineers for the whole VW emissions fiasco. I think handsome bonuses are in the works due to management for uncovering this subterfuge.
So 12 years ago, 2005, Equifax had no redundancy for patching software? Equifax, one of only a handful of companies responsible for massive amounts of private consumer data, following years of headlines of corporate hacks and leaks, while Microsoft themselves were facing declines specifically because of viruses, had one guy. A company which has server farms and millions of transactions daily, connected to every major multinational bank and government had one guy. By his own admission one guy on a 220 man security team, what did the others do exactly? What if that one guy decided to do the hack himself? Did he not have a supervisor? Is everyone at the company illiterate? It is obviously a lie. The arrogance of trying to make that claim in public deserves prosecution on its own.
The Ex-CEO, talking about the guys who cashed in their stock, said (from TFA):
I’ve know these individual for up to 12 years. They’re men of integrity.
First, his comments about the "one individual" demonstrates that he himself isn't a man of integrity, so his vouching for them means nothing.
Second, "men of integrity"? Hahahahahahaha!
It is the CIO's responsibility to see that systems are put in place to insure that the responsibility does not rest on one person, and that the company's systems cannot fail without multiple extreme and uncontrollable events occurring. They create the organization that will see that things happen properly even if individuals drop the ball. The technical buck stops at the CIO.
.25B has been spent and one person can fail with these sort of results, the shareholders need to sue them for failure to discharge their duties, and possible fraud.
The CEO is responsible for hiring a CIO that do their fucking job properly. Moreover, if
Nobody owns a fuck up of this scale but the CEO and CIO. These assholes need to be sent to an extra-rapey prison.
HA! I just wasted some of your bandwidth with a frivolous sig!
When I was in grad school one of my professors talked about his. Many weak leaders, when faced by a crisis, will respond with a form of "A small man must die," instead of taking responsibility for the weakness in leadership and design that allowed the crisis to evolve in the first place.
Expecting the CEO to know _anything_ about what goes on in the IT department is expecting too much. Executives have no clue what's going on outside of the boardroom, and the only time they ever get any sort of information is from management consultants or the odd 'red alert' that bubbles up to the CFO/CIO/COO/CSO. There is absolutely zero chance that the CEO of Equifax has any idea what patch level of Apache Struts is running on their Internet-facing services.
I wonder if he just went to the CIO and said, "give me a name, anyone remotely responsible for patching, so I can say I fired someone over this." I've never had it happen to me, but I have worked with people who were scapegoats in a major incident. Sucks when you're the one holding the bag...
blame one person for no security. Company with that data should assume their webserver will get hacked and act accordingly by implementing multiple layers of security. Web server should have been in DMZ with limited view to data (and no access to sensitive data). That is 101 security. $225m/3y where did that go? To an audit that showed nothing?
What a miserable, no good, lying, sniveling, double crossing, douchebag, fuckface, fucktard, dickwad lying little bitch.
From his resignation letter:
"I'm outta here suckers! Let me throw a few of you worms under the bus on my way out. Not my fault. Fuck you and goodnight."
Love, dickwad in charge, Ret.
P.S. Bitch better have my moneyyyy!
"God, What An Ass-Hole!"
Who is then the person who checks that person's work? And the person who is in charge of creating procedures and checks to detect quickly if one person didn't do the job? Or the various people in charge to check on password security? Those who monitor data streams and stop any data dump to a destination not on the approved list? That was a collective failure. Plenty of people just didn't give a damn.
And the scapegoat is named.
- Tjp
I am in wallow with my inner money grubbing capitalistic pig. ... Oink!
How could a single person be responsible for this, with nobody assigned to verify? No redundancy or assistance whatsoever? For something so important?
They need to find out who is responsible for setting things up so stupidly.
Twinstiq, game news
IT people are not well liked. Maybe it's because lots of us are nerds. Maybe it's because the only time people interact with us is when something is broken. But either way, we're a perfect scapegoat in any company. Always have been too.
Regular people don't like us. They never have. When computers made it so they had to depend on us that didn't make them like us. It lead to resentment and deepened their hate.
Mark my words, this'll work like a charm.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
In my security enclave, I automatically run patches on test systems as soon as they are released, I don't even have to do anything and monitors would let me know as soon as a critical event occurs.
And then all I have to do is move the patches from the testing channel to production and they get deployed, but even that is something that could be scripted or automated if the testing doesn't fail.
I literally spend less than 1% of my time on patching systems anymore and I manage almost 200 of them by myself.
Custom electronics and digital signage for your business: www.evcircuits.com
This wasn't a single person failure. If it was, that means the policy that setup that single person to be able to fail a mission critical issue is at fault. Also at fault is the actual PROTECTION OF DATA! How in 2016 and 2017 does ANY COMPANY have UNENCRYPTED PERSONAL INFORMATION on ANY COMPUTER/DATABASE which is attached to the INTERNET?!??! And this is in a company that is touting that it has spent billions on cybersecurity. Sure you may have spent the money on cybersecurity, but you certainly didn't take their advise and spend the money to change your processes which relied on using unencrypted data!!!!
Any number of reasonable things could have caused the patch to be missed, but you'd expect $250M spent over three years to provide a few more security processes beyond, "Fred forgot to apply the patch." The attackers were spreading through their systems over several months without detection.
Also, way to lead from behind. Every corporate officer I've met has shared one tenet with all others: they are responsible for everything that their team does, good and bad. If some employee several rungs down the corporate ladder fails, it's because the leadership above them failed to hire or train them correctly or put in the right processes.
Freedom to fear. Freedom from thought. Freedom to kill.
I guess the War on Terror really is about freedom!
If you work in engineering, you need to see the writing on the wall. No longer are you going to be indemnified for mistakes you make at work, even if you are forced to make them by bad management policy or lack of basic resources. No longer will the penalty for grievous error be a simple firing.
Face the music. If you make a mistake that causes what ends up being a tortious harm, you are going to jail.
Who tolerated an environment where there was no concern for the security of the data they collected on all of us.
That's the one person who is responsible. Not the scapegoat he is pointing at.
Any good stuff that happens I did, give me a big bonus. Any bad stuff that happens, blame. The old saying *hit rolls downhill has never been truer.
And that is a leadership failure. If you do this right, for each critical role, there is one person that does the change, one that verifies it has been done and and at least one that can take either role if one of the others is sick or on vacation.
Anyways, in the end it is _always_ the CEO that is responsible. This person is a coward and unfit for a leadership position, i.e. typical large-company CEO material in this sad world we live in.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
So you're upset that, contrary to the news reports and comments here on Slashdot, the CEO explicitly did take responsibility in front of a Congressional committee hearing?
Hi, where do I sign up to sitting in front of a Congressional committee and embarrassing myself for $90m?
That's the sort of responsibility I can handle.
Sigh. None of those 225 people had responsibility for applying the critical patch. I haven't looked into the Struts issue in detail but it's possible development work would be needed to implement the updated version.
Even without that, there's a full deploy/test/release cycle required, so you have a full dev/test team, a release team and sysadmin support. That's not the security team.
The security team will have at least a dozen people doing nothing other than user privilege management. Who can access which systems.
Others will be providing security guidance to projects and development teams. Others will be providing security governance and assessment of those change activities. There'll be a couple of teams monitoring regular and ad hoc activities on servers, on the network, at the interfaces between Equifax and other entities.
Then you've got the various security policy, management, research, the teams that do internal investigations, customer engagement and supplier management. There'll also be a group of people monitoring external influences - such as vulnerabilities in technologies used at the company.
That team will provide information, insight and guidance, but they're still not fucking responsible for patching anything. Let alone something like Struts.
Maybe you should read more than the summary.
If the security of all that data relies on one patch being applied, then that is yet another colossal failure by Equifax. For something with this sort of impact, there should be multiple layers of safeguards not just patching a web server. There were a long line of failures here, not just a missing patch.
This posting is provided 'AS IS' without warranty of any kind, implied or otherwise.
In Argentina they used Admin/Admin as user/password. Must be the same person.
Don't fight for your country, if your country does not fight for you.
Yeah I call BS, not as if we've never seen this kind of scapegoating but it's still annoying....
End of Line.
From a security point of view, this is 100% accurate. Also, nobody should be using Spring for security-sensitive applications for similar reasons.
Yeah, and I'd bet dollars to donuts that "one person" didn't apply the patch for a reason, something like
a "I'm patching today"
b "Oh, hold off we're in the middle of an important deployment"
a "But this is the scheduled day"
b "Just hold until next week, this is an expensive product and we can't delay launch"
a "OK, I'll hold off until Monday at the latest"
[Monday rolls around]
a "OK, I'm patching now"
b "Go ahead, but DON'T apply that Tomcat patch, it breaks stuff in the new application"
a "It's a critical patch! We'll have a massive vulnerability in our system if we don't address it!"
b "You can't. You'll break the system. You'll have to wait until the new version of the code is released that works around the issue. Don't worry it's next week"
a "..."
[weeks go by, code is still not updated, patch is still not applied, system gets hacked]
It is true that it comes down to one person to deploy the patch. But somebody somewhere else in the process should be reviewing the list of unpatched servers and asking "Hey - what's up?! how come this list of servers still isn't patched?"
Hard to believe that the have a flow down org and hope the bottom feeders are doing their jobs....without any oversight.
No no - somebody higher up isn't doing their job either !!!
You're the (former) CEO. It's your fault, period. You failed to manage effectively, period. Blame is irrelevant, you have been fired. Man up.
'Hey, we need to apply a critical security patch. Can we take down the servers?' "How much downtime?' 'Just about 30 min' 'No, too long. Can you do it on the weekend?' 'Sure it will take just 30min' 'NO, too long! I might need to look at an email from my car broker.'
If there was any doubt about Equifax's respect for adequate security instead of good enough: How do you blame the breach on one person when their should be at least one to verify that the patch was done. More than asking whether the job was done, I mean. So one can't blame it on one person but two at the least.
You are obviously still living in 1992, when encryption was considered a Magic Bullet.
I believe in 1992, encryption was considered literal bullets, weapons-grade munitions not fit for export.
What this CEO fails to understand is that the companies rise or failure depends on them. The reason they get paid millions is to reap the public success or fall on their sword in failure. This is letting the bullshit run downhill until it kills someone. Hereâ(TM)s what the jackoff should have said: âoeOne of my duties as CEO is to ensure public confidence in the company and I failed by not impementing the type of processes and internal culture that ensures that. For this, I deeply apologize to the American public.â But because he named an IT position, there will now be a witch hunt.
And what, pray tell, is YOUR job, Mr Smith?
Did YOU follow up on verifying the task was complete?
Or, did another staffer down the chain of command lie to you that it was patched?!
Self-importance and self-indulgence is the root of ALL evil.
"If you cannot keep up with this reality, quit the IT sphere !"
Easy to say. Hell, those patches will often be over 90 days old when rolled out by OS vendor and they only have that one job.