Slashdot Mirror
Former Equifax CEO Blames Breach On One Individual Who Failed To Deploy Patch (techcrunch.com)
Equifax's recently departed CEO is blaming the largest data breach in history on a single person who failed to deploy a patch. TechCrunch reports: Hackers exposed the Social Security numbers, drivers licenses and other sensitive info of 143 million Americans earlier this summer by exploiting a vulnerability in Apache's Struts software, according to testimony heard today from former CEO Richard Smith. However, a patch for that vulnerability had been available for months before the breach occurred. Now several top Equifax execs are being taken to task for failing to protect the information of millions of U.S. citizens. In a live stream before the Digital Commerce and Consumer Protection subcommittee of the House Energy and Commerce committee, Smith testified the Struts vulnerability had been discussed when it was first announced by CERT on March 8th.
Smith said when he started with Equifax 12 years ago there was no one in cybersecurity. The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team. However, Smith had an interesting explainer for how this easy fix slipped by 225 people's notice -- one person didn't do their job. "The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not," Smith, who did not name this individual, told the committee.
Smith said when he started with Equifax 12 years ago there was no one in cybersecurity. The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team. However, Smith had an interesting explainer for how this easy fix slipped by 225 people's notice -- one person didn't do their job. "The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not," Smith, who did not name this individual, told the committee.
He's Spartacus!
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
Sucks that you don't do configuration management.
love is just extroverted narcissism
If .25Bn has been invested then there's sure as hell no process that could have allowed a single critical patch go unchecked as described. There's teams, or should be teams of people watching these things.
I smell a really shitty cop-out excuse.
~ People that think they are better than anyone else for any reason are the cause of all the strife in the world.
Lack of personal responsibility.
To quote Thomas Jefferson, "The Tree of Bare Fucking Minimum Standards of Responsibility and Decorum must be refreshed from time to time with blood."
"They were pure niggers." – Noam Chomsky
Anyone who has worked with sensitive processes (esp computer security processes) knows that relying on one person for a mission-critical function is not a "human error" - it's a process failure. If this person's communication job was that essential, they should have had a team-based process in place with multiple individuals charged with making sure the process got executed, backed up by computerized records and nag alerts if not done. Seems like this "human error" would have happened if the person had gone on vacation, gotten fired, or went off their meds. That's not a human error. That's execs failing to make sure they build a resilient security process. Quarter billion in expenditure won't buy common sense, it seems.
Buggy whips are gone, but the need for horsehoe nails remains.
Prove anything by multiplying Huge Number times Tiny Number
"It was his fault. That's why I sold my company stock when I found out about the breach rather than inform anyone except the other folks in the executive suite."
First, one guy saying that they can't do anything because the hackers will get them anyway, now they are pinning the blame on someone manually applying fixes.
Lets be real here. An IT shop that big does not have one single person. They have to have at least 3-4, because of separation of duties. This was likely a fuck-up from management, and IT is being thrown under the bus, which is typical.
"The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not,"
What a scummy thing to say, and he doesn't even realize that the statement makes Equifax look even worse.
With a couple of hundred people on the security team, the idea that it's a single person's responsibility to tell everyone to apply a patch is ludicrous. If it's true, then that's institutional incompetence.
I've been working in computer security for years, and do you know what I and all of my coworkers do? We keep up on computer security developments, particularly newly discovered vulnerabilities. And we discuss them. And send emails about them.
Even if the one team (not individual) who is responsible for ensuring that our own systems are patched for some reason fails to do that job, there is exactly zero chance that this would go unnoticed.
If that's not how it works at Equifax, that's the fault of Equifax, not some single individual.
Richard Smith. He's the head honcho there; the buck stops with him.
bollocks. Yes, that.
Any security organization which relies on a single individual's action or inaction to remain in good standing is simply fairytale.
Every good process which involves a human in the loop, should always ensure that at least one more is present to enforce check-and-balance objectives.
There is a good reason why all commercial flights have two pilots as a default.
Let me state this: when you see management pointing one single downstream individual for such an event, there are at least TWO levels of management at fault.
The buck stops with the CEO! If the CEO knew about vulnerability that needed patching, he should have been expecting a report regarding the application of the patch. If he didn't get that he should have come down on the admin or system owner for not installing it. Unless of course that wasn't in the security policy in which case it still falls on the back of the CEO. DUE CARE and DUE DILIGENCE! Non existent.
"Gentlemen, you can't fight in here! This is the War Room!" -- Dr. Strangelove
The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team.
Spending $225 million over 3 years isn't really that much when you consider the type and amount of personal data Equifax has on us.
JP Morgan Chase spent $500 million in 2016 alone, Bank of America spent $400 million on cyber security in 2016 although they have an unlimited cyber security budget, Citibank's cyber security budget topped $400 million and Wells Fargo spends roughly $250 million per year.
Failing to apply the patch would be the failure of that one person to order the patch applied, plus the failure of his superior to notice that an action item hadn't been handled, plus a failure of the security team to notice that a ticket hadn't been completed, plus the failure of the head of the security team to notice his subordinates had uncompleted tickets sitting there. All this stuff should be tracked, and where I work it is and we have daily status meetings where stuff like this gets asked about, and development team managers and product managers have weekly status meetings where lack of progress on tickets and what needs done about it is a standard agenda item.
Accountability means managers and executives are just as accountable for work getting done or not getting done as low-level employees are expected to be.
Struts is an application framework, which means it is an application dependency. That means that every Struts-using application within Equifax would have needed to be upgraded, to be tested at least on the new version. That is the job of more than one person!
It is possible that Equifax's application servers (Tomcat, JBoss, etc) were configured with Struts being provided at the container level, but even that would be a full upgrade of multiple application servers within the company - a platforms team responsibility. However I suspect Struts would have been incorporated into the application itself at build time (as a dependency library).
I do not know how many applications Equifax's systems are made up of, but certainly the company I work for has dozens or hundreds to build up a trading platform (or two or three!). I imagine it is similar at Equifax.
I also cannot imagine a security team of 225 people having just one person be responsible to notifying and reminding of critical library vulnerabilities and updates for the entire business.
This smells of "VW Single Rogue Engineer" to me. Clearly bullshit.
Forgot to turn on auto-update for the Flash player? PLAY AND YOU PAY!
who was too busy posting on /.
Sign on the desk of CxO's everywhere
(contrast this with the US Navy, where the captain of the Fitzgerald was relieved, even though he was not on deck when the collision occurred and in fact was almost killed by the accident. Subsequently, the Navy relieved several higher ranking officers, including Flag officers, for supervisory failures.)
Your entire operation is one under paid and overworked sys admin away from disaster? Did I get that right?
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
Failing to "deploy a patch" is an error of omission. It's wrong, but no where near "selling off stock before it the price tanks" based on that error. Blaming some peon down in the boiler room while the lazy CEO scumbag maneuvers himself into optimal escape route, is just more proof that these people should face serious charges.
Somebody in Management decided to hire a totally incompetent and unqualified CSO. Nice omission there Mr. BS CEO.
Reminds me of the time 'a couple of rogue engineers for the whole VW emissions fiasco. I think handsome bonuses are in the works due to management for uncovering this subterfuge.
a 'c' level exec almost took personal responsibility.
So 12 years ago, 2005, Equifax had no redundancy for patching software? Equifax, one of only a handful of companies responsible for massive amounts of private consumer data, following years of headlines of corporate hacks and leaks, while Microsoft themselves were facing declines specifically because of viruses, had one guy. A company which has server farms and millions of transactions daily, connected to every major multinational bank and government had one guy. By his own admission one guy on a 220 man security team, what did the others do exactly? What if that one guy decided to do the hack himself? Did he not have a supervisor? Is everyone at the company illiterate? It is obviously a lie. The arrogance of trying to make that claim in public deserves prosecution on its own.
The Ex-CEO, talking about the guys who cashed in their stock, said (from TFA):
I’ve know these individual for up to 12 years. They’re men of integrity.
First, his comments about the "one individual" demonstrates that he himself isn't a man of integrity, so his vouching for them means nothing.
Second, "men of integrity"? Hahahahahahaha!
It is the CIO's responsibility to see that systems are put in place to insure that the responsibility does not rest on one person, and that the company's systems cannot fail without multiple extreme and uncontrollable events occurring. They create the organization that will see that things happen properly even if individuals drop the ball. The technical buck stops at the CIO.
.25B has been spent and one person can fail with these sort of results, the shareholders need to sue them for failure to discharge their duties, and possible fraud.
The CEO is responsible for hiring a CIO that do their fucking job properly. Moreover, if
Nobody owns a fuck up of this scale but the CEO and CIO. These assholes need to be sent to an extra-rapey prison.
HA! I just wasted some of your bandwidth with a frivolous sig!
When I was in grad school one of my professors talked about his. Many weak leaders, when faced by a crisis, will respond with a form of "A small man must die," instead of taking responsibility for the weakness in leadership and design that allowed the crisis to evolve in the first place.
Expecting the CEO to know _anything_ about what goes on in the IT department is expecting too much. Executives have no clue what's going on outside of the boardroom, and the only time they ever get any sort of information is from management consultants or the odd 'red alert' that bubbles up to the CFO/CIO/COO/CSO. There is absolutely zero chance that the CEO of Equifax has any idea what patch level of Apache Struts is running on their Internet-facing services.
I wonder if he just went to the CIO and said, "give me a name, anyone remotely responsible for patching, so I can say I fired someone over this." I've never had it happen to me, but I have worked with people who were scapegoats in a major incident. Sucks when you're the one holding the bag...
blame one person for no security. Company with that data should assume their webserver will get hacked and act accordingly by implementing multiple layers of security. Web server should have been in DMZ with limited view to data (and no access to sensitive data). That is 101 security. $225m/3y where did that go? To an audit that showed nothing?
What a miserable, no good, lying, sniveling, double crossing, douchebag, fuckface, fucktard, dickwad lying little bitch.
From his resignation letter:
"I'm outta here suckers! Let me throw a few of you worms under the bus on my way out. Not my fault. Fuck you and goodnight."
Love, dickwad in charge, Ret.
P.S. Bitch better have my moneyyyy!
"God, What An Ass-Hole!"
Who is then the person who checks that person's work? And the person who is in charge of creating procedures and checks to detect quickly if one person didn't do the job? Or the various people in charge to check on password security? Those who monitor data streams and stop any data dump to a destination not on the approved list? That was a collective failure. Plenty of people just didn't give a damn.
And the scapegoat is named.
- Tjp
I am in wallow with my inner money grubbing capitalistic pig. ... Oink!
It must have been Stephen Pax.
What the "smartest guys" at Enron tried to do was blamed the guy who blew brains out as being solely responsible for the ensuing scandal that came about.
Source: "The Smartest Guys in the Room: The Amazing Rise and Scandalous Fall of Enron".
How could a single person be responsible for this, with nobody assigned to verify? No redundancy or assistance whatsoever? For something so important?
They need to find out who is responsible for setting things up so stupidly.
Twinstiq, game news
It is never one person. Wait till the lawsuits start. Depositions will tell a different story.
What a really small man to have said that.
From his preliminary statement before the House Energy and Commerce Committee:
Let me say clearly: As CEO I was ultimately responsible for what happened on my watch. Equifax was entrusted with Americans’ private data and we let them down. To each and every person affected by this breach, I am deeply sorry that this occurred.
The claim that the CEO threw a single employee under the bus is patently false and is fake news if anyone bothers to read his testimony. Smith laid out a timeline where several failures by the company was explained. The failure of one person of applying the available patch and reporting it is entirely possible. It doesn't take a whole department of server admins to apply a patch across a server farm. Usually its just 1 senior server admin to supervise his junior admins to do it. And all it takes is for one guy to fat finger an error to a backbone router config and the whole enterprise is down.
IT people are not well liked. Maybe it's because lots of us are nerds. Maybe it's because the only time people interact with us is when something is broken. But either way, we're a perfect scapegoat in any company. Always have been too.
Regular people don't like us. They never have. When computers made it so they had to depend on us that didn't make them like us. It lead to resentment and deepened their hate.
Mark my words, this'll work like a charm.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
In my security enclave, I automatically run patches on test systems as soon as they are released, I don't even have to do anything and monitors would let me know as soon as a critical event occurs.
And then all I have to do is move the patches from the testing channel to production and they get deployed, but even that is something that could be scripted or automated if the testing doesn't fail.
I literally spend less than 1% of my time on patching systems anymore and I manage almost 200 of them by myself.
Custom electronics and digital signage for your business: www.evcircuits.com
" "The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not," Smith, who did not name this individual, told the committee."
So basically he's saying it was Anonymous that did it.
What a weak dick
This wasn't a single person failure. If it was, that means the policy that setup that single person to be able to fail a mission critical issue is at fault. Also at fault is the actual PROTECTION OF DATA! How in 2016 and 2017 does ANY COMPANY have UNENCRYPTED PERSONAL INFORMATION on ANY COMPUTER/DATABASE which is attached to the INTERNET?!??! And this is in a company that is touting that it has spent billions on cybersecurity. Sure you may have spent the money on cybersecurity, but you certainly didn't take their advise and spend the money to change your processes which relied on using unencrypted data!!!!
Any number of reasonable things could have caused the patch to be missed, but you'd expect $250M spent over three years to provide a few more security processes beyond, "Fred forgot to apply the patch." The attackers were spreading through their systems over several months without detection.
Also, way to lead from behind. Every corporate officer I've met has shared one tenet with all others: they are responsible for everything that their team does, good and bad. If some employee several rungs down the corporate ladder fails, it's because the leadership above them failed to hire or train them correctly or put in the right processes.
Freedom to fear. Freedom from thought. Freedom to kill.
I guess the War on Terror really is about freedom!
Yeah. One guy. If it came down to one guy, fire the CEO and the CTO and the CSO. Three people to blame that it came down to one guy.
If you work in engineering, you need to see the writing on the wall. No longer are you going to be indemnified for mistakes you make at work, even if you are forced to make them by bad management policy or lack of basic resources. No longer will the penalty for grievous error be a simple firing.
Face the music. If you make a mistake that causes what ends up being a tortious harm, you are going to jail.
In other news, the main Equifax office has gone completely dark. The CEO explained that while they have a large team team for building maintenance, the person responsible for telling them that the lightbulbs need to be replaced didn't do their job.
Of course, the fall on your sword analogy is synonymous with every CEO when there really isn't a sword. If the CEO were to ultimately face prison or name names...they would of course name someone else. This whole "my responsibility" crap really irks me when there really isn't a punishment. Let's see if they sing the same tune if they owed every person involved in the breach $2...show me a CEO that would step to that plate.
Hee hee forgot to patch apache! It's just a patchy server!
Who tolerated an environment where there was no concern for the security of the data they collected on all of us.
That's the one person who is responsible. Not the scapegoat he is pointing at.
So you're upset that, contrary to the news reports and comments here on Slashdot, the CEO explicitly did take responsibility in front of a Congressional committee hearing? Huh.
>This whole "my responsibility" crap really irks me when there really isn't a punishment.
Why would he get punished beyond getting fired for how dismally Equifax handled the aftermath of the breach? He wasn't with the hackers who broke into the system and thus committed the crime.
Any good stuff that happens I did, give me a big bonus. Any bad stuff that happens, blame. The old saying *hit rolls downhill has never been truer.
Yeah, the CEO.
And that is a leadership failure. If you do this right, for each critical role, there is one person that does the change, one that verifies it has been done and and at least one that can take either role if one of the others is sick or on vacation.
Anyways, in the end it is _always_ the CEO that is responsible. This person is a coward and unfit for a leadership position, i.e. typical large-company CEO material in this sad world we live in.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I deal with root cause analysis frequently and do not believe human error is a root cause. It can be a contributing factor, but something left the system vulnerable to a mistake. When I hear someone say human error I see an attempt to fix the blame and not the problem.
If I don't patch a server, in less than 30 days the following people know:
* about 20 coworkers
* my branch head
* his branch head
* the commanding officer
* information assurance branch head
* IA deputies
He wasn't fired. He retired with his multi-million dollar golden parachute.
So you're upset that, contrary to the news reports and comments here on Slashdot, the CEO explicitly did take responsibility in front of a Congressional committee hearing?
Hi, where do I sign up to sitting in front of a Congressional committee and embarrassing myself for $90m?
That's the sort of responsibility I can handle.
You are obviously still living in 1992, when encryption was considered a Magic Bullet.
Here is the protip: it is NOT.
If a corporation wants to work with data, it must sooner or later be in PLAINTEXT. Maybe you can encrypt at the database level, but as soon as it is in a server and doing useful stuff, it must be decrypted. So the key for decryption is in the server. But what if the server itself is compromised due to a cyber weakness ? Then the attacker can either read the key or use procedures/methods on the server to obtain plaintext data.
To conclude: Encryption is an important security tool, but it is not a defence against cybernetic weaknesses of business systems.
In 2014 the first computer was hacked. It was the Fourteen-Worm, do you remember ?
If it takes you 90 days to patch, the attacker has at least 80 days to mess with your systems.
This is 2017 and the intertubes are full of international criminals who actively monitor the CVE database and who actively diff versions of popular software in order to find exploits.
Plus there are thousands of criminal software engineers actively searching for bugs in popular software.
If you cannot keep up with this reality, quit the IT sphere !
The CEO is looking for a Fall Guy, and he has one it his crosshairs.
Never mind the Chief Security Officer did not have a Computer Science degree.
Never mind the 224 other "security" people did not realize the missing patch.
Never mind that (probably) 80% of systems at Equifax are weeks behind the necessary patch levels. Never mind it just blew up with the Apache Struts issue. Now please check their Oracle databases, their MQSeries, their Solaris servers etc etc.
Anybody using struts should immediately be shoot in the head. The remote arbitrary code execution occurrence rate is far too high. Insecure by design (OGNL and other shits, I am looking at you!)
Richard Cheney and George W.Bush. And Richard Fuld.
But you know what ? Nothing of the like will happen. A corrupt elite protects each other.
SO a company as big a Equfax relies on ONE GUY to do patching. Even if one guy does it, there should have been corporate security processes to catch that the one guy didnt do his job. A few terms Equifax should look up.
Configuration management?
Auditing
Automation.
That is the operative word. Maybe the CEO is BSing everybody and they simply declared 80% of their IT budget as "security".
We have seen so much lying and deceiving at the elite level that this is entirely possible.
And finally: Just because you spend money does not mean it is spent effectively. The money spent for the "music degree" CSO was PROVEN wasted. Probably several millions a year just that CSO.
Seriously, this is just scapegoating.
If the security of all that data relies on one patch being applied, then that is yet another colossal failure by Equifax. For something with this sort of impact, there should be multiple layers of safeguards not just patching a web server. There were a long line of failures here, not just a missing patch.
This posting is provided 'AS IS' without warranty of any kind, implied or otherwise.
In Argentina they used Admin/Admin as user/password. Must be the same person.
Don't fight for your country, if your country does not fight for you.
Yeah I call BS, not as if we've never seen this kind of scapegoating but it's still annoying....
End of Line.
"The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not," Smith, who did not name this individual, told the committee."
So, he is blaming the CIO? After all, it is the CIO's job to maintain their security stance.
Then again, it is the CEO's job to shield the company from lawsuits by making sure the rest of the executive committee do THEIR jobs by providing sufficient funding and GUIDANCE to do them.
If there is a procedure in place that he did not follow then maybe he is responsible however in my experience, management just hope we do things properly in which case there is no process to follow, making management culpable.
Yeah, and I'd bet dollars to donuts that "one person" didn't apply the patch for a reason, something like
a "I'm patching today"
b "Oh, hold off we're in the middle of an important deployment"
a "But this is the scheduled day"
b "Just hold until next week, this is an expensive product and we can't delay launch"
a "OK, I'll hold off until Monday at the latest"
[Monday rolls around]
a "OK, I'm patching now"
b "Go ahead, but DON'T apply that Tomcat patch, it breaks stuff in the new application"
a "It's a critical patch! We'll have a massive vulnerability in our system if we don't address it!"
b "You can't. You'll break the system. You'll have to wait until the new version of the code is released that works around the issue. Don't worry it's next week"
a "..."
[weeks go by, code is still not updated, patch is still not applied, system gets hacked]
It is true that it comes down to one person to deploy the patch. But somebody somewhere else in the process should be reviewing the list of unpatched servers and asking "Hey - what's up?! how come this list of servers still isn't patched?"
Hard to believe that the have a flow down org and hope the bottom feeders are doing their jobs....without any oversight.
No no - somebody higher up isn't doing their job either !!!
told ya.
You're the (former) CEO. It's your fault, period. You failed to manage effectively, period. Blame is irrelevant, you have been fired. Man up.
'Hey, we need to apply a critical security patch. Can we take down the servers?' "How much downtime?' 'Just about 30 min' 'No, too long. Can you do it on the weekend?' 'Sure it will take just 30min' 'NO, too long! I might need to look at an email from my car broker.'
It is nigh on impossible to manually update hundreds or thousands of servers which run easily hundreds of often bespoke systems in a company like Equifax.
Also, it is very hard to track the set of discovered exploits against your inventory of used software packages/libraries.
Only if you have at least an automated way of listing the unpatched-exploitable systems it is hard to believe that you will get this done manually. Too many other priorities and "deliverables" which will push these efforts back.
Systems must be easily inventorized and patched, otherwise companies will lose the battle against the cyber crims and spies.
There are lots of amateurs in systems development - both as developers and as managers. They often think that "delivering functionality" trumps everything, including sound design, formal specification and son on.
The resulting systems are quite often easy to hack. That is because quick-an-dirty approaches are usually exploitable.
These people must be driven out of the IT business or the entire IT field will suffer from the fools. If you cant write a proper scanner and a parser, quit !
A corporation like Equifax better have highly competent IT systems experts or simply go under. Maybe that is what we currently see...
After all, their core business is about data communications, storage and retrieval. All automated, not manual.
Almost certainly highest VW leadership mandated the fraud. So did lots of other auto corporation leaders.
But they found convenient Fall Guys, very low down the hierarchy tree.
what you describe is a bureaucratic clusterfuck of "we cannot possibly be secure due to fat processes" instead of a sharp, effective, not-overworked team of experts.
...write a nice letterof the issue to the CTO, CSO and so on an keep a physical copy of that latter. For the FBI/police/court of law.
If there was any doubt about Equifax's respect for adequate security instead of good enough: How do you blame the breach on one person when their should be at least one to verify that the patch was done. More than asking whether the job was done, I mean. So one can't blame it on one person but two at the least.
What this CEO fails to understand is that the companies rise or failure depends on them. The reason they get paid millions is to reap the public success or fall on their sword in failure. This is letting the bullshit run downhill until it kills someone. Hereâ(TM)s what the jackoff should have said: âoeOne of my duties as CEO is to ensure public confidence in the company and I failed by not impementing the type of processes and internal culture that ensures that. For this, I deeply apologize to the American public.â But because he named an IT position, there will now be a witch hunt.
I guess he should get the 90M dollar severance package.
Doh ! I think we do... He was fired from his position of the CEO, right? What am I missing?
Absolutely ridiculous. How did this CEO get past the Board?
And what, pray tell, is YOUR job, Mr Smith?
Did YOU follow up on verifying the task was complete?
Or, did another staffer down the chain of command lie to you that it was patched?!
Self-importance and self-indulgence is the root of ALL evil.