Slashdot Mirror
Got Malware? Get a Hammer!
FuzzNugget writes "After the Economic Development Administration (EDA) was alerted by the DHS to a possible malware infection, they took extraordinary measures. Fearing a targeted attack by a nation-state, they shut down their entire IT operations, isolating their network from the outside world, disabling their email services and leaving their regional offices high and dry, unable to access the centrally-stored databases. A security contractor ultimately declared the systems largely clean, finding only six computers infected with untargeted, garden-variety malware and easily repaired by reimaging. But that wasn't enough for the EDA: taking gross incompetence to a whole new level, they proceeded to physically destroy $170,500 worth of equipment (PDF), including uninfected systems, printers, cameras, keyboards and mice. After the destruction was halted — only because they ran out of money to continue smashing up perfectly good hardware — they had racked up a total of $2.3 million in service costs, temporary infrastructure acquisitions and equipment destruction."
You mean I get to release my pent-up anger by destroying physical systems *and* get paid a boatload of money to do it? Where do I sign up?
... and yet I'm still furloughed on Friday...
It sounds like they were using this as an excuse to buy new equipment, so they destroyed extra equipment hoping that someone would allow them to chalk up the expense to the virus and thus give them shiny new stuff.
Sounds like a good start.
You know, to be sure?
Will that infect my lawnmower? I'd better destroy it then before it gets dangerous...
Nae king! Nae laird! Nae yurrupiean pressedent! We willna be fooled again!
EDA: did you guys just smash a bunch of computers with a hammer because of viruses?
DHS: Yes, but there havent been any terrorist attacks since we smashed everything with hammers. clearly the operation was a massive success.
EDA: I dont even.....
DHS: yep. Freedom isnt free.
Good people go to bed earlier.
Because, RTFA?
"The total cost to the taxpayer of this incident was $2.7 million: $823,000 went to the security contractor for its investigation and advice, $1,061,000 for the acquisition of temporary infrastructure (requisitioned from the Census Bureau), $4,300 to destroy $170,500 in IT equipment, and $688,000 paid to contractors to assist in development a long-term response. Full recovery took close to a year."
-- "So they told me that using the download page to download something was not something they anticipated." - Bill Gates
Any IT upgrade could be spun exactly like this story, if you wanted... "why did you get a new mouse with that new system, the old one was working perfectly fine and now it's going in the trash!"
Devil's advocate:
I've worked at private companies, for education institutions, in the public sector, and in the Federal government. None are perfect, none are completely horri-bad.
All places have had those people who I had zero clue what their function was, but they always had a nice office.
It is easy to pick on government, but go to almost any work environment, and you will find the same thing.
It's was the dreaded "PC LOAD LETTER" virus. Smashing is the only recourse.
Yup. Likely what happened here is that the million-dollar security contractors gave the advice to do this bug hunt in the first place, and then provided the temporary replacement infrastructure, and walked away from the whole fiasco with a tidy profit. The reason this happens is because the government isn't generally allowed to hire people to do work like this, because "private industry is better." Of course, this sort of private industry is just a mechanism for siphoning off tax dollars, and the people who believe that hiring government employees to do government work is wasteful are actually responsible for fiascos like this, which are depressingly common.
Even when the contractors aren't crooked, the cost of employing them instead of federal employees is typically several times higher. But "corporations good, government wasteful." If we keep repeating that long enough maybe it will come true.
No reason to believe it wasn't cleaned up.
If they truly believe that it was the work of a nation-state, there is every reason to think it isn't cleaned up. Stuxnet didn't even reside just in computers. It infected programmable logic controllers attached to centrifuges, and then could re-infect computers on the network after they've been cleaned. If you really believe that Russia, or China has really compromised their network, and you have information that's worth more than a million dollars to them, then you should assume that everything (printers, routers, video-conferencing equipment, everything with a jack, plus the bios of all your computers) may be infected.
People tend to view $170,000 as a lot of money. But it's not. Computers for office workers can easily run under $1000. Hourly labor to clean things may be $50 per hour when you include overhead and benefits. And you're not even sure you got rid of the infection. If you mostly run apps that are resident on hardened servers, use imaging to make it easy to deploy new PCs, and don't have a lot of high end hardware, it may make sense to just replace everything with clean hardware. Honestly, for departments where you do think that there's stuff that sophisticated attackers may want, it may make sense to occasionally do this kind of purge occasionally even if you don't know there's been an attack. Take a look at the Sony Playstation breach for an idea of what getting compromised can cost. It's a hell of a lot more than $170,000.
Yeah baby, it's a great way to stimulate the economy. We know whst gets done is less important than things get done, and money gets pushed from person to person.
Buying computers to destroy employs people, as does destroying them. Hell, what we should do is just increase taxes and hire the tens of millions of unemployed to dig ditches and then fill them back in over and over.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
And why the hell would there be $2.3 million in service costs to destroy $170,500 worth of equipment?
RTFS.
service costs, temporary infrastructure acquisitions and equipment destruction
Or, RTFA for the details:
The total cost to the taxpayer of this incident was $2.7 million: $823,000 went to the security contractor for its investigation and advice, $1,061,000 for the acquisition of temporary infrastructure (requisitioned from the Census Bureau), $4,300 to destroy $170,500 in IT equipment, and $688,000 paid to contractors to assist in development a long-term response. Full recovery took close to a year.
Still outrageously stupid, but I think $4,300 to destroy $170,500 is a reasonable cost. I think the other costs--the ones with 6 or 7 figures--are the ones you should focus on.
But really, isn't giving US companies #2.3 million what the Economic Development Administration is supposed to be doing anyways? Better than spending it on the salaries for these government employees.
Battlemaster--Game with friends in medival realms
Oh, don't get me wrong, I'm combining my love for Alien and my inexplicable whoring for 'funny' upmods(that don't even net me the 'karma' I don't care about), rather than phoning in a reliable 'insightful' rant about THem Gummunit Union Beurocrats! in part because it amuses me more, and in part because (especially if your hardware is old shit) a sledgehammer is probably the best approach if you actually think that a state-caliber attacker is on your ass(for larger jobs consider a shredder rather than a hammer).
In this specificcase, given that their analysis found only a small quantity of chickenshit malware, and because the EDA is kind of a low-priority target for the really cool attacks, I strongly suspect that it was an overrreaction(and, if it wasn't an overreaction, doing more aggressive analysis, in order to better understand the adversary's capabilities, in terms of OS, Application, and hardware/firmware level malware would have been more responsible than just shredding it all).
That said, though, you'd be hard pressed to be paranoid enough about the potential for even seemingly innocuous devices, in the hands of a capable attacker, to be malicious. The BIOS has had slightly unnerving powers ever since SMM; but these days it's a second OS, more or less, USB devices are highly likely to be full, potentially reprogrammable, devices that are just implementing whatever they are supposed to be in software(OEM cost-cutting reduces the risk that there would be space/power to hide anything really cool; but some pretty weedy microcontrollers can handle being whatever flavor of USB slave device they are set to emulate. Even monitors get a full i2c bus for DDC, no idea how well your graphics driver, occupying its position of relative privilege within the system, watches that interface...
I would say that they screwed up, because if they genuinely suspected the worst, shredding the evidence rather than analyzing it is unhelpful in preventing future attacks, and if they didn't suspect the worst, dumping clean images on the systems and getting on with life would have been a lot cheaper; but it is true that, if you suspect a genuinely capable attacker, you are sufficiently fucked that just burning it with fire is probably the cheapest option...
Not always true. I've seen many incompetent people continue to get promoted in industry. The government ones just get more press.
It seems to be a symptom of some underlying pathology in a democracy when so much effort is put into protecting the head of government. At least in the ideal it doesn't matter who is president;
You're completely missing the point of protecting the Head of State - it's not because an assassination would cause a change in policies, but to keep extremists from using threat of assassination to to blackmail a Head of State into changing those policies.
In other words, if the POTUS has to fear for his life as a result of every decision he makes, he is going to be pressured to cater to the most radical and violent groups.
$823,000 for the security contractor that adviced them to do that destruction?
Read the story, or at least read the summary.
The contractor did not tell them to do that. The contractor found exactly 6 machines, which they recommended by re-imaged.
This stupidity was not the contractors fault.
Sig Battery depleted. Reverting to safe mode.
The actual destruction costs were only: $4,300 (still too much). The rest of that price tag is the total cost of doing the destruction - temporary infrastructure and so on. Not sure why a temporary replacement would cost 10x what was being replaced, though. Still plenty of government waste in the story.
Well except for the mice. You know how mice breed. Destroying those infected mice can take forever, because you find them breeding in closets, junk drawers, sometimes in their original boxes if bought at a TwoFer sale. And the wireless ones can be found a long way away from their nest, under desks, leaving their dongles everywhere.
They were lucky they managed to nip the infestation in the bud. It could have gotten totally out of hand had they owned any traveling laptops with mice. Entire countries might need quarantine. One mouse on a plane, and its game over.
Sig Battery depleted. Reverting to safe mode.
best buddy system.
that's why.
That could be true; however, have you read the audit paper written by OIG in PDF (http://www.oig.doc.gov/OIGPublications/OIG-13-027-A.pdf)? It is very interesting and contains what the auditor (OIG) thinks where to blame (although those who are at fault simply brush the responsibility to others). Everything seems to be from miscommunication between DOC CIRT and EDA, and both did not know about this miscommunication until too late (the end of 2012, about a year after the incident).
What happened (from the audit paper) was that the incident handlers from DOC CIRT sent out 2 notifications to EDA regarding the US CERT notification. The first notification simply listed all 146 components, and EDA thought all of them were infected. Then the incident handlers from DOC CIRT sent the 2nd notification with accurate analysis of only 2 infected commponents, but the notification did not clarify or mention that the 1st notification was inaccurate (wrong). As a result EDA thought all 146 components were still infected.
Then the EDA selected and submitted 2 components to the DOC CIRT as a process to verify whether they were infected. Apparently, the EDA submitted the 2 components mentioned in the 2nd notification, and the result came back positive. As a result, the EDA thought that all 146 components were infected.
It got worse when EDA already knew that their IT system is outdated and needed a lot of updates/patches (since 2006 from NSA and OIG system reviewed) but they never fixed the issues. They believed this incident was an attack from nation-state actors (hackers), so their system could be extremely vulnerable to the attack. As a result, their system could open a hole to other systems' access. Therefore, the system was isolated.
Keep in mind, the Chief Information Officer (CIO) believed that this incident is from hackers. Then the EDA hired an external security company (contractor) to come in and assess the situation/system. The contractor found no actual malware infections. However, the CIO of EDA asked for a guarantee that there is non-existing of infection at all in the system [CIO is trying to safe his behind because of his belief]. The contractor could not give a guarantee due to the different between "could not exist" and "did not exist" of infections. That let to destroying the hardware part.
During the wait for recovery, the EDA entered into an agreement with Census to use their resources (e-mail, Internet, laptops, etc).
This is not done yet (and not included in the summary of this topic). The EDA did not listen to the recommendation from NSA or DHS about recovery plan -- quickly & fully recovery IT system. The EDA wanted a whole new system. This would cost $26 millions in total and won't be finished until the end of FY2014.
In summary, the miscommunication and other factors escalate the issue to be worse and worse. 1.DOC CIRT incorrectly handled the notification
1.DOC CIRT did not admit that their 1st notification was wrong to EDA
2.EDA did not verify the 2nd notification against the 1st with DOC CIRT
3.EDA did not submit random components (from 146) for verification
4.EDA IT system is outdated and has never been fixed/patched
5.CIO of EDA wanted to cover his behind by asking for a guarantee which is unrealistic
6.EDA wanted a whole new IT system which cost $26 millions
What do these people learn from the incident? No punishment but simply recommendations Deputy Assistant Secretary and the CIO of EDA (page 17 of the report/page 22 of the PDF file)! This situation is very similar to a big corporation making a mistake, and as a result, tax payers paid the price and nobody who were involved in the incident was punished.
You should read the report.
The contractor was on site for months, because EDA asked them to do the impossible, Prove that it was impossible for them to be infected.
The whole report is an amazing clusterfuck of misunderstandings and agencies speaking to each other in government-cover-your-ass-ees.
The DHS CIRT team told the EDA initially that 146 systems were infected with highly persistent malware. Then they sent them another report
with the exact same name that said only 2 systems were infected.
Within 2 weeks of beginning its incident response activities, EDA’s cybersecurity
contractor found the initial indications of extremely persistent malware were false
positives—not actual malware infections. However, EDA’s CIO sought guaranteed
assurance that the components were infection-free and no malware could persist.
External incident responders were unable to provide the assurance EDA’s CIO sought,
because doing so involved proving that an infection could not exist rather than that one
did not exist. By April 16, 2012, despite months of searching, EDA’s cybersecurity
contractor was unable to find any extremely persistent malware or indications of a
targeted attack on EDA’s systems. Further, the NSA and US-CERT did not find nationstate activity or extremely persistent malware.
If anything it appears there were only a few (2 or 6 depending on what part of the report you read) machines infected, but
worrywart EDA management insisted that the contractor keep looking.
The more you look at it the less the contractor seems to be at fault.
Had they just walked away, do you think they would ever get hired again?
Sig Battery depleted. Reverting to safe mode.