Slashdot Mirror
Got Malware? Get a Hammer!
FuzzNugget writes "After the Economic Development Administration (EDA) was alerted by the DHS to a possible malware infection, they took extraordinary measures. Fearing a targeted attack by a nation-state, they shut down their entire IT operations, isolating their network from the outside world, disabling their email services and leaving their regional offices high and dry, unable to access the centrally-stored databases. A security contractor ultimately declared the systems largely clean, finding only six computers infected with untargeted, garden-variety malware and easily repaired by reimaging. But that wasn't enough for the EDA: taking gross incompetence to a whole new level, they proceeded to physically destroy $170,500 worth of equipment (PDF), including uninfected systems, printers, cameras, keyboards and mice. After the destruction was halted — only because they ran out of money to continue smashing up perfectly good hardware — they had racked up a total of $2.3 million in service costs, temporary infrastructure acquisitions and equipment destruction."
You mean I get to release my pent-up anger by destroying physical systems *and* get paid a boatload of money to do it? Where do I sign up?
... and yet I'm still furloughed on Friday...
It sounds like they were using this as an excuse to buy new equipment, so they destroyed extra equipment hoping that someone would allow them to chalk up the expense to the virus and thus give them shiny new stuff.
And why the hell would there be $2.3 million in service costs to destroy $170,500 worth of equipment?
Get free satoshi (Bitcoin) and Dogecoins
Sounds like a good start.
Have you seen the things that have been popping up on slashdot over the past couple years? USB drives in mice, intrusion software in power strips and keyboards, and more.
I don't think in this scale though. I suppose it drives the economy though - which is to say that they should seriously check what kind of benefits the guy in charge was receiving from their vendor-to-be.
world was created 5 seconds before this post as it is.
You know, to be sure?
Will that infect my lawnmower? I'd better destroy it then before it gets dangerous...
Nae king! Nae laird! Nae yurrupiean pressedent! We willna be fooled again!
And why the hell would there be $2.3 million in service costs to destroy $170,500 worth of equipment?
best buddy system.
that's why.
world was created 5 seconds before this post as it is.
Best Practices:
1. Take off and nuke the site from orbit, it's the only way to be sure.
like how we developed Iraq, destroy good infrastructure so contractors with gov officials in their pockets make a pile of money
or how government has developed inner cities over the past few decades, making fodder for the huge prison system business and food stamp system etc.
With users like this, who needs Malware?
EDA: did you guys just smash a bunch of computers with a hammer because of viruses?
DHS: Yes, but there havent been any terrorist attacks since we smashed everything with hammers. clearly the operation was a massive success.
EDA: I dont even.....
DHS: yep. Freedom isnt free.
Good people go to bed earlier.
Most cost overruns are due to scope creep. Customer solicits bids, contractors bid, one wins, shortly after contract is awarded the customer changes requirements.
General cycle is:
Customer asks if they can change a requirement
Contractor says it'll cost $$$ (usually a pretty big number, because many requirements are difficult to change after you've architected your system to the original requirements)
Customer says "sure"
Costs skyrocket.
As an example, with the last presidential helicopter.
Government requested bids
Companies submitted bids
Lockheed won
Shortly after contract award, White House came up with a list of "we need this, this, and this, or THE TERRORISTS WILL WIN!" - effectively wanting to transform the new aircraft from an executive transport aircraft to a flying tank
Lockheed gave a pretty big number because these new requirements blew the original weight budget of the selected aircraft
Customer said "sure"
Aircraft now needs uprated engines, an improved transmission, strengthened rotors, etc. - original bid was a minimally modified airframe, just avionics systems integration.
Costs went through the roof.
Because, RTFA?
"The total cost to the taxpayer of this incident was $2.7 million: $823,000 went to the security contractor for its investigation and advice, $1,061,000 for the acquisition of temporary infrastructure (requisitioned from the Census Bureau), $4,300 to destroy $170,500 in IT equipment, and $688,000 paid to contractors to assist in development a long-term response. Full recovery took close to a year."
-- "So they told me that using the download page to download something was not something they anticipated." - Bill Gates
Any IT upgrade could be spun exactly like this story, if you wanted... "why did you get a new mouse with that new system, the old one was working perfectly fine and now it's going in the trash!"
The feds are over reliant on contractors for everything. Contractors are there to just milk as much money as they can out of the system. They do a pretty good job.
-- "So they told me that using the download page to download something was not something they anticipated." - Bill Gates
OK, be honest now, who among us hasn't wanted to do this?
Admittedly, destroying mice and keyboards is a little excessive, but I bet there's not a single person here who isn't dreaming of needlessly destroying a large quantity of computer gear in a very dramatic manner.
Lost at C:>. Found at C.
Devil's advocate:
I've worked at private companies, for education institutions, in the public sector, and in the Federal government. None are perfect, none are completely horri-bad.
All places have had those people who I had zero clue what their function was, but they always had a nice office.
It is easy to pick on government, but go to almost any work environment, and you will find the same thing.
It's was the dreaded "PC LOAD LETTER" virus. Smashing is the only recourse.
Yup. Likely what happened here is that the million-dollar security contractors gave the advice to do this bug hunt in the first place, and then provided the temporary replacement infrastructure, and walked away from the whole fiasco with a tidy profit. The reason this happens is because the government isn't generally allowed to hire people to do work like this, because "private industry is better." Of course, this sort of private industry is just a mechanism for siphoning off tax dollars, and the people who believe that hiring government employees to do government work is wasteful are actually responsible for fiascos like this, which are depressingly common.
Even when the contractors aren't crooked, the cost of employing them instead of federal employees is typically several times higher. But "corporations good, government wasteful." If we keep repeating that long enough maybe it will come true.
And they wasted an additional $1.5 million paying various "contractors" who apparently didn't know what they were doing.
Or maybe they did, if you get my drift.
No reason to believe it wasn't cleaned up.
If they truly believe that it was the work of a nation-state, there is every reason to think it isn't cleaned up. Stuxnet didn't even reside just in computers. It infected programmable logic controllers attached to centrifuges, and then could re-infect computers on the network after they've been cleaned. If you really believe that Russia, or China has really compromised their network, and you have information that's worth more than a million dollars to them, then you should assume that everything (printers, routers, video-conferencing equipment, everything with a jack, plus the bios of all your computers) may be infected.
People tend to view $170,000 as a lot of money. But it's not. Computers for office workers can easily run under $1000. Hourly labor to clean things may be $50 per hour when you include overhead and benefits. And you're not even sure you got rid of the infection. If you mostly run apps that are resident on hardened servers, use imaging to make it easy to deploy new PCs, and don't have a lot of high end hardware, it may make sense to just replace everything with clean hardware. Honestly, for departments where you do think that there's stuff that sophisticated attackers may want, it may make sense to occasionally do this kind of purge occasionally even if you don't know there's been an attack. Take a look at the Sony Playstation breach for an idea of what getting compromised can cost. It's a hell of a lot more than $170,000.
Hopefully they'll be the ones in charge of healthcare.
Paying taxes to buy civilization is like paying a hooker to buy love.
Yeah baby, it's a great way to stimulate the economy. We know whst gets done is less important than things get done, and money gets pushed from person to person.
Buying computers to destroy employs people, as does destroying them. Hell, what we should do is just increase taxes and hire the tens of millions of unemployed to dig ditches and then fill them back in over and over.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
I'm sure nothing fishy was going on in this government center. I imagine they didn't want 3rd parties looking at their computers too closely. #tinfoilhaton
And why the hell would there be $2.3 million in service costs to destroy $170,500 worth of equipment?
RTFS.
service costs, temporary infrastructure acquisitions and equipment destruction
Or, RTFA for the details:
The total cost to the taxpayer of this incident was $2.7 million: $823,000 went to the security contractor for its investigation and advice, $1,061,000 for the acquisition of temporary infrastructure (requisitioned from the Census Bureau), $4,300 to destroy $170,500 in IT equipment, and $688,000 paid to contractors to assist in development a long-term response. Full recovery took close to a year.
Still outrageously stupid, but I think $4,300 to destroy $170,500 is a reasonable cost. I think the other costs--the ones with 6 or 7 figures--are the ones you should focus on.
But really, isn't giving US companies #2.3 million what the Economic Development Administration is supposed to be doing anyways? Better than spending it on the salaries for these government employees.
Battlemaster--Game with friends in medival realms
Not always true. I've seen many incompetent people continue to get promoted in industry. The government ones just get more press.
A bit OT, but:
It seems to be a symptom of some underlying pathology in a democracy when so much effort is put into protecting the head of government. At least in the ideal it doesn't matter who is president; they're ultimately a representative of the popular will and, to first order, one will do just as well as the next. There is even ideological continuity, since the vice president is selected by the president (you couldn't shoot Bush to end the Iraq war, since then you'd get Cheney). Historically assassinations have had little effect on the policy course of the nation. From the perspective of the citizenry, getting a president shot is really not all that big of a deal -- we just elect a new one and go about our way. This isn't saying that we should have no security around POTUS, but it seems rather disingenuous to pay for massive security for him, with its huge cost and disruption to people's lives, when that money, invested in health care or education or police presence in the worst neighborhoods, would reduce the death rate far more than the reduction in the (already small) assassination rate provided by the presidential security apparatus.
It seems to be a symptom of some underlying pathology in a democracy when so much effort is put into protecting the head of government. At least in the ideal it doesn't matter who is president;
You're completely missing the point of protecting the Head of State - it's not because an assassination would cause a change in policies, but to keep extremists from using threat of assassination to to blackmail a Head of State into changing those policies.
In other words, if the POTUS has to fear for his life as a result of every decision he makes, he is going to be pressured to cater to the most radical and violent groups.
There are industries and use-cases when "smash first, don't bother asking questions later" this is the appropriate response.
However, such times are rare and they should be spelled out ahead of time and they should only include destroying equipment which either 1) is at least theoretically possible to infect in a way that cannot be cleaned, ever (e.g. an infected BIOS), or 2) is deemed too expensive to clean and the data-storage media cannot be sterilized in a cost-effective manner or at all (e.g. a very cheap but hard to sterilize device, or write-once media).
Basically, if you are one of the very few shops that would need to resort to such things, you should know ahead of time the scenarios in which:
1) after a short investigation, you know cleaning is sufficient
2) after a short investigation, you know cleaning will be insufficient so just skip it and go straight to data recovery and destruction
3) the "edge cases" where it's worth spending some amount of extra time figuring out if it's 1 or 2 or, if you still can't figure it out, assuming 2.
Unless you've got special mice and keyboard that can be infected in a way that makes them not cost effective to clean, there's no reason to destroy them just because of a virus infection. As for printers and cameras, the dumber the device, the more likely you have no reason to destroy it. As for uninfected systems - how can you be sure it's uninfected? You can be 99.999% or 99.999999% or some other "%" sure but if the system was connected to a compromised system, unless it's "infection-proof" like a dumb mouse or you are 100% sure that the compromised systems weren't infected with anything that could have been passed on to the allegedly-uninfected system, you can't really be sure.
By the way, there is one other element of the calculation I didn't mention: Unless you can be 100% sure the replacement systems aren't infected, you may be just as well off keeping a 99.999%-sure-we-are-uninfected system than buying a replacement that you are only 99.999% sure isn't uninfected. After all, if I were a state actor, and I managed to infiltrate the PC-provider for a US-government agency and was slipping in BIOS-spyware-hooks in newly-purchased equipment, and I knew that infecting 6 of agencies computers with run-of-the-mill malware would force them to buy all-new equipment... bwuhahahaha.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
PJ O'Rourke said it best: "Giving money and power to government is like giving whiskey and car keys to teenage boys."
$823,000 for the security contractor that adviced them to do that destruction?
Read the story, or at least read the summary.
The contractor did not tell them to do that. The contractor found exactly 6 machines, which they recommended by re-imaged.
This stupidity was not the contractors fault.
Sig Battery depleted. Reverting to safe mode.
The actual destruction costs were only: $4,300 (still too much). The rest of that price tag is the total cost of doing the destruction - temporary infrastructure and so on. Not sure why a temporary replacement would cost 10x what was being replaced, though. Still plenty of government waste in the story.
Well except for the mice. You know how mice breed. Destroying those infected mice can take forever, because you find them breeding in closets, junk drawers, sometimes in their original boxes if bought at a TwoFer sale. And the wireless ones can be found a long way away from their nest, under desks, leaving their dongles everywhere.
They were lucky they managed to nip the infestation in the bud. It could have gotten totally out of hand had they owned any traveling laptops with mice. Entire countries might need quarantine. One mouse on a plane, and its game over.
Sig Battery depleted. Reverting to safe mode.
Chester Arthur is one of the few really impressive presidents in American history. When James Garfield was assassinated, Arthur declared that assassins would not be allowed to steer the course of US politics, and adopted Garfield's stance on the issue the assassin cared about (spoils system), obviously putting his life at risk by doing so.
It's hard to imagine a modern president doing that, and sadly as you point out he's been the exception, not the rule.
Socialism: a lie told by totalitarians and believed by fools.
best buddy system.
that's why.
That could be true; however, have you read the audit paper written by OIG in PDF (http://www.oig.doc.gov/OIGPublications/OIG-13-027-A.pdf)? It is very interesting and contains what the auditor (OIG) thinks where to blame (although those who are at fault simply brush the responsibility to others). Everything seems to be from miscommunication between DOC CIRT and EDA, and both did not know about this miscommunication until too late (the end of 2012, about a year after the incident).
What happened (from the audit paper) was that the incident handlers from DOC CIRT sent out 2 notifications to EDA regarding the US CERT notification. The first notification simply listed all 146 components, and EDA thought all of them were infected. Then the incident handlers from DOC CIRT sent the 2nd notification with accurate analysis of only 2 infected commponents, but the notification did not clarify or mention that the 1st notification was inaccurate (wrong). As a result EDA thought all 146 components were still infected.
Then the EDA selected and submitted 2 components to the DOC CIRT as a process to verify whether they were infected. Apparently, the EDA submitted the 2 components mentioned in the 2nd notification, and the result came back positive. As a result, the EDA thought that all 146 components were infected.
It got worse when EDA already knew that their IT system is outdated and needed a lot of updates/patches (since 2006 from NSA and OIG system reviewed) but they never fixed the issues. They believed this incident was an attack from nation-state actors (hackers), so their system could be extremely vulnerable to the attack. As a result, their system could open a hole to other systems' access. Therefore, the system was isolated.
Keep in mind, the Chief Information Officer (CIO) believed that this incident is from hackers. Then the EDA hired an external security company (contractor) to come in and assess the situation/system. The contractor found no actual malware infections. However, the CIO of EDA asked for a guarantee that there is non-existing of infection at all in the system [CIO is trying to safe his behind because of his belief]. The contractor could not give a guarantee due to the different between "could not exist" and "did not exist" of infections. That let to destroying the hardware part.
During the wait for recovery, the EDA entered into an agreement with Census to use their resources (e-mail, Internet, laptops, etc).
This is not done yet (and not included in the summary of this topic). The EDA did not listen to the recommendation from NSA or DHS about recovery plan -- quickly & fully recovery IT system. The EDA wanted a whole new system. This would cost $26 millions in total and won't be finished until the end of FY2014.
In summary, the miscommunication and other factors escalate the issue to be worse and worse. 1.DOC CIRT incorrectly handled the notification
1.DOC CIRT did not admit that their 1st notification was wrong to EDA
2.EDA did not verify the 2nd notification against the 1st with DOC CIRT
3.EDA did not submit random components (from 146) for verification
4.EDA IT system is outdated and has never been fixed/patched
5.CIO of EDA wanted to cover his behind by asking for a guarantee which is unrealistic
6.EDA wanted a whole new IT system which cost $26 millions
What do these people learn from the incident? No punishment but simply recommendations Deputy Assistant Secretary and the CIO of EDA (page 17 of the report/page 22 of the PDF file)! This situation is very similar to a big corporation making a mistake, and as a result, tax payers paid the price and nobody who were involved in the incident was punished.
You should read the report.
The contractor was on site for months, because EDA asked them to do the impossible, Prove that it was impossible for them to be infected.
The whole report is an amazing clusterfuck of misunderstandings and agencies speaking to each other in government-cover-your-ass-ees.
The DHS CIRT team told the EDA initially that 146 systems were infected with highly persistent malware. Then they sent them another report
with the exact same name that said only 2 systems were infected.
Within 2 weeks of beginning its incident response activities, EDA’s cybersecurity
contractor found the initial indications of extremely persistent malware were false
positives—not actual malware infections. However, EDA’s CIO sought guaranteed
assurance that the components were infection-free and no malware could persist.
External incident responders were unable to provide the assurance EDA’s CIO sought,
because doing so involved proving that an infection could not exist rather than that one
did not exist. By April 16, 2012, despite months of searching, EDA’s cybersecurity
contractor was unable to find any extremely persistent malware or indications of a
targeted attack on EDA’s systems. Further, the NSA and US-CERT did not find nationstate activity or extremely persistent malware.
If anything it appears there were only a few (2 or 6 depending on what part of the report you read) machines infected, but
worrywart EDA management insisted that the contractor keep looking.
The more you look at it the less the contractor seems to be at fault.
Had they just walked away, do you think they would ever get hired again?
Sig Battery depleted. Reverting to safe mode.
The stupidity appears to be with the CIO firmly convinced that there is persistent malware present that can't be removed with reimaging. Ie, assumed that the worst case scenario exists with malware from a nation state that remains hidden and undetectable. There was also some confusion that advice to "rebuild your network" meant destroying the existing infrastructure.
Sort of reminds me of the scene in Zoolander when the two idiots say "she said the files are IN the computer!" and then proceed to literally take the computer apart trying to find the files.