Atlanta Projected To Spend At Least $2.6 Million on Ransomware Recovery (zdnet.com)

Atlanta is setting aside more than $2.6 million on recovery efforts stemming from a ransomware attack, which crippled a sizable part of the city's online services. ZDNet reports: The city was hit by the notorious SamSam ransomware, which exploits a deserialization vulnerability in Java-based servers. The ransom was set at around $55,000 worth of bitcoin, a digital cryptocurrency that in recent weeks has wildy fluctated in price. But the ransom was never paid, said Atlanta city spokesperson Michael Smith in an email. Between the ransomware attack and the deadline to pay, the payment portal was pulled offline by the ransomware attacker. According to newly published emergency procurement figures, the city is projected to spend as much as 50 times that amount in response to the cyberattack. Between March 22 and April 2, the city budgeted $2,667,328 in incident response, recovery, and crisis management.

Comments

Ouch

[Errol backfiring]

That's a lot of money to restore a backup.

Re:Ouch

[msauve]

More than "a backup," likely thousands of backups, with re-imaging of systems first. Plus, fixing the vulnerability and re-entering any manually processed data since the backup date. And that's assuming they have off-line backups which weren't affected by the attack.

Re:Ouch

[bartle]

A company can have a 100% backup solution and it may still be worth their while to pay the ransom. The decryption process can be applied to all machines simultaneously, bringing them back online in perhaps a few hours. Alternatively, a thorough restore from tapes fetched from Iron Mountain could take a week or two.

Restoring from backup is a great solution for individuals, but large networks are unlikely to have a backup solution that can scale as well as a ransomware worm can. For large organizations, their money is best spent on preventing infection in the first place and mitigating it when it does occur.

Re:Ouch

[Opportunist]

For 26 millions I'd assume all this and a few things more, yes.

Re:Ouch

[msauve]

If you think making trite comments indicating a shallow understanding of the subject makes you clever, it doesn't.

Re:Ouch

[rahvin112]

It also covers the security "consultants" brought in to review things which is probably half the bill.

Re:Ouch

[Wycliffe]

A company can have a 100% backup solution and it may still be worth their while to pay the ransom.

Yes, assuming you can trust the criminal, it could possibly be cheaper but you should NEVER pay a ransom. It only open you and everyone else up for more ransom. I would much rather see paying ransoms outlawed and the government require everyone to carry ransom insurance and then have the insurance company pay to fix the problem. The advantage of this approach would be that if the insurance company pays for the recovery it reduces the incentive to pay the ransom and hopefully ransomware disappears. If we want ransomware to disappear, we need to make sure that it's cheaper and easier to not pay a ransom than it is to pay a ransom so that noone is tempted to pay a ransom. Another alternative is to make sure that the penalty for paying the ransom is so severe that noone is tempted.

Re:Ouch

[rickb928]

This is simple. If Americans will never, ever be ransomed, then nothing is lost by killing the American captives.

And this ensures that those nations that will pay are further convinced of the willingness of the captors to kill their captives, and more likely to pay.

This is reinforcing. Changing the policy of those nations that would pay will likely result in dead captives for a period, until the captors are convinced there is no money in the enterprise. This is a high cost, and the policy could be rolled back under pressure. The cycle begins again.

Re:Ouch

[rally2xs]

Wouldn't click on that supposed youtube video for all the tea in China. Gotta be malware at the other end...

Re:Ouch

[runenfool]

This would prove to be enormously expensive of a mandate on businesses and thus it will never happen.

Re:Ouch

[Wycliffe]

As a devil's advocate, ransomware may be a good thing. It stops a company from functioning, which PHBs might consider something that doesn't "optimize their synergies", so they might actually give a thought to security.

Ransomware insurance should achieve the same effect as presumably by proving you are more secure (or that it's less costly to recover your data) your premiums should be lower which would make the PHBs happier.

Re: Ouch

[arglebargle_xiv]

I use BackupFactoryFactoryFactoryFactoryFactoryFactoryFactory. Unfortunately my BackupFactoryFactoryFactoryFactoryFactory was still in the process of manufacturing my BackupFactoryFactoryFactoryFactory, so I never got the backups done before the ransomware hit.

Re:Ouch

[Agripa]

How does management fire itself? Is that even possible?

Re: Ouch

[ExEm2SS]

That's your problem right there. You should have used AbstractDerivedSimpleMultiplexedBackupBackupBackupBackupBackupFactory instead.

Good job they made that figure public

[Oswald McWeany]

Now hackers know how much they can reasonably demand from Atlanta.

Re:Good job they made that figure public

[Anonymous Coward]

Not sure why you responded to yourself, but, I would say the exact opposite. Atlanta's government has sent a message that they'd rather spend 2.6 million dollars recovering data than 55,000 in ransom.

Why bother trying to extort someone that is willing to spend orders of magnitude more to tell you to F yourself?

Re:Good job they made that figure public

[PPH]

Not really. What the hackers know is that Atlanta will spend at least 5x the ransom demand rather then pay it. And I wonder how much of this $2.6 mill is a bounty on the hackers. The guys that bragged about taking the city for $55K has got to be wondering who their friends really are.

Re:Good job they made that figure public

[steveo777]

Well, they may need to pull in some analysts. Because $2,667,328 is being spent over weeks. Perhaps a cool $3M now up front is a bargain.

Or they could invest in real storage/backup/BC/DR solutions for much, much less.

Re:Good job they made that figure public

[nzkbuk]

Now hackers know how much they can reasonably demand from Atlanta.

They can demand all they want. The question is will Atlanta ever pay?
The core of the issue boils down to something like blackmail. As soon as you pay once you'll end up paying over and over again. At which point do you say no? Is the no point at the second time they ask for $55,000, the 10th, maybe after you've spent $5 million?
While I get "A sensible business decision dictate that you pay the original $55,000 rather than the estimated $2.6 million" I've also got to question if the original sum would have gotten their data back. There have been many occasions where paying the ransom did not get the data back

Re:Good job they made that figure public

[olsmeister]

If you are looking for ransom, I can tell you I don't have money. But what I do have are a very particular set of skills; skills I have acquired over a very long career. Skills that make me a nightmare for people like you.

Re:Good job they made that figure public

[rickb928]

Are you overlooking the other costs of recovery? Paying the ransom and getting your systems decrypted is only the beginning.

And most of those costs would be the same whether you pay the ransom or not.

I doubt this is costing much more at all. For instance, you'll have to have all your systems scanned and reviewed to make the best effort to remove any other infestations, quite possibly replacing some or all outright. And then rebuilding the data security systems, training everyone to try and prevent this again, new network security, blah blah blah.

This is not cheap or easy to recover from if you're doing it right.

Re: Good job they made that figure public

[rickb928]

Correct.

Re: Good job they made that figure public

[Anonymous Coward]

Why would you think that? Atlanta did not pay a dime to the hackers.

Re:Good job they made that figure public

[david_thornley]

"Once you have paid the Danegeld/You will never be rid of the Dane" - Kipling.

even if they had paid

[bugs2squash]

Even if they had paid the ransom they would still need to fix the security holes though, so at least some of the extra expenditure is well justified.

Re:even if they had paid

[sl3xd]

I also remember seeing that the majority of those that pay ransomware are unable to recover data anyway.

Paying the ransom does only two things:

1. Encourages more ransomware, as it "works" as a business model
2. Would cost Atlanta another 55,000 in addition to the $2.6+ M to fix the problem.

Re:even if they had paid

[pr0fessor]

The never ending onslaught of maleware, ransomware, etc... annoys and frustrates me. To bad they are probably in a country where we can't extradite them.

Re:even if they had paid

[Kaenneth]

Drones and Gitmo =P

Re:even if they had paid

[sl3xd]

What do you mean? Microsoft is based in the US. They’re the one who refuses to stop making horribly insecure software.

They can’t even get Windows Update to work without rendering customer machines unusable.,,

Re:even if they had paid

[pr0fessor]

I thought SamSam exploited JBoss which is developed by Red Hat.

Re:even if they had paid

[sl3xd]

Lazy reporters no doubt see reports from 2-3 years ago where JBoss was widely used to proxy into a network, but they’re not paying attention: once they were “in” they used the proxy to attack systems inside.

Several other vectors have been added since 2016; SamSam attempting to exploit holes in Remote Desktop/RDP sessions is pretty common now.

Re:even if they had paid

[ebvwfbw]

Even if they had paid the ransom they would still need to fix the security holes though, so at least some of the extra expenditure is well justified.

If they do that. I bet they won't. Did you see the stupid law they passed down in Georgia banning security research? It was because government officials were embarrassed over an election exposure of passwords. Not a hack. They called the FBI on the researchers, who promptly cleared them. So I don't expect they'll fix stuff. They'll just blame anyone that points it out. Nope, emperor has clothes... Can't you see them?

Good to hear it works.

[houghi]

Always good to hear that it works. Remember people: backups are not about the fact if you take backups, but how fast you restore WHEN you need to.
The same goes for contingency. You do not check if the procedures are in place. You test it so you are ready WHEN it is needed.

One should always assume that something happens to all your data.

Also know that a copy of your data is not the same as a backup. One does not exclude the other.

I personally have a copy of my large data (movies, music and images) as those are basically read only. I have incremential data of other things AND a copy of the incremential data.

And I know what risks I take by having it all in the same house. Very few things I have off-site encrypted on two separate servers. That is about 20MB of data that is absolutely critical for me.

If I am able to figure out how to do it and what the risks are, they should be able to do so as well. Because had they invested that money in their ability to restore data, it would have saved a LOT of monies.

And paying out just atracks others to do the same (or even the same ones)

On an unrelated note, what is their IP address and email?

Re:Good to hear it works.

[UnknownSoldier]

This reminds me of a similar saying in the motorcycle world:

It is not a matter of IF you will wipe but WHEN you will wipe.

As a result we have the acronym: ATGATT: All the gear, all the time.
i.e. You don't wear gear for the 99.99%, but for that 0.01% of the time.

Bringing this back on top: It doesn't matter how fast you can do backups if your restore procedure is completely botched! You DID test it, right?

Re:Good to hear it works.

[afidel]

backups are not about the fact if you take backups, but how fast you restore WHEN you need to.

Amen to that, at job[-1] we had no problem hitting our backup windows but when we did a restore for a discovery request we found out that the interleving that allowed the tape drives to fly during backups made restores crawl to the point where our 48 hour and 72 hour SLAs were a joke. That led us to a disk to disk to tape solution which could restore files in minutes from the appliance and where if we had to reseed from tapes the restores were done to the appliance as one long streaming block which went at full LTO speeds. Best of all for critical systems the appliances even included the ability to act as an iSCSI target for the VMWare hosts so you could restore in place if the storage arrays blew up and you needed to get critical systems up an running ASAP.

Re: Good to hear it works.

[UnknownSoldier]

Sorry, never heard of KGIII. Who is that?

Backups?

[Opportunist]

Could I maybe take a look at it? I might be able to offer you a solution for 25 millions a year...

Better than paying ransom

[Anonymous Coward]

Better to pay 50x than to pay the ransom:

"We never pay any-one Dane-geld,
    No matter how trifling the cost;
For the end of that game is oppression and shame,
    And the nation that pays it is lost!"

- Rudyard Kipling, 1911

Re:Solution

[Opportunist]

...said the lawyer.

The problem is that you can sue someone into oblivion (usually a ltd company that goes *poof* the moment you try to squeeze money from it) means jack shit when your whole administration grinds to a halt and you can't get anything done sensibly anymore, constituents get REALLY pissed at you and vote the other guy in next time.

Who then gets your job AND whatever they can squeeze from the husk. Well done. Really. *golfclap*

Outsourcing != Problems vanishing

[sjbe]

Contract out most of the work done by the city. Then if one of the contractors gets hit with ransomware, it's their problem. If that contractor can't meet obligations, switch contractors.

Here in the real world it's not that simple. You need to think it through. Just because you outsource something doesn't make the problems magically go away. In many cases it actually is harder and more expensive to oversee the contractors than it is to do the job in house. There are real world consequences to suppliers not delivering and fixing problems is very often not as simple as switching suppliers. Good luck replacing the water treatment plant administration or the public transportation authority or the police or the fire department when they can't meet their obligations. When a building contractor fails to deliver it generally means huge cost overruns and switching can be difficult or impossible in many cases. How do you plan to replace the public schools that you now are contracting? Have fun replacing the company contracted to plow your roads in the middle of a snowstorm. Do you seriously think that any contractor with a brain isn't going to insist on clauses that make them difficult to remove?

Frankly there is a lot of stuff you absolutely do NOT want your city to contract out. Profit motives can be difficult to align with the interests of the citizenry and some important activities simply aren't profitable enough to contract out even if you wanted to.

Re:The price of using Windows,

[sl3xd]

Nah, the time to switch to Linux was before Windows 10 started pushing upgrades which remove critical drivers.

In the past few weeks I've multiple fixed family & friend computers which were horked by Windows 10 Update deleting the SATA drivers, followed by input device drivers.

Who needs ransomware when Microsoft is bricking its user's computers?

So would disaster recovery have been worth it?

[Nkwe]

Clearly the city of Atlanta didn't have "proper" disaster recovery procedures in place. The interesting question is "Should they have?" From a pure financial point of view, would it have cost them more or less than $2.6 million to have put in place and regularly tested a disaster recovery procedure? I don't know the answer, but would be interested in hearing opinions. Sure, lots of people will say that "I can do backups for less than that", but an actual disaster recovery plan is way more than just doing backups. You have to test them and in the case of employee workstations you have to interrupt work. In the case of back end systems, even if they are redundant and highly available, certain kinds of restore operations will also interrupt work (an Active Directory restore for example if you are on a Microsoft platform, and whatever you are using for centralized authentication and configuration management for other platforms.) It would be interesting to see an analysis of the ongoing costs of disaster recovery plans (that can deal with a ransomware attack) vs the expected ongoing costs of such attacks.

Re:So would disaster recovery have been worth it?

[Anonymous Coward]

DR for a single system is (relatively) easy. E.g. a mainframe system: IPL system on mirrored disks at remote datacenter. We do this all the time, works fine.
DR for a network of systems is a nightmare, and the DR tests are either risky or useless.
Bring up DR mainframe, isolated network - fine, but doesn't do a proper test.
Open the network with addresses supposedly mapped to 'test' servers? Oops, you've just connected the DR test mainframe system to a production server...mayhem ensues as production data is fed into a test system while the real production mainframe loses its data feed.

Re:So would disaster recovery have been worth it?

[be951]

Sure, lots of people will say that "I can do backups for less than that", but an actual disaster recovery plan is way more than just doing backups.

That's true, but if they had decent backups at a minimum, they would be assured of getting all their data back. From what I've read, it is not clear that they did.

Re:So would disaster recovery have been worth it?

[Anonymous Coward]

I'm a Disaster Recovery Admin for a fortune 500 company. I can assure you that the amount of money it cost us to build our primary redundant datacenter and train everyone on the failover procedures is *well* over $2.6 million. If you compare that to the money we would lose if we were down as long as they were, it's chump change. As parent post states, disaster recovery is way more than just doing backups. We've been hit by WannaCry, power outages, hardware failures you name it. We can have mission-critical systems completely failover over to a datacenter hundreds of miles away in ~15 min tops. To get all the web apps and non critical systems up is usually ~1.5 hours (2hrs total to have them tested by the devs and signed off.) Sometimes it's not worth having clustered systems for non-critical systems. I suspect that the 2.6mil they're paying is cheaper than what they'd have to pay to have a highly available setup, BUT you can't always put a price on the 'trust' they're losing from their 'customers.'

Re:So would disaster recovery have been worth it?

[aaarrrgggh]

But you have no guarantees that the high availability replication processes in place don't end up getting infected as well-- you don't even (necessarily) know the root vulnerability that was exploited. Did they get in through the router, propagate to the switches, back themselves up to the copiers, and then perform ransom attack on servers, or was it a direct attack on the servers? Did they update the EFI?

When you have truly been screwed, it is almost impossible to know what parts of the system/network can still be trusted.

Sure, you can mitigate via compartmentalization, but it doesn't eliminate the risks and it extends recovery time for a wholesale problem.

Re:So would disaster recovery have been worth it?

[Phics]

Security is layered, and anyone who thinks DR and business continuity plans are all you need to protect against these threats is really doing things backwards. With appropriate next gen firewalls in place with proper UTM and endpoint protection, it's completely possible to track exploits, infections, and intrusions even through complex networks if you have the right security appliances in place. It's also possible to head these things off at the pass before they do extensive damage to a network by isolating the affected systems in the network. This can happen -very- fast, and can be handled in an autonomous fashion. What you're describing is Armageddon... the kind that sinks large businesses in a day. If you're spending that much money on DR, I'd expect there'd be a budget for the kinds of security solutions that would prevent or at least mitigate and isolate the actual damage in the first place. Recovering a few systems is one thing. Recovering a majority of your network sounds like your RTO just jumped from hours to weeks.

But hey... these things don't go down well at the budgetary meetings, do they?

Re:The price of using Windows,

[afidel]

Java doesn't care which platform it's running on...

Seems familiar

[DontBeAMoran]

Between the ransomware attack and the deadline to pay, the payment portal was pulled offline by the ransomware attacker.

Start something, then remove it before it gets popular. Sounds like something Google would do.

Re:I hope some systems...

[DontBeAMoran]

Lemonade from lemons

“When life gives you lemons, don’t make lemonade. Make life take the lemons back! Get mad! I don’t want your damn lemons, what the hell am I supposed to do with these? Demand to see life’s manager! Make life rue the day it thought it could give Cave Johnson lemons! Do you know who I am? I’m the man who’s gonna burn your house down! With the lemons! I’m gonna get my engineers to invent a combustible lemon that burns your house down!” - Cave Johnson

Commendable and irresponsible

[reanjr]

If I payed taxes to Atlanta, I'd probaly be miffed. But since I don't, I commend them for telling the hackers to fuck off.

Re: Should be a response?

[reanjr]

Generally speaking, security inside a corporate office is handled privately. The police don't guard buildings. Similar roles apply here. Unless Atlanta is handling DOD information or some such thing, it's not really the feds role to secure that. It's like the FBI looking into a robbery. Doesn't happen unless there's a federal angle.

Re:Solution

[david_thornley]

If the city has a responsibility to plow roads, then the city has the responsibility to make sure the roads get plowed. As Truman said, "The buck stops here." If the city has contracted the plowing to someone that can't deliver, that's a failure on the city's part. Either the city needs to find reliable contractors, or the city needs to find a way to plow that doesn't involve contractors.

Switching contractors can be painful on a small job, like repairing a roof. When you're talking about providing city services, there's likely to be nobody else available - and, if there is, the cost of hiring the new contractor is going to be pretty high. "Nice two-foot drifts you've got blocking all the streets. I'm sure we can arrive at an acceptable price without having to haggle a long time. Here's what I want to be paid."

Re:The price of using Windows,

[vandamme]

So how are they enjoying Linux, and what distros did you install?

Re:The price of using Windows,

[sl3xd]

I stick with a “rolling upgrade” capable distributor - Debian or OpenSuSE tumbleweed.

No complaints from anybody. Google Chrome and Firefox (and by extension, Netflix, Hulu, YouTube and Facebook) are pretty much the same everywhere.

Even the gamer is happy as his games are on Steam (a bit of a lucky break, but it’s working for him).

And I get to relax because I don’t have to worry about a Windows 10 update deciding to remove critical drivers.

Honestly, desktop Linux achieved feature parity a while ago. If you’re not a gamer whose game is Windows only, switching to Linux is as hard as going from Windows 7 to 10.

Re:The price of using Windows,

[vandamme]

But how did you replace the Windows malware download client??

Re:The price of using Windows,

[sl3xd]

I thought I was pretty clear that Windows is no longer on the systems. No Windows binaries of any kind.

So I’m not sure how any Windows program affects those systems. There’s certainly no Windows Update pushing anything to the machines anymore.