Atlanta Projected To Spend At Least $2.6 Million on Ransomware Recovery (zdnet.com)

Atlanta is setting aside more than $2.6 million on recovery efforts stemming from a ransomware attack, which crippled a sizable part of the city's online services. ZDNet reports: The city was hit by the notorious SamSam ransomware, which exploits a deserialization vulnerability in Java-based servers. The ransom was set at around $55,000 worth of bitcoin, a digital cryptocurrency that in recent weeks has wildy fluctated in price. But the ransom was never paid, said Atlanta city spokesperson Michael Smith in an email. Between the ransomware attack and the deadline to pay, the payment portal was pulled offline by the ransomware attacker. According to newly published emergency procurement figures, the city is projected to spend as much as 50 times that amount in response to the cyberattack. Between March 22 and April 2, the city budgeted $2,667,328 in incident response, recovery, and crisis management.

Comments

Ouch

[Errol backfiring]

That's a lot of money to restore a backup.

Re:Ouch

[Anonymous Coward]

Yes, can we fire the person who said 'backups are for wimps', or Oh, and this risk plan here - everyone nods heads. Competency first, is not their motto

Re:Ouch

[msauve]

More than "a backup," likely thousands of backups, with re-imaging of systems first. Plus, fixing the vulnerability and re-entering any manually processed data since the backup date. And that's assuming they have off-line backups which weren't affected by the attack.

Re:Ouch

[houghi]

If you think that backup is the same as a copy of the data, you are doing it wrong.

Re:Ouch

[bartle]

A company can have a 100% backup solution and it may still be worth their while to pay the ransom. The decryption process can be applied to all machines simultaneously, bringing them back online in perhaps a few hours. Alternatively, a thorough restore from tapes fetched from Iron Mountain could take a week or two.

Restoring from backup is a great solution for individuals, but large networks are unlikely to have a backup solution that can scale as well as a ransomware worm can. For large organizations, their money is best spent on preventing infection in the first place and mitigating it when it does occur.

Re:Ouch

[Opportunist]

For 26 millions I'd assume all this and a few things more, yes.

Re:Ouch

[msauve]

If you think making trite comments indicating a shallow understanding of the subject makes you clever, it doesn't.

Re:Ouch

[rahvin112]

It also covers the security "consultants" brought in to review things which is probably half the bill.

Re:Ouch

[Anonymous Coward]

Restore? Back-up ! You're a kidder, right ? Who do you think runs Atlanta Gub'mnt ?? Dem nibbaz donno baakup ...'ceptn when dat bitch be hotted up.

Re:Ouch

[Wycliffe]

A company can have a 100% backup solution and it may still be worth their while to pay the ransom.

Yes, assuming you can trust the criminal, it could possibly be cheaper but you should NEVER pay a ransom. It only open you and everyone else up for more ransom. I would much rather see paying ransoms outlawed and the government require everyone to carry ransom insurance and then have the insurance company pay to fix the problem. The advantage of this approach would be that if the insurance company pays for the recovery it reduces the incentive to pay the ransom and hopefully ransomware disappears. If we want ransomware to disappear, we need to make sure that it's cheaper and easier to not pay a ransom than it is to pay a ransom so that noone is tempted to pay a ransom. Another alternative is to make sure that the penalty for paying the ransom is so severe that noone is tempted.

Re:Ouch

[Anonymous Coward]

How about people who say, "backups have no ROI"?

Re:Ouch

[Anonymous Coward]

This is an AWESOME video. Thank you for posting this video. More people should view it. I found it funny and delightful to watch. It is amazing the slashdot editors would ban ANYONE for a video this good.

Re:Ouch

[Anonymous Coward]

He's right though. If your backups can't restore system states on some level, you're doing backups wrong.

Re: Ouch

[Anonymous Coward]

Iâ(TM)ll assume the city was using Windows.

If a GPO is set to enable windows backup and restore, then Apple Time Machine like functionality is enabled. If those files are backed up... done.

System imaging shouldnâ(TM)t be a concern if using laptops with a restore partition. This is a standard part of the WAIK.

I think all of this is part of the official cert guide for each version of Windows. So... it Windows 101.

Not sure if Stinkpads support this. I wonder if any Surface machines were effected.

Re:Ouch

[Anonymous Coward]

NPR's Planet Money took a look at the ransom issue. Episode 792 "The Ransom Problem".

[Noelle] KING[, host]: And he looked at how government policy works out in real life. So OK. There are two groups. There are the no-concessions countries, the United States, Canada, the U.K. And then there's a second group. These are countries that will make a deal - the French, the Italians, the Spanish, the Swiss, the Germans.

[Bryant] URSTADT[, host]: They don't go around advertising it, but these governments pay ransom. They deliver cash, and they mark it in their budget as foreign aid. So the world has been running this kind of horrible experiment. And here's what Peter Bergen found.

[Peter] BERGEN[, journalist]: The outcomes for Americans were twice as bad as they were for every other Westerners. And the only people who came close in terms of bad outcomes were the British.

URSTADT: And when you say twice as bad, what are you saying?

BERGEN: Well, double the number of Americans proportionately were killed by their captors.

KING: Americans are twice as likely to be killed in captivity. This could mean killing Americans in and of itself has value. It has propaganda value. Or it could mean that Europeans paying ransom gets people home. Or it could mean both of those things. But either way, if not everybody sticks to the same policy, it's a problem.

Re:Ouch

[Anonymous Coward]

Devil's advocate here: Do we want ransomware to go away?

Think about it. Back in the early 2000s, what got businesses to even think about security in the first place were viruses that would cause non-trivial hardware damage, erase BIOSes, fry monitors (back when if you fed a 200 Hz signal to most CRTs, the flyback transformer would pop, so kiss that high end Trinitron buh-bye. Those things were not cheap either. When companies had to buy new hardware repeatedly after the latest worm took out the entire form's PCs, the PHBs went, "gee, maybe we should do something about malware", and thus security got on the map for a bit, good enough to force Microsoft to move to multi-user operating systems on the desktop, and to get companies actually bothering with firewalls.

Now, we are back at the same thing. Most firms don't care about security since it doesn't bring any money in. In fact, getting hacked is actually a profit generator, since the C-levels can short their stock before making a public announcement.

As a devil's advocate, ransomware may be a good thing. It stops a company from functioning, which PHBs might consider something that doesn't "optimize their synergies", so they might actually give a thought to security.

Re:Ouch

[rickb928]

This is simple. If Americans will never, ever be ransomed, then nothing is lost by killing the American captives.

And this ensures that those nations that will pay are further convinced of the willingness of the captors to kill their captives, and more likely to pay.

This is reinforcing. Changing the policy of those nations that would pay will likely result in dead captives for a period, until the captors are convinced there is no money in the enterprise. This is a high cost, and the policy could be rolled back under pressure. The cycle begins again.

Re:Ouch

[rally2xs]

Wouldn't click on that supposed youtube video for all the tea in China. Gotta be malware at the other end...

Re:Ouch

[runenfool]

This would prove to be enormously expensive of a mandate on businesses and thus it will never happen.

Re:Ouch

[Wycliffe]

As a devil's advocate, ransomware may be a good thing. It stops a company from functioning, which PHBs might consider something that doesn't "optimize their synergies", so they might actually give a thought to security.

Ransomware insurance should achieve the same effect as presumably by proving you are more secure (or that it's less costly to recover your data) your premiums should be lower which would make the PHBs happier.

Re: Ouch

[Anonymous Coward]

I always back up my data using Java's serialization mechanism. Works great.

Re: Ouch

[Anonymous Coward]

I use byte arrays cast into FactoryAdaptors using JWT. It's the best. Also, nodeJS.

Re: Ouch

[Anonymous Coward]

Careful. Malware detected in this link chain.

Re: Ouch

[Anonymous Coward]

Yous a big fine woman, won't you back that ass up, call me big daddy when you back that ass up. - juvenile - back dat azz up.

Re:Ouch

[Anonymous Coward]

creimer, I reported you to youtube and keep reporting every spam post you make so all these spam posts will do is bring your view count in negative territory for a given day since youtube barred your stupid click-bot and your spam posts. minus 53 views for yesterday!

MODDOWN! ; creimer youtube spam post again!

creimer wants you to click on his youtube channel, then click on his stupid amazon affiliate link spam on Youtube. There is nothing of value on creimer youtube channel. Only creimer click-bot goes there.

Re: Ouch

[Anonymous Coward]

American policy towards random dates back to Jefferson.

A million on defense but not a penny in tribute.

Re: Ouch

[Anonymous Coward]

Backup is a complicated and expensive process. It is not like you can use an open source file system like BTRFS,with per user/machine samba/cifs network shares and rsynch to have an automatic hourly backup of any critical files for free. I mean an 8TB hard drive is like $200. Who has that kind of money to backup every eighty 100GB machines? Much less the technical ability to setup tftp to boot machines and automatically image them over the network via a startup menu "Press z to reimage this machine" on boot.

Or is this time portal not pointing to 1955?

Re: Ouch

[arglebargle_xiv]

I use BackupFactoryFactoryFactoryFactoryFactoryFactoryFactory. Unfortunately my BackupFactoryFactoryFactoryFactoryFactory was still in the process of manufacturing my BackupFactoryFactoryFactoryFactory, so I never got the backups done before the ransomware hit.

Re:Ouch

[Agripa]

How does management fire itself? Is that even possible?

Re: Ouch

[ExEm2SS]

That's your problem right there. You should have used AbstractDerivedSimpleMultiplexedBackupBackupBackupBackupBackupFactory instead.

Re:Ouch

[Anonymous Coward]

This is what happens when the Green Arrow and his team can't stop the evil hacker.
More seriously expect them to fork up at least as much to finally catch those responsible, because for that kind of money they never stop looking for you.

Re: Ouch

[Anonymous Coward]

Having worked at a place where our system was compromised more is necessary than just backup. Where I was at the attackers compromised printers, routers and other non-PC devices as well as the internal and external server appliances.
Restores were ineffective because we were reinfected as soon as systems were brought on line.
Short of replacing all appliances you can get into a really bad place. The fact you have backup doesn't really matter because even the BIOS and UEFI may be compromised.

Solution

[Kohath]

Contract out most of the work done by the city. Then if one of the contractors gets hit with ransomware, it's their problem. If that contractor can't meet obligations, switch contractors.

Re:Solution

[Opportunist]

...said the lawyer.

The problem is that you can sue someone into oblivion (usually a ltd company that goes *poof* the moment you try to squeeze money from it) means jack shit when your whole administration grinds to a halt and you can't get anything done sensibly anymore, constituents get REALLY pissed at you and vote the other guy in next time.

Who then gets your job AND whatever they can squeeze from the husk. Well done. Really. *golfclap*

Re:Solution

[david_thornley]

If the city has a responsibility to plow roads, then the city has the responsibility to make sure the roads get plowed. As Truman said, "The buck stops here." If the city has contracted the plowing to someone that can't deliver, that's a failure on the city's part. Either the city needs to find reliable contractors, or the city needs to find a way to plow that doesn't involve contractors.

Switching contractors can be painful on a small job, like repairing a roof. When you're talking about providing city services, there's likely to be nobody else available - and, if there is, the cost of hiring the new contractor is going to be pretty high. "Nice two-foot drifts you've got blocking all the streets. I'm sure we can arrive at an acceptable price without having to haggle a long time. Here's what I want to be paid."

Good job they made that figure public

[Oswald McWeany]

Now hackers know how much they can reasonably demand from Atlanta.

Re:Good job they made that figure public

[Oswald McWeany]

Now hackers know how much they can reasonably demand from Atlanta.

I can't help thinking that announcing such a budget has put a large bulls-eye right on the center of Atlanta's servers.

Re:Good job they made that figure public

[Anonymous Coward]

Not sure why you responded to yourself, but, I would say the exact opposite. Atlanta's government has sent a message that they'd rather spend 2.6 million dollars recovering data than 55,000 in ransom.

Why bother trying to extort someone that is willing to spend orders of magnitude more to tell you to F yourself?

Re:Good job they made that figure public

[PPH]

Not really. What the hackers know is that Atlanta will spend at least 5x the ransom demand rather then pay it. And I wonder how much of this $2.6 mill is a bounty on the hackers. The guys that bragged about taking the city for $55K has got to be wondering who their friends really are.

Re:Good job they made that figure public

[steveo777]

Well, they may need to pull in some analysts. Because $2,667,328 is being spent over weeks. Perhaps a cool $3M now up front is a bargain.

Or they could invest in real storage/backup/BC/DR solutions for much, much less.

Re:Good job they made that figure public

[nzkbuk]

Now hackers know how much they can reasonably demand from Atlanta.

They can demand all they want. The question is will Atlanta ever pay?
The core of the issue boils down to something like blackmail. As soon as you pay once you'll end up paying over and over again. At which point do you say no? Is the no point at the second time they ask for $55,000, the 10th, maybe after you've spent $5 million?
While I get "A sensible business decision dictate that you pay the original $55,000 rather than the estimated $2.6 million" I've also got to question if the original sum would have gotten their data back. There have been many occasions where paying the ransom did not get the data back

Re:Good job they made that figure public

[olsmeister]

If you are looking for ransom, I can tell you I don't have money. But what I do have are a very particular set of skills; skills I have acquired over a very long career. Skills that make me a nightmare for people like you.

Re:Good job they made that figure public

[Anonymous Coward]

it doesn't just cost $55k for the ransom. even if you payed the ransom you then still have to put forth additional time and effort to secure everything to make sure it doesn't happen again, and verify that you have cleaned all the affected computers. that costs more time and money than just the ransom.

Re:Good job they made that figure public

[Anonymous Coward]

If you are looking for ransom,

Nope. Just your name and address.

skills I have acquired over a very long career.

We have people with skills as well. Skills they have acquired over a very long career. Mainly spent at rifle ranges.

Re:Good job they made that figure public

[rickb928]

Are you overlooking the other costs of recovery? Paying the ransom and getting your systems decrypted is only the beginning.

And most of those costs would be the same whether you pay the ransom or not.

I doubt this is costing much more at all. For instance, you'll have to have all your systems scanned and reviewed to make the best effort to remove any other infestations, quite possibly replacing some or all outright. And then rebuilding the data security systems, training everyone to try and prevent this again, new network security, blah blah blah.

This is not cheap or easy to recover from if you're doing it right.

Re:Good job they made that figure public

[Anonymous Coward]

And most of those costs would be the same whether you pay the ransom or not.

Correct. So why pay it?

Re: Good job they made that figure public

[Anonymous Coward]

I agree with everything you said. It is very well thought out and you are obviously a handsome, clever individual.

Re: Good job they made that figure public

[rickb928]

Correct.

Re: Good job they made that figure public

[Anonymous Coward]

Why would you think that? Atlanta did not pay a dime to the hackers.

Re: Good job they made that figure public

[Anonymous Coward]

Your math needs some work.

Re:Good job they made that figure public

[david_thornley]

"Once you have paid the Danegeld/You will never be rid of the Dane" - Kipling.

even if they had paid

[bugs2squash]

Even if they had paid the ransom they would still need to fix the security holes though, so at least some of the extra expenditure is well justified.

Re:even if they had paid

[sl3xd]

I also remember seeing that the majority of those that pay ransomware are unable to recover data anyway.

Paying the ransom does only two things:

1. Encourages more ransomware, as it "works" as a business model
2. Would cost Atlanta another 55,000 in addition to the $2.6+ M to fix the problem.

Re:even if they had paid

[pr0fessor]

The never ending onslaught of maleware, ransomware, etc... annoys and frustrates me. To bad they are probably in a country where we can't extradite them.

Re:even if they had paid

[Kaenneth]

Drones and Gitmo =P

Re:even if they had paid

[sl3xd]

What do you mean? Microsoft is based in the US. They’re the one who refuses to stop making horribly insecure software.

They can’t even get Windows Update to work without rendering customer machines unusable.,,

Re:even if they had paid

[pr0fessor]

I thought SamSam exploited JBoss which is developed by Red Hat.

Re:even if they had paid

[sl3xd]

Lazy reporters no doubt see reports from 2-3 years ago where JBoss was widely used to proxy into a network, but they’re not paying attention: once they were “in” they used the proxy to attack systems inside.

Several other vectors have been added since 2016; SamSam attempting to exploit holes in Remote Desktop/RDP sessions is pretty common now.

Re:even if they had paid

[ebvwfbw]

Even if they had paid the ransom they would still need to fix the security holes though, so at least some of the extra expenditure is well justified.

If they do that. I bet they won't. Did you see the stupid law they passed down in Georgia banning security research? It was because government officials were embarrassed over an election exposure of passwords. Not a hack. They called the FBI on the researchers, who promptly cleared them. So I don't expect they'll fix stuff. They'll just blame anyone that points it out. Nope, emperor has clothes... Can't you see them?

Good to hear it works.

[houghi]

Always good to hear that it works. Remember people: backups are not about the fact if you take backups, but how fast you restore WHEN you need to.
The same goes for contingency. You do not check if the procedures are in place. You test it so you are ready WHEN it is needed.

One should always assume that something happens to all your data.

Also know that a copy of your data is not the same as a backup. One does not exclude the other.

I personally have a copy of my large data (movies, music and images) as those are basically read only. I have incremential data of other things AND a copy of the incremential data.

And I know what risks I take by having it all in the same house. Very few things I have off-site encrypted on two separate servers. That is about 20MB of data that is absolutely critical for me.

If I am able to figure out how to do it and what the risks are, they should be able to do so as well. Because had they invested that money in their ability to restore data, it would have saved a LOT of monies.

And paying out just atracks others to do the same (or even the same ones)

On an unrelated note, what is their IP address and email?

Re:Good to hear it works.

[Anonymous Coward]

There is no 'k' in "attract".

Re:Good to hear it works.

[UnknownSoldier]

This reminds me of a similar saying in the motorcycle world:

It is not a matter of IF you will wipe but WHEN you will wipe.

As a result we have the acronym: ATGATT: All the gear, all the time.
i.e. You don't wear gear for the 99.99%, but for that 0.01% of the time.

Bringing this back on top: It doesn't matter how fast you can do backups if your restore procedure is completely botched! You DID test it, right?

Re:Good to hear it works.

[afidel]

backups are not about the fact if you take backups, but how fast you restore WHEN you need to.

Amen to that, at job[-1] we had no problem hitting our backup windows but when we did a restore for a discovery request we found out that the interleving that allowed the tape drives to fly during backups made restores crawl to the point where our 48 hour and 72 hour SLAs were a joke. That led us to a disk to disk to tape solution which could restore files in minutes from the appliance and where if we had to reseed from tapes the restores were done to the appliance as one long streaming block which went at full LTO speeds. Best of all for critical systems the appliances even included the ability to act as an iSCSI target for the VMWare hosts so you could restore in place if the storage arrays blew up and you needed to get critical systems up an running ASAP.

Re:Good to hear it works.

[Anonymous Coward]

i have infected that word of his post with ransomware. pay me $55k in bitcoin to receive the decryption key to decrypt it.

Re: Good to hear it works.

[Anonymous Coward]

KGIII is that you?

Re: Good to hear it works.

[UnknownSoldier]

Sorry, never heard of KGIII. Who is that?

Needs regulation

[Anonymous Coward]

How can someone set up a payment portal that is not regulated? With regular banking, any transaction of at least $5000 is flagged and monitored. This is why everything cryptocurrency-related is perceived as criminal activity and the who damn shack will go down in flames.

Backups?

[Opportunist]

Could I maybe take a look at it? I might be able to offer you a solution for 25 millions a year...

Bezos is watching..

[Anonymous Coward]

He can strike Atlanta right off that Amazon HQ2 list.. 1. Horrible traffic. 2. Stupid city officials who can't protect themselves from cyber attack. 3. They picked Microsoft Azure for their platform.

Re: Bezos is watching..

[Anonymous Coward]

Or move them to top of the list.

1. Suckers who will pony up whatever is asked for, becuz cyber.

Better than paying ransom

[Anonymous Coward]

Better to pay 50x than to pay the ransom:

"We never pay any-one Dane-geld,
    No matter how trifling the cost;
For the end of that game is oppression and shame,
    And the nation that pays it is lost!"

- Rudyard Kipling, 1911

The price of using Windows,

[ReneR]

Maybe time to switch to Linux ;-)!

Re:The price of using Windows,

[sl3xd]

Nah, the time to switch to Linux was before Windows 10 started pushing upgrades which remove critical drivers.

In the past few weeks I've multiple fixed family & friend computers which were horked by Windows 10 Update deleting the SATA drivers, followed by input device drivers.

Who needs ransomware when Microsoft is bricking its user's computers?

Re:The price of using Windows,

[afidel]

Java doesn't care which platform it's running on...

Re:The price of using Windows,

[vandamme]

So how are they enjoying Linux, and what distros did you install?

Re:The price of using Windows,

[sl3xd]

I stick with a “rolling upgrade” capable distributor - Debian or OpenSuSE tumbleweed.

No complaints from anybody. Google Chrome and Firefox (and by extension, Netflix, Hulu, YouTube and Facebook) are pretty much the same everywhere.

Even the gamer is happy as his games are on Steam (a bit of a lucky break, but it’s working for him).

And I get to relax because I don’t have to worry about a Windows 10 update deciding to remove critical drivers.

Honestly, desktop Linux achieved feature parity a while ago. If you’re not a gamer whose game is Windows only, switching to Linux is as hard as going from Windows 7 to 10.

Re:The price of using Windows,

[vandamme]

But how did you replace the Windows malware download client??

Re:The price of using Windows,

[sl3xd]

I thought I was pretty clear that Windows is no longer on the systems. No Windows binaries of any kind.

So I’m not sure how any Windows program affects those systems. There’s certainly no Windows Update pushing anything to the machines anymore.

Should be a response?

[Anonymous Coward]

Seems like if the US has a cyber defense group, this should be on their radar.

This act needs a consequence.

Re:Should be a response?

[Anonymous Coward]

I agree. A higher level of government (whether the state or the feds) should punish the city workers for their irresponsible practices. Why the fuck are they installing malware? Why, after installing the malware, do they choose to run it as a user who has write access to everything? They shouldn't be doing that (as evidenced by this example) and it's a very costly thing to do to their constituents. If I were an Atlanta taxpayer, I would be furious.

Don't run malware. If your current policy is to run malware, you should change it. And if you're doing it as a government employee, that's basically a hostile act against the people who are represented by you. Just stop fucking doing that.

Malware will always be available, but that doesn't mean you should be using it! WTF gives people the idea to do that?!

Re: Should be a response?

[reanjr]

Generally speaking, security inside a corporate office is handled privately. The police don't guard buildings. Similar roles apply here. Unless Atlanta is handling DOD information or some such thing, it's not really the feds role to secure that. It's like the FBI looking into a robbery. Doesn't happen unless there's a federal angle.

Re: Should be a response?

[Anonymous Coward]

The federal angle is that if Atlanta goes belly up they're going to be going to the feds for a bailout.
Also it's likely that the ransomware is hosted outside the state of Georgia, that makes it an interstate or international matter, which means the Feds.
If Atlanta hasn't gone to the feds yet then the Governor of Georgia should be doing it.

I hope some systems...

[Anonymous Coward]

I don't live in Atlanta, but I hope some of their systems -- things like parking ticket records, low-level court dockets, and video surveillance -- are permanently ruined and erased and it will be too expensive to reconstruct the data. It would be nice if this hack actually gave middle-class and poor residents of Atlanta a break. Lemonade from lemons :)

Re:I hope some systems...

[DontBeAMoran]

Lemonade from lemons

“When life gives you lemons, don’t make lemonade. Make life take the lemons back! Get mad! I don’t want your damn lemons, what the hell am I supposed to do with these? Demand to see life’s manager! Make life rue the day it thought it could give Cave Johnson lemons! Do you know who I am? I’m the man who’s gonna burn your house down! With the lemons! I’m gonna get my engineers to invent a combustible lemon that burns your house down!” - Cave Johnson

Outsourcing != Problems vanishing

[sjbe]

Contract out most of the work done by the city. Then if one of the contractors gets hit with ransomware, it's their problem. If that contractor can't meet obligations, switch contractors.

Here in the real world it's not that simple. You need to think it through. Just because you outsource something doesn't make the problems magically go away. In many cases it actually is harder and more expensive to oversee the contractors than it is to do the job in house. There are real world consequences to suppliers not delivering and fixing problems is very often not as simple as switching suppliers. Good luck replacing the water treatment plant administration or the public transportation authority or the police or the fire department when they can't meet their obligations. When a building contractor fails to deliver it generally means huge cost overruns and switching can be difficult or impossible in many cases. How do you plan to replace the public schools that you now are contracting? Have fun replacing the company contracted to plow your roads in the middle of a snowstorm. Do you seriously think that any contractor with a brain isn't going to insist on clauses that make them difficult to remove?

Frankly there is a lot of stuff you absolutely do NOT want your city to contract out. Profit motives can be difficult to align with the interests of the citizenry and some important activities simply aren't profitable enough to contract out even if you wanted to.

So would disaster recovery have been worth it?

[Nkwe]

Clearly the city of Atlanta didn't have "proper" disaster recovery procedures in place. The interesting question is "Should they have?" From a pure financial point of view, would it have cost them more or less than $2.6 million to have put in place and regularly tested a disaster recovery procedure? I don't know the answer, but would be interested in hearing opinions. Sure, lots of people will say that "I can do backups for less than that", but an actual disaster recovery plan is way more than just doing backups. You have to test them and in the case of employee workstations you have to interrupt work. In the case of back end systems, even if they are redundant and highly available, certain kinds of restore operations will also interrupt work (an Active Directory restore for example if you are on a Microsoft platform, and whatever you are using for centralized authentication and configuration management for other platforms.) It would be interesting to see an analysis of the ongoing costs of disaster recovery plans (that can deal with a ransomware attack) vs the expected ongoing costs of such attacks.

Re:So would disaster recovery have been worth it?

[Anonymous Coward]

DR for a single system is (relatively) easy. E.g. a mainframe system: IPL system on mirrored disks at remote datacenter. We do this all the time, works fine.
DR for a network of systems is a nightmare, and the DR tests are either risky or useless.
Bring up DR mainframe, isolated network - fine, but doesn't do a proper test.
Open the network with addresses supposedly mapped to 'test' servers? Oops, you've just connected the DR test mainframe system to a production server...mayhem ensues as production data is fed into a test system while the real production mainframe loses its data feed.

Re:So would disaster recovery have been worth it?

[be951]

Sure, lots of people will say that "I can do backups for less than that", but an actual disaster recovery plan is way more than just doing backups.

That's true, but if they had decent backups at a minimum, they would be assured of getting all their data back. From what I've read, it is not clear that they did.

Re:So would disaster recovery have been worth it?

[Anonymous Coward]

I'm a Disaster Recovery Admin for a fortune 500 company. I can assure you that the amount of money it cost us to build our primary redundant datacenter and train everyone on the failover procedures is *well* over $2.6 million. If you compare that to the money we would lose if we were down as long as they were, it's chump change. As parent post states, disaster recovery is way more than just doing backups. We've been hit by WannaCry, power outages, hardware failures you name it. We can have mission-critical systems completely failover over to a datacenter hundreds of miles away in ~15 min tops. To get all the web apps and non critical systems up is usually ~1.5 hours (2hrs total to have them tested by the devs and signed off.) Sometimes it's not worth having clustered systems for non-critical systems. I suspect that the 2.6mil they're paying is cheaper than what they'd have to pay to have a highly available setup, BUT you can't always put a price on the 'trust' they're losing from their 'customers.'

Re:So would disaster recovery have been worth it?

[aaarrrgggh]

But you have no guarantees that the high availability replication processes in place don't end up getting infected as well-- you don't even (necessarily) know the root vulnerability that was exploited. Did they get in through the router, propagate to the switches, back themselves up to the copiers, and then perform ransom attack on servers, or was it a direct attack on the servers? Did they update the EFI?

When you have truly been screwed, it is almost impossible to know what parts of the system/network can still be trusted.

Sure, you can mitigate via compartmentalization, but it doesn't eliminate the risks and it extends recovery time for a wholesale problem.

Re:So would disaster recovery have been worth it?

[Anonymous Coward]

" It would be interesting to see an analysis of the ongoing costs of disaster recovery plans (that can deal with a ransomware attack) vs the expected ongoing costs of such attacks."

The ongoing DR costs will always be less than the costs of any attack, unfortunatly most people see the IT department as a cost center for which they need to reduce todays costs regardless of tomorrows risks. This is simply because of a focus on the next quarter by every operation in existance.

DR plans and Infrastructure roll out plans should always be considered for years into the future and while predictions are never 100% accurate and ounce of prevention is better than a pound of pain. 2.6 Mil would have paid for a couple highly skilled OPS guys as well as a couple educators and the hardware to back them up for several years. Why the educators you ask? well its simple, the educators handle the employees and mitigate the future risk or these things happening by properly informing people of how to use the tools that the organization provides them to do their jobs.

It infuriates me constantly that people fail to see that work place computers are tools for work and they should actually bother to learn to properly use the tools provided. Your personal devices may be for your amusement but work things are for work (that includes the network so no you cant put your phone on the office wifi)

Re:So would disaster recovery have been worth it?

[Phics]

Security is layered, and anyone who thinks DR and business continuity plans are all you need to protect against these threats is really doing things backwards. With appropriate next gen firewalls in place with proper UTM and endpoint protection, it's completely possible to track exploits, infections, and intrusions even through complex networks if you have the right security appliances in place. It's also possible to head these things off at the pass before they do extensive damage to a network by isolating the affected systems in the network. This can happen -very- fast, and can be handled in an autonomous fashion. What you're describing is Armageddon... the kind that sinks large businesses in a day. If you're spending that much money on DR, I'd expect there'd be a budget for the kinds of security solutions that would prevent or at least mitigate and isolate the actual damage in the first place. Recovering a few systems is one thing. Recovering a majority of your network sounds like your RTO just jumped from hours to weeks.

But hey... these things don't go down well at the budgetary meetings, do they?

java?

[Anonymous Coward]

more like .net vulnerability using java to do its dirty work..
you say randomware, i hear micro$oft tax.

I found the real culprit...

[Anonymous Coward]

"deserialization vulnerability in Java-based servers" Only the government would decide to run a Java backend and use the built-in serialization routines. Gotta sanitize your inputs. Always assume you are being attacked.

As for backup software, I use cloud backup software from cubiclesoft. Never had any issues and have restored from backups several times too.

Seems familiar

[DontBeAMoran]

Between the ransomware attack and the deadline to pay, the payment portal was pulled offline by the ransomware attacker.

Start something, then remove it before it gets popular. Sounds like something Google would do.

Commendable and irresponsible

[reanjr]

If I payed taxes to Atlanta, I'd probaly be miffed. But since I don't, I commend them for telling the hackers to fuck off.