Do We Need Regular IT Security Fire Drills?

An anonymous reader writes: This article argues that organizations need to move beyond focusing purely on the prevention of security incidents, and start to concentrate on what they will do when an incident occurs. IT security "fire drills," supported by executive management should be conducted regularly in organizations, in order to understand the appropriate course of action in advance of a security breach. This includes recovering evidence, identifying and resolving the root cause of the incident (not just the symptoms), and undertaking a forensic investigation.

Comments

Pro- vs Re-

[hel1xx]

I see no issue with being proactive, vs. Reactive. No sense in shutting the barn door after all the horses have ran out?

Re:Pro- vs Re-

[epyT-R]

I've seen several departments that made reactive approaches a policy. Proactive employees were criticized and repeat offenders let go. I don't get it at all. It costs more money and makes more work and stress. Who wants to keep patching the same problem over and over?

Re:Pro- vs Re-

[Anonymous Coward]

Call the police, have the goon arrested then walk over and plug the server in. easy as lyin.

Re:Pro- vs Re-

[dbIII]

Reach for the tapes or other offline storage in case the other servers are mirrors of damaged garbage (as happened at a web hosting place near me that had a mirror but no backups). Same goes for snapshots - nice most of the time but if the machine has been taken over by someone those snapshots could be gone or changed.
IMHO a backup is not a backup unless there is something preventing you from immediately changing it - preferably an air gap of some sort.

Re:Pro- vs Re-

[war4peace]

"There's no back-up, I quit, you're screwed".

Re:Pro- vs Re-

[Capt.Albatross]

well but if your "proactive" is doing a fake reactive to the point of doing a "forensics investigation"... then you're just playing games.

When your proactive penetration testing finds a vulnerability, or one of your vendors issues a critical patch, follow through as if it were for real.

Incident Response Plan

[IT.luddite]

Write one, test it, maintain it. Otherwise by the time you realize you need one it's too late.

Re:Incident Response Plan

[plopez]

When in danger or in doubt,
Run in circles, scream and shout.

p. 101. Herman Wouk: THE CAINE MUTINY. Garden City, NY: Doubleday & Co., Inc. 1951. (p. 120 of the 1954 Doubleday pb ed.)

Heinlein lifted a lot of things. And it seems to be even older than that if google can be trusted

That's a different skill-set

[phantomfive]

This includes recovering evidence, identifying and resolving the root cause of the incident (not just the symptoms), and undertaking a forensic investigation.

That is not a skill set most IT departments have.

Re:That's a different skill-set

[Livius]

That is not a skill set most IT departments have.

I think that's the point.

Re:That's a different skill-set

[silas_moeckel]

Having a plan can be we have a contract with these guys to do this sort of work along with all the info they need. Along with all the paperwork and checking required.

Re:That's a different skill-set

[Lumpy]

90% of all IT departments can be driven bat shit crazy by installing a simple light timer on a router or switch and hiding it in the rats nest of power and other cables. Set the timer to be "anti burgular" mode where it adds randomness and have it drop power to a piece of gear for only 10 minutes once a day, because in 10 minutes by the time they get to the network closet, it will be back on and running.

It will drive them nuts and it will take MONTHS for them to find it, bet you they replace the router/switch befoer they find the timer. Bonus points if you make a decoy cable so that the timer is in the center of the cable hidden in the power tray and both ends look factory standard IEC.

Re:That's a different skill-set

[hel1xx]

You are evil. I like you.

Re:That's a different skill-set

[bill_mcgonigle]

That is not a skill set most IT departments have.

Many IT departments don't even have enough skill overage to deal with one guy being sick, much less have excess expert capacity.

Back in the 90's I watched a big medical center show the door to the guy who maintained the disaster recovery plan. He was "a cost center and never produced anything that anybody used."

That's about the timeframe when professional IT ended in the general population. Or maybe it's just when the general population got an IT staff.

Answer....

[bobbied]

Yes.... a million times YES

The "Be Prepared" motto isn't just for Boy Scouts, and it is not just about having what you need at hand, it's also about KNOWING what to do and being mentally prepared to do it quickly when required.

Re:Great!

[xaotikdesigns]

You need the "fire drill" plan created and typed out, then scheduled? Sure I'll put on my Fire Marshal hat and get to work on that...

Oh, what's that, you say that I'm also the new graphic designer, and I have deadlines for that stuff? OK, I'll get to that first...

You say that they can't print in Accounting either? And someone is having issues with their mouse, but you aren't sure who it was or what the problem was, but it needs fixed right now, and all the guys we hired three months ago had their passwords expire and they need them reset right now so they can log in and work?

I'll get right on it I guess...

Nope

[sexconker]

Just like real fire drills, they're pretty pointless and no one takes them seriously because there's no fire.
So you either have a fruitless exercise that costs money because of all the interruptions, or you have a semi-fruitful exercise that costs a lot of money because of the extended interruptions caused by trying to simulate a real event.

The latter will marginally improve the response to an actual incident. Neither will fly, because they cost money and aren't mandated by law.

Hopelesss

[fuzzyfuzzyfungus]

We don't need 'fire drills', we need Cold War style 'bend over and kiss your ass goodbye' drills. Unfortunately, I don't know of anyone, or any technique that prevents drills from turning into impromptu coffee breaks within a couple of rounds. People sharp enough to be with drilling just aren't fooled, and the dumb ones aren't much use. Unless IT security gets real, non drill, respect, what's the point? Any moron can point at a production environment and say "yeah, we could be doing that; but users and/or management would punch us." And this isn't even referring to esoteric stuff, I'm talking about boring, included-by-default stuff like software restriction policies(make sure that user-writeable locations and executable locations are a disjoint set and watch most trivial drive by and phishing attacks melt away...) Until we get to at least that level, why fuck around?

Re:Hopelesss

[pla]

Unless IT security gets real, non drill, respect, what's the point?

IT security won't get real respect until they actually know more than the people they annoy with their (literally) useless rules.

When you have some moron with a CISSP telling people who write network protocol stacks for a living what browsers they can use (this week), do you really expect to see a lot of "respect" flowing in that direction?

Modern InfoSec amounts to little more than snake-oil. AV vendors have admitted that their products can't keep us safe, while Mr. CISM insists on cranking up the settings to the point that an 24-core behemoth can barely get out of its own way.

Meanwhile, we hear about yet another fortune-500 compromise, with its own highly-paid head of IT security, on a daily basis.

You want respect? I get my job done. Try doing the same.

Re:Hopelesss

[fuzzyfuzzyfungus]

Arguably (on a systemic level, not on the level of how wonderful your current IT guy isn't) 'IT' being something that attracts actual talent qualifies as 'non drill respect'.

As long as "IT" means 're-image the desktops and reboot the mailserver when it needs it, monkey!', you aren't exactly going to get the IT people whose prowess impresses you. On the plus side, you'll save money. On the minus side, it's going to be a bloodbath if you get unlucky in terms of hostile attention.

So long as 'IT' is handled as a cost-center, necessary-evil, bunch of obstructionist ethernet janitors, that's how it'll be. On the plus side, modern technology is actually pretty easy to use, so if nothing atypically bad happens you can get away with some fairly dubious expertise at the wheel, and save accordingly; but if that's the philosophy at work you probably won't end up with an IT group capable of rising very far to the occasion should things go to hell(either because something that shouldn't have been complex went bad, or because lizard squad is on you).

What is unclear, at present, is how, culturally and financially, any but the most zealously paranoid and deep pocketed companies and state entities are going to have IT groups that are good for much more than the bare minimum. So long as you don't expect IT to be much better than a bunch of fuckups, there really isn't any reason to pay more or recruit more carefully(doing day-to-day IT is really more logistics and a little scripting than anything even remotely approaching CS or even code monkeying); but if that is how IT groups are recruited, no sane person will expect better of them; because why would they be capable of better?

(Please note, I freely acknowledge, as an institution's IT person, that I'd be up shit creek if something genuinely nontrivial came gunning for me. I'm a hell of a lot cheaper than a real expert, I have good rapport with the users, strong command of standard logistics and management tools, things go nice and smooth; but I'm hardly a guru, nor do I expect to be treated as one. However, that's why I'm skeptical about this 'drill' thing. If you want to know that We Are Fucked if things get serious, I can tell you that for free(though we do have backup tapes, and I am perfectly capable of restoring, were the hypothetical attack to stop); but if you aren't interested in doing anything that might actually make you less fucked; because that'd cost a whole lot more, upset users, or both, what's the drill for? Perhaps there are organizations that actually live in ignorance, believing that they have hardcore experts willing to do routine IT stuff at relatively low prices; but those are likely a delusional minority. Everyone else just knows that having a bulletproof IT team would be an eye-watering outlay(that would spend most of its time twiddling its thumbs and swappping the occasional toner cartridge until something actually happens), while having an adequate-for-daily-use IT team is markedly cheaper and you can always claim that you 'followed industry best practices' if something goes pear shaped.)

Re:No.

[plover]

That reminds me of one of those classic lists of airline mechanic log entries:
"Evidence of oil leak on landing gear. Signed, Joe Pilot"
"Evidence removed. Signed, Bob Mechanic"

test your backups / disaster recovery TODAY

[raymorris]

Just a friendly reminder - test your backups TODAY.
The MAJORITY of home and small business backups don't actually work when you try to restore. Often, it quit backing up 18 months ago and nobody noticed.

Disaster recovery is part of security, so that's one security drill. To handle an intrusion, often the best course of action is to unplug the network cable and call your expert. Do not power down the machine. Do not delete anything. Do not try to fix it. Just unplug the network and call the guy. That shouldn't be hard, but it is hard if you don't know who to call. If you're shopping for somebody during a panic, you'll likely pay too much for somebody who isn't as expert as you'd like. So find your expert ahead of time and you're most of the way there.

This is what security firms do

[GoodNewsJimDotCom]

They come in, test security via social engineering like if someone falls for phishing or whatnot. Then they educate based on what failed.

I interviewed with a firm once, and said,"Hey, maybe people don't even know they need your security product. How about sending phishing emails to all companies you might want to work for :P" He got a laugh and he said something like,"The window salesman doesn't go around throwing rocks through people's windows to stir up some business." I don't think the analogy is applicable, but my marketing suggestion was mostly a joke anyway.

These are simply audits

[cwills]

What you described is nothing more then a full security / disaster recovery audit. If your data center (and management) is really serious about it the company will need to invest both time and money to protect itself.

• Create your security policies. This has to be directed from a management level that can put teeth into it, as well as people who understand what the real risks to the business are. Company lawyers and people with business continuity experience might be involved depending on the consequences of what a data breach or disaster might do to the business.
  • determine what risks your business has
  • determine what needs to be done to mitigate the identified risks
  • determine what needs to be logged in order to allow forensic analysis (assume that the compromised system(s) logs themselves may have been corrupted as part of the breach)
• Make sure that the policies do not break the business. Also realize that security policies may require some processes to change.
• Understand that implementing security polices can be expensive.
• Employee education is a necessary step. Make sure employees understand what is being asked of them, and make sure that they understand what the policies are.
• Ensure that you have a designated security focal point.
• You will probably need an exception process. Make sure that any exceptions are documented with management, what is being done to mitigate any risks the exception have exposed and how long the exception needs to be in place.

Once you have your policies in place and everyone has "signed off" that they are in compliance, you can start with the auditing.

• Have some level of auditing where it's a "friendly" review of the systems.
• Audits should not instill fear, however there may need to be real consequences for negligent audit failures (depending on the business and type of data).
• Depending on the business, you may want to have an independent auditing group come in and review your systems and policies
• During an audit, system or process owners should only be held accountable to what is in the security policies. If the audit finds issues that are outside the policies, then management and the policy owner needs to respond.

One additional comment, depending on the size of the organization, there may be a security group. If there is one, then it should be the responsibility of this group to perform any security monitoring or testing. Individuals outside the group should not be performing their own security or intrusion testing of systems that they are not directly responsible for. If a vulnerability is uncovered, it should be documented and reported to the security focal point and management.