Slashdot Mirror
Do We Need Regular IT Security Fire Drills?
An anonymous reader writes: This article argues that organizations need to move beyond focusing purely on the prevention of security incidents, and start to concentrate on what they will do when an incident occurs. IT security "fire drills," supported by executive management should be conducted regularly in organizations, in order to understand the appropriate course of action in advance of a security breach. This includes recovering evidence, identifying and resolving the root cause of the incident (not just the symptoms), and undertaking a forensic investigation.
I see no issue with being proactive, vs. Reactive. No sense in shutting the barn door after all the horses have ran out?
IT Professional.
Write one, test it, maintain it. Otherwise by the time you realize you need one it's too late.
This would help our IT department if they weren't too busy constantly unfucking our 20 year old solaris machines.
Well, I do have my instant pop-up Blame Finger ready. (Careful, don't confuse those things with the Commute Finger.)
Table-ized A.I.
Another thing to add to my plate of executive orders.
Seriously, IT in an organization isn't like most other jobs. We always have many tasks to accomplish. While I absolutely agree and have worked in various places to ensure that we are prepared for all sorts of disasters (fire, PSU\PDU failures, data breach, etc), it is hard to schedule it in with all of the other work that needs done. Leading towards overworked people that become tired and unhappy whose pay may or may not be worth it to them for these extra exercises.
If you every worked in IT not management, you would know finding the "root cause" most times is a wild goose chase. Do you think doctors besides House ever find a "root cause".? No you recognize the symptoms, and fix accordingly. I post this as the same time Slashdot just gave me a 503 error, please tell me the "root cause". Your current "server instability" is not the answer management is looking for in this case.
This includes recovering evidence, identifying and resolving the root cause of the incident (not just the symptoms), and undertaking a forensic investigation.
That is not a skill set most IT departments have.
"First they came for the slanderers and i said nothing."
Yes.... a million times YES
The "Be Prepared" motto isn't just for Boy Scouts, and it is not just about having what you need at hand, it's also about KNOWING what to do and being mentally prepared to do it quickly when required.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
the only reason we do fire drills is their mandated by law. Every business I know of is trying to cut IT costs. There's no way in hell this idea would fly. It's always cheaper to pick up the pieces as long as you don't really care about the damage.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
next question?
If you were me, you'd be good lookin'. - six string samurai
Would this not be the equivalent of hiring a pentesting firm on an x basis to give you a detailed report of your security flaws? This would give you your "root cause" I suppose -- and give management something else to berate their IT department over.
IT Professional.
My bosses do everything they can to undermine security. Spending time and money to test is out of the question as their opinion is, and I quote: "Why would anyone ever do that?" *facepalm*
Just like real fire drills, they're pretty pointless and no one takes them seriously because there's no fire.
So you either have a fruitless exercise that costs money because of all the interruptions, or you have a semi-fruitful exercise that costs a lot of money because of the extended interruptions caused by trying to simulate a real event.
The latter will marginally improve the response to an actual incident. Neither will fly, because they cost money and aren't mandated by law.
Yes there should be. It would mean IT security and correct procedures would be much more likely to be followed. It would also raise the profile of IT within the organisation. Too often IT is treated like the red-headed step child janitor, until it hits the fan.
Obviously sending everyone home, putting a note on the door, and definitely closing the gym are first in the list.
We don't need 'fire drills', we need Cold War style 'bend over and kiss your ass goodbye' drills. Unfortunately, I don't know of anyone, or any technique that prevents drills from turning into impromptu coffee breaks within a couple of rounds. People sharp enough to be with drilling just aren't fooled, and the dumb ones aren't much use. Unless IT security gets real, non drill, respect, what's the point? Any moron can point at a production environment and say "yeah, we could be doing that; but users and/or management would punch us." And this isn't even referring to esoteric stuff, I'm talking about boring, included-by-default stuff like software restriction policies(make sure that user-writeable locations and executable locations are a disjoint set and watch most trivial drive by and phishing attacks melt away...) Until we get to at least that level, why fuck around?
I'm not agreeing with anything until I get to hear what Bennett Hasselton has to say about it, preferably in a 5000 word wall of text.
If your information security department isn't investigating issues and possible incidents on the regular, they probably aren't doing any monitoring of any kind.
Most companies would risk losing ten dollars before they spend one to prevent a loss because a dollar spent is a dollar lost, and until they incur a real loss the spent dollar looks like a waste.plus, spending is easier once a real loss happens due to insurance etc.
I think it would make a ton of sense for every organization to do a DR "drill" periodically where they attempt to actually use their DR plan (restore a group of servers, reload a switch configuration, etc).
This just seems like a sensible part of that.
What worries me, though, is how they will know when to actually implement a security plan and deal with the consequences. A lot of security breaches are subtle, and you don't know they've happened or at least not always with a definitive sign like a defacement page, etc.
I would assume a "real" security response would be something akin to putting a lot of resources "in lockdown" -- shutting down servers, cutting network links, etc, which could have major business consequences. I can see where uncertainty about a breech and hesitancy to isolate key systems (perhaps necessary to contain a breech) could lead to a real clusterfuck.
I think a key part of developing the plan is deciding when you know there is a real breach and making sure that the responses are well-known ahead of time to avoid a lot of head-scratching and internal conflict.
Not much more to be said about it. The staff will know how to react when there's real problems rather than searching for passwords and documentation for some system they haven't touched in 6 months..
TFA is utterly void. I suspect it was written by a bot.
Just a friendly reminder - test your backups TODAY.
The MAJORITY of home and small business backups don't actually work when you try to restore. Often, it quit backing up 18 months ago and nobody noticed.
Disaster recovery is part of security, so that's one security drill. To handle an intrusion, often the best course of action is to unplug the network cable and call your expert. Do not power down the machine. Do not delete anything. Do not try to fix it. Just unplug the network and call the guy. That shouldn't be hard, but it is hard if you don't know who to call. If you're shopping for somebody during a panic, you'll likely pay too much for somebody who isn't as expert as you'd like. So find your expert ahead of time and you're most of the way there.
At a minimum, you should have a DR plan, you should periodically review your DR plan, and you should from time to time actually test your DR plan. There's a zillion other things you can do above and beyond that -- but many an organization has had their DR plan utterly fail them in the face of a real emergency because nobody took it seriously.
No boom today; boom tomorrow. Always boom tomorrow. Plan for it, and you might come out of it fine. Don't, and you could be screwed.
If the executive fails to understand this and fund it, you need a new CIO/CTO -- because nobody is worrying about business risk.
Lost at C:>. Found at C.
I'd love to see management come in during a work day, pull a couple plugs, push a couple buttons and take down everything so we can blame them. 100% HA is impractical, so we spend our budget ensuring the most common 90% of outage causes are protected against. There will always be a way to take down everything, and you should save those for yourself on the day you get fired.
Up to a certain point, the plethora of options with the inbuilt human element makes doing drills based on incident response, prone to alienating the IT staff, after all then it would start from having to examine the supply chain, up to the physical security up to the laying of the lines, corroboration of system installation, sampling of motherboard chip integrities, peripheral integrity, management vs user network separartion, having an appropriate byod policy, whitelists, blacklists, hygeine policies , etc. the root cause analysis could be sheer lunacy. Which is why red team excercises should be followed up with post mortem for all involved, because tracing the causes means the root cause is the managment itself.
But, taking a leaf from web scale companies, what it seems is the use of homogeneous components with built in failure tolerances, virtualization and software defined networking, might do quite a bit to mitigate intrusion and infection, and from exfiltration. It is not by chance that these guys develop their own servers, use cots equipment and participate in open source projects.
They come in, test security via social engineering like if someone falls for phishing or whatnot. Then they educate based on what failed.
:P" He got a laugh and he said something like,"The window salesman doesn't go around throwing rocks through people's windows to stir up some business." I don't think the analogy is applicable, but my marketing suggestion was mostly a joke anyway.
I interviewed with a firm once, and said,"Hey, maybe people don't even know they need your security product. How about sending phishing emails to all companies you might want to work for
God spoke to me
Quick, cut the Internet connection! Ok, restore the connection, drill is done.
Do your company also do real life security "fire drills," supported by executive management should be conducted regularly in organizations, in order to understand the appropriate course of action in advance of a physical security breach. This includes recovering evidence, identifying and resolving the root cause of the incident (not just the symptoms), and undertaking a forensic investigation?
No? Then perhaps you don't need to do IT security fire drills for the same reason.
Oliver.
IT departments get plenty of field testing.
APK quotes people (including myself) without context and should not be trusted. Just thought you should know.
... Very simply, either have someone in your IT department or an outside consultant hack your system or compromise it in some way.
Then task the department to deal with it.
Let us say your fake attacker gets a hold of some admin passwords? Or they slip a remote access program through your security? Something like that. Then task the department to solve the problem and then make the system harder to compromise.
Ultimately what needs to happen is that systems need to be compartmentalized so that the compromising of one system does not lead to the compromising of EVERYTHING. That in and of itself is hugely helpful.
And then keep in mind the distinction between system security in areas that you don't care about and system security in areas you do.
So for example, if there is a part of the system that deals with non-sensitive information that you honestly don't care about then the security there doesn't need to be that stiff. And if we're talking about credit card records or private company memos then clearly that needs to be secured tight.
The problem has been that all these systems tend to be linked such that if you can compromise the low security area you have access to everything. Compartmentalize the systems so that that doesn't happen and then tailor each security system to be reasonable for what it is securing. That means amongst other things not annoying users with security theater for things they don't need to worry about.
The security systems need to be well designed, fluid, and effective.
One thing which I'd like to see implemented more in operating systems is whitelisting systems. That is, instead of trying to keep on top of every bit of bad code you instead make a list of all good code. Any time bad code wants to run, it has to be verified by the IT department. Here someone will say "what about scripts"... you can white list scripts too guys. Don't be silly. You have the system keep an international record of the name and precise to the bit size of every valid script as well as some sort of hash value. And then when a script wants to run, it has to have the same file name, file size, and hash value. Am I missing something here? Someone is doubtless going to say I didn't think of one thing or another. I accept and welcome your criticism in advance, gents. we all need to listen to each other and come up with some good solutions here together.
Cheers.
I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
What you described is nothing more then a full security / disaster recovery audit. If your data center (and management) is really serious about it the company will need to invest both time and money to protect itself.
Once you have your policies in place and everyone has "signed off" that they are in compliance, you can start with the auditing.
One additional comment, depending on the size of the organization, there may be a security group. If there is one, then it should be the responsibility of this group to perform any security monitoring or testing. Individuals outside the group should not be performing their own security or intrusion testing of systems that they are not directly responsible for. If a vulnerability is uncovered, it should be documented and reported to the security focal point and management.
That's a very good point.
A separate issue is bare metal restore drills for things with complex procedures, but that's a one per person per type of complex system issue instead of a regular drill idea. If in three years time the next version of whatever has a few differences that probably not enough to have to rerun the "drill".
I often see where IT departments do all the restore, but never check the backup. The tape was never changed, because the procedure to do that was never followed up when 1 person left and a few months later his backup left as well.
The tape was now without any metal.
Another was where the backup was in testmode and never actually ran.
A third one was where they did not do incremental backups, but only a copy instead, resulting in not having the data that was needed.
This all due to various reasons. Mostly because they were interested if the backup were running. Backups are just a tool, not a goal. Restoration of data is the goal and the best way to do that is with backups.
Do you really think they would be able to have a working retention plan? Only the larger companies I have seen actually test their retention plan. Cut everything off and see if their backup site will handle it.
Don't fight for your country, if your country does not fight for you.
Um ... "supported by executive management" ... they're the ones outsourcing IT in the first place. What are they going to support? Firing more people?
I thought they would be awesone until I realized what they were. Mostly a way to show off to higher ups. The bulk of them end up being about showing off pretty charts and dashboards no matter how useless those charts are. How you can make these work is tell your staff that management will be hiring a pen test sometime in the next six months but they won't get any more detail. This allows you to test your staff whole making them be more on their toes in case a real attack happens.
1. Find out which salesman caused it
2. Fire them
http://www.vthreat.com/ was founded, by Marcus Carey, accelerated by http://www.mach37.com/ and recently funded to provide "IT fire drills" to organizations. I'd say if you can get funded & launch a product, it's an important thing to be doing. At the very least have some table top exercises where you or others ask some what if's, then take the answers or lack there of and fix them, and do it again.
TKrabec Pahh
Annual drills ( especially Infosec related ones ) have been a part of every company I've ever worked for for the last 20 years. Any one who isn't required to do them needs to take a hard look at their business priorities.
When ever systemd appears on a machine, it needs to be immediately contained and quashed!
Speaking from the viewpoint of someone high up in the echelon of LARGE U.S. corporation I.T. Middle management on up through top management are NOT even slightly interested in data retention. Primarily because it would incriminate them. AND I know this because I've had that conversation WITH upper management.
What they do prevent changes in security controls until the deadline, then panic! Add in a serious vulnerability that needs to be patched too for good measure. Of course this could be just bad management as well.
Of course policies and procedures should be developed and tested. Otherwise, it's crap. Seen it my entire life. Untested code/procedures don't work.