Slashdot Mirror
Do You Have Your 'Crisis Week'?
pmbarth asks: "This week, the large company I work at is having a 'Crisis Week', where we simulate different types of problems, and have training on how to deal with them. Beyond the normal fire drills or chemical spills, a new addition was 'Attack on IT Infrastructure'. I was wondering how many other companies out there are actually training their non-IT employees on how to be aware of, and perhaps even counteract these types of issues?" It's an interesting idea, and one can't tell when an extra skill one learns on the job may come in use in a critical situation. Do other companies have something similar? Do you think such drills are particularly effective?
Attention! E-Mail coming in! It has an evil attachment; keep your eyes away from it! Don't open it, lest civilization as we know it ends forever!
Now try to find the delete key... press it... done. Ah, life can go on now.
Yeah, software companies should have "Stand Down" week, where the company goes off crisis mode for 7 days and people go home to sleep.
Someone you trust is one of us.
Along with allowing us to test our procedures once a year, in case of a real emergency, our critical systems can be reloaded at their facilities and brought back up until the neccesery repairs/reloads are performed here.
When you're doing a test, or an actual disaster recovery, they also have a full staff of experienced sysadmins to help. This is really valuable as even the most experienced sysadmin doens't get as much practice at disaster recovery as those guys do.
As you can imagine, services like this aren't cheap, but they are aimed at large companies that need this type of protection. They also have a whole host of other services they provide, mostly network monitoring services; this may include testing your site's ability to defend against a hacker attack, but I'm not sure. If you work for a large (or growing) company and don't yet have a company providing these services for you, I highly, highly recommend Comdisco.
--- Rectum?! Damn near killed em'! - Confucius
Then one day, we actaully had a fire, at about 6 pm. Three of us were containing it, and called the QuarterDeck (front office) to sound the alarm, which they did, except they announced that it was a drill!
Sooo, the people who usually run the drill (officers=managers) called the QuarterDeck and told them to cancel it because there was no drill for the day.
Needless to say, we spent a VERY LONG time on the phone before the QuarterDeck got the story right, and the fire crew finally arrived.
Moral of the story: Don't get so caught up in doing drills that you miss the actual fire!
Never never never smoke crack before geometry class!
Marcus Ranum gave an interesting talk on intrusion detection systems and security including physical threats at ALS last year. I'd also recommend secrets and lies by Schneier. It also takes an interesting look at physical security issues. As for crisis week the last one I can think of was Y2K but that wasn't really a mock up type thing. The only other crisis preparedness we were trained for was 'fire'.
Of course telling people that you're going to have a simulated crisis is not very effective at all. It just has to happen without warning or the workers are definitely going to be prepared.
:-)*
One day I came in to work and I was told that the CVS server went down. The support staff knew exactly when it went down because NetSaint sent messages to their phones.
I'm not normally support/admin, but I have experience in it so I jumped in to help. Here is what we did:
- Went to the console and tried to boot it up. No go
- I booted from a rescue disk and tried to boot it that way. Nope
- Tried to mount the partitions, found that the partition table was gone
- We then split into two different efforts: I mentioned gpart (guesses lost partition tables) and started running it with various options while the other team began rebuilding the server from backups
- gpart didn't work so I just partitioned it again with the original settings (I've done that successfully before on a home computer)
- That didn't work, but the replacement server was ready by then so we plugged it into the network
Once the backup server was up the head of development announced that he had replaced the CVS server's hard drive with a blank one early that morning.
We all wrote reports on what we did and, while we were pissed for a minute ("You WHAT!?!?!?"), the drill was determined to be a success.
I was freaked out mainly due to the fact that I volunteered to help out... Me and my big mouth
I sysadmin for a government research lab. You'd better believe every week is an IT crisis week. If it's not crackers in China looking for revenge for the embassy accident, it's some dumbfsck college kid trying to telnet past the routers or something.
Those aren't the crises, though (the routers keep those jerks out). The actual crises begin when the logfiles get too big to fit on the backup tape. Then I have to scrounge around to find more tapes, 'cause they won't let me buy any more on the government budget (yes dammit I'd raise my own grandmother's taxes if it means I have money to buy backup tapes), and then I have to decide whether the stuff currently on the tapes can be sacrificed for the holy cause (backups! backups always take priority!). This decision-making process usually requires some caffeine, and the single soda machine within reach charges a freaking dollar for a 20-oz bottle, so there's another twelve or thirteen dollars gone.
Don't talk to me about "planned" crisis week.
You cannot apply a technological solution to a sociological problem. (Edwards' Law)
This is one of the topics covered in the CISSP exam, I think the CISA also has it. Methods for disaster recovery, which are often ignored by many companies. Often I wonder how much a company has prepared for a disaster, via way of anything imaginable, hurricanes, fires, break-ins, etc.
;) ... For those with higher ranking positions I suggest you go out and get the "Information Management Handbook -- Tipton/Krauss" which has tons of informative information regarding safeguarding data, disaster recovery techniques, etc. Its one of the best books I ever bought.
Personally I think companies grow too fast and focus on growing, growing, growing, rarely stopping to take the time to implement measures against disaster recovery.
One of the things we do @ my place is once every other month we have a sit in with beers, pizza, etc., and focus on security via way of games. Why do you need a safe password is based on a guess your co-workers info to see how much we can gather by knowing them to see if we could guess their pw's, we also have a twist on Jeopardy where we use the names obtained from Attrition.org, and make a question about the company, so we could say "yes this company was owned this/last month" in order to make our workers aware of the risks involved on the `net'.
Its better than ramming security down their throats and constantly lecturing people. We also have little twists on dealing with all sorts of issues, voicemail management to avoid having pw's cracked, social engineering games, and makeshift scenarios where someone comes in to social engineer their way into information.
keep us on our toes
Want Root?
I'm glad the editors are posting stuff that piques their interest, but maybe it's time for a bit more editorial creativity? A vocab building class perhaps? Or maybe they should change the site name to "Slashdot: An interesting idea."
[ yes, this is offtopic. It's probably also flaimbait. But I, for one, think it's funny. Or at least intersting. hehehe ]
---
Ah! Another NT domain...
---
Book(n): Utensil used to pass time while waiting for the TV repairman
If you work in IT, you owe it to your company it your coworkers to practice this essential drill. I believe OSHA is considering making it mandatory for all businesses with more than 45mbps of bandwidth total (across all locations).
In order to perform an effective ping flood drill, you'll need every employee in the building to be equipped with the proper ping flood protective gear: two buckets, a mop, a snorkel, and a waterproof flashlight.
The drill should come as a surprise, so employees learn to react quickly and safely in the event of a real ping flood.
To begin the drill, a senior IT staffer should use the in-building paging system (if the building is not so equipped, a megaphone may be substituted).
Announce in a clear, calm voice, "Your attention please! We are currently experiencing a ping flood! All employees to ping flood response stations! This is not a drill!"
IT staffers should walk the building, making sure that employees are using their buckets and mops properly. The most common mistake non-IT staff makes when dealing with a ping flood is to not echo-reply properly. Unless you are practicing an IRC ping flood, people should *not* be saying "PONG!" This is a common panic response among employees, and part of the reason for the ping flood drill.
IT staff should also ensure that everyone in the building is mopping properly, and bailing the buckets out of the window, you may halt the drill. In the drill, of course, there will be no actual pings in the buckets, but it's important to have complete realism. Some buildings may have to have their windows knocked out with a chair or piece of computer equipment. The expense is well worth it in the event of an actual ping flood.
Although an actual ping flood can last for hours, you should limit a ping flood drill to no more than 45 minutes, as exhaustion may set in and render employees unable to deal with a real ping flood, should one occur immediately after the drill.
If you are in IT and not practicing this essential drill, you are negligent and irresponsible. If upper management refuses to allow you to stage ping flood drills, it is your moral obligation to do so anyway. When a real ping flood occurs, they will thank you for it.
Cheers
-b
If I wanted a sig I would have filled in that stupid box.
We don't bother with simulating an IT crisis, we simply allow people to log into the network and do their daily tasks.
main(i){(10-putchar(((25208>>3*(i+=3))&7)+(i ?i-4?100:65:10)))?main(i-4):i;}
Beyond the normal fire drills or chemical spills, a new addition was 'Attack on IT Infrastructure'.
They took away the coffeemaker?
must... have... caffeine... to... code...