Slashdot Mirror
Do You Have Your 'Crisis Week'?
pmbarth asks: "This week, the large company I work at is having a 'Crisis Week', where we simulate different types of problems, and have training on how to deal with them. Beyond the normal fire drills or chemical spills, a new addition was 'Attack on IT Infrastructure'. I was wondering how many other companies out there are actually training their non-IT employees on how to be aware of, and perhaps even counteract these types of issues?" It's an interesting idea, and one can't tell when an extra skill one learns on the job may come in use in a critical situation. Do other companies have something similar? Do you think such drills are particularly effective?
I have my "Crisis Week." It's called, "finals." In 37 hours, 23 minutes, and 8 seconds, I am gonna be... so fucking drunk
--
I think there is a world market for maybe five personal web logs.
So, when they simulate the network hacking attack crisis, just run down the hall to the data center. Hit the big red switch on the wall.
You will see that the network hacking attack exercise will be successfully ended, and everyone will begin the simulation of what to do when the data center loses power.
If tits were wings it'd be flying around.
Right on, my brother. It's been a couple of years (gee, time flies) but I was trained that it was always better to know where to look things up then to try and know everything. STAN-EVAL just ate that shit up. I wish we could have done more with the continuity binders though, we were always so busy putting out fires and holding the office together we didn't have much time for things, like proper docs and recurring training.
I was a 1W051 (Weather Observer) and every unit I was at always seemed to be running as fast as they could just to stay in one place. I was never able to make much progress trying to make the place better, it was very frustrating and is one of the reasons that I didn't reenlist.
Of course I'm in the same boat with my current job, running as fast as I can just to keep from being swamped. The difference is that I have hope in my current job that it will get better sooner rather than later. Also in my current job I really can make a difference and make things better for myself and the others around me. Sometimes it helps being small.
-- Remember: Wherever you go, there you are!
...and if it really is a child trapped under a vehicle, you'll be able to count on Lassie to let you know.
Therefore, if it's not from Lassie, it's not a crisis.
You are in a maze of twisty little passages, all alike.
Then I got to the bottom of the page, and saw this quote:
Your love life will be... interesting.
Ya gotta love irony.
--
I can't get to CNN/eBay/Am I Hot or Not/online bingo/every other non-business essential site.
Of course, what's the first thing I do when the router goes down? Dial my laptop to a local ISP and make sure I can get my /. cause, dammit, this is important!
No boom today. Boom tomorrow. There's always a boom tomorrow. - Cmdr. Susan Ivanova
1. CEO Fired [How fast can you find a new one?]
2. CFO Fired [How fast can you forge his signature all those PO's you've been wanting.]
3. Stock Devaluation [Speed selling]
4. Stock Devaluation at a dot com [Resume update]
Get a DLT (digital linear tape) and create a cron script to run nightly/weekly according to your discretion. DLT's have dropped since there are other alternatives, so their inexpensive (considering you have a 15tb set up) or you could always have a Clariion purchased for this. Even a Netapp using bzip2 on a sys will do more justice than people realize. Many companies don't have any idea how important a backup plan is until the shit hits the fan and they're shit out of luck.
Its always good to have at least 90 days worth of backups in case something may have been corrupted, whats more is the storage of that data. You don't want to just leave it lying around. Consider renting small storage space for a monthly dump of the tapes or keep em locked up in a secure place such as a safe, or encrypt the tapes so the average joe can't read it should they get their hands on them (the tapes)
I hate even thinking about the idea of going to a new company, and having to deal with this since its a painstaking task, but once its done, its all a matter of following up on things, and making it part of daily/weekly/monthly work. It gets easier once you've done it and gotten it over with.
Want Root?
---
...is identifying how we can pass the crisis-response knowledge on to the next generation of employees. One focus in US military exercises is not only on how each situation is dealt with, but how each person knows what to do in the first place. Rather than simply train each person from scratch then hoping they remember what to do under every single set of circumstances, the US Air Force often creates multi-purpose response checklists and "continuity" binders/folders that contain everything from basic response overviews to detailed information on how to deal with various problems. Saying "I'm not sure but the answer is written right *here*" is very nearly as good as having the answer memorized especially if the answer is available to everyone in the organization.
It's a huge pain in the a$$ to prepare this material but it helps insure that there's always SOMEONE around who knows what to do and there is a source of info to check when the crap hits the fan. A little investment in time to create these documents pays off in a big way, one incident at a time. These are living documents instead of regulations, so they are continually subject to improvement or even disposal if they become obsolete or something better comes along.
This kind of thing ought to work in the corporate world especially in a company where there is a measure of procedural inertia that carries on regardless of who is holding down each particular job.
Sounds like a lot of folks (at least in charge) with too little to do. The environment I'm in is constant crisis. I'd sign on for a 'NonCrisis Day' in a heartbeat.
I find that massaging the back right under the ribs also helps. Also, there is a program for the palm pilot that is helpful in keeping track when the next one will occur.
-no broken link
A lack of planning on your part does not constitute an emergency on my part.
---
Book(n): Utensil used to pass time while waiting for the TV repairman
I for one am tired of all these interesting stories. I demand boring stories on Slashdot!= \=\=\=\=\=\
=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\
Lordy God, you know people are getting jaded when they think Ethernet switches are a foregone conclusion. Knowing how to set up a network to reduce collisions, are knowing what a bus topology reeeeealy dates you these days.
Vintage computer games and RPG books available. Email me if you're interested.
Oooh good idea. I know what I'm coding up tomorrow.
Vintage computer games and RPG books available. Email me if you're interested.
Considering how many times I've seen our non-technical employees warn the company about hoaxes and spread email viruses I wouldn't be surprised if these people would give out dangerous information to a clever social engineer.
Rats would be more funny if they could fart.
Actually, my company has a daily drill.
Someone codes something good and shouts something like 1 4M S000 13373 D00DZ!!11 (or something else in 13373 speak), then we beat the ever living crap outta them.
I'd like to see a script kiddie pop their head up in my company.
BTW - This article sounds like a preface to a new BOFH.
Good quote, too many chars. Seriously, the slashdot 120 char limit sucks!
Those people are merely sharing how common crisis mode is.
Even if it is just due to managers believing that you can do with half the staff, or half the hardware, or half the budget you need to get the results they want.
Maybe it was a silly question to begin with, but it sure seems like whoever had the 150 moderator points had a problem with that much unexpected opinion.
Check out the Vinny the Vampire comic strip
"It is a greater offense to steal men's labor, than their clothes"
Well, crisis has a little different meaning in the Milittary. Try codeing in MOPP-4 (full chem) gear. Its not fun. Yes we have the regular bomb threat practice and the runs Tornado Shelter, both practice and real, I'm stationed At Tinker AFB in Oklahoma. Its a very, for lack of a better word, "Interesting" Job.
Sanchi
"They said we couldn't do it [Athlon]... but we built it, we shipped it... and we didn't have to recall it." Rich Heye
I was thinking though, technically most mail that I get falls under the category of "unexpected" :)
dynamo
Well, I have to admit, it's solid in theory, but my experience ( both electronically and physically ) is that folks like to loose all their "cool" in situations of a critial nature ( SysAdmins not excluded ). People like to run during fire alarms ( when they're fot real ), tornados, when their the targets of crackers, etc. I guess drilling probably does give one person a very cushy job though.
My "original" and "personal" expressions go here.
For me, a good month is a month when I can get one week of project work done, where I'm not constantly being called away to put out fires.
It's disappointing to see that as time goes by, more and more of my work putting out fires. It reflects badly on the management of the organization as a whole, but that's the nature of IT. You don't hear from people unless thay have a problem. Crisis week once in a while huh? Try having crisis week three times per month.
--
--Got Lists? | Top 95 Star Wars Line
Most likely running under Netware...
political_news.c: warning: comparison is always true due to limited range of data type
"I wouldn't have that code finished until next week, I'm practicing having crisis this week."
OR
"NOooooooooo! this is the third time this week I've split bong water on my keyboard."
--
M0571y H@rml355.
I know you're kidding, but I've recently rebelled against a perpetual sense of crisis at the workplace. It's the product of an over-caffienated Calvinism or something, but in many workplaces there's an unfocused and pervasive attitude of constant emergency. When any of it drifts my way, I now have a general response: if there's something actually and concretely urgent that is comparable with a child being trapped under a vehicle, then I'll freak out with the best of them. Otherwise, it's not really an emergency; rather, it's a frantic toadying, and I'll have none of it.
I have brought this up to my superiors several times and then get reminded that we only have 1 link to the "internet" and am told there is no need. They don't understand that attacks can come from within and that even 1 outside connection is all it takes.
I've started putting some info together on possible situations and what should be done but but I'm not a security guru and could use some tips on possible senarios or some form of an example that might wake the management up in this company.
And for those that seem to think you are reading between the lines, thats not an open invite to attack.
Trying to be different, just like everyone else.
This previous "Ask Slashdot" discusses this, and deserves reference on the current subject.
Prevention of social engineering is critical to corperate and personal information security.
EveryDNS. Use it. It works.
AC's need not reply
Is that between 'Major Layoff Week' and 'Sexual Harassement Week' or is just after 'Clueless Management Week'??
--
Je t'aime Stéphanie
You can just imagine the script kiddies already getting busy, tracking down this company and taking advantage of the situation..
".. and now you can observe how easily the intruder is gaining access to our classified documents. But don't worry folks, remember; it's just a simulation!"
-- If no truths are spoken then no lies can hide --
I work on an army base. About a month ago, the entire Installation had our little practice excercises. It lasted one week, and we made it all the way up to Threatcon DELTA (The highest level of threat). During this time there was...
Bombings
Spies Peering into places
Security Checkpoints almost everywhere
Attacks on networks (software AND hardware)
Power outages
Phone outages
All simulated of course. From the Army's standpoint, it was a great training excercise. From the contractor's standpoint, it was a huge hassle, but I learned a lot regardless. That was my experience though.
And it's practiced every month or so. Luckily, I can usually keep track and bring flowers to lighten the mood. I find the best thing to do during crisis week is to just shut up and get out of the way. Let the experts handle most of the issues.
Dancin Santa
I accidentally opened the email. I didn't think IT would send me a virus.
at the failed dot-com i worked for, our employees were the double clicking, attachment getting, outlook using, microsoft sheep that graze the non-technical side the the IT industry. And as training, i as sysadmin would send bogus emails with VBS attachments that just open up a browser and took them to a page on our intranet that said: PLEASE DON'T OPEN VBS OR EXE FILES FROM YOUR EMAIL... then script would email me and i'd go have a chat with the person.... This worked wonderfully! People hate to be humilated...
.cig
For the last two years (at least, I've only been here 3), my company I work for has done presentations on security risks to our IT infrastructure. These presentations have been more of a "How do you prevent X from happening.", but do cater to informing the non-IT workers how to protect our data, and their own. I think they have been good wake up calls, even to me, and I work in an IT environment of the company. But certain issues that the regular Joe brings up in the meetings do get addressed and are taken very seriously by our Security folks. If your company isn't having meetings regarding IT security these days, you're missing the boat!
Attention! E-Mail coming in! It has an evil attachment; keep your eyes away from it! Don't open it, lest civilization as we know it ends forever!
Now try to find the delete key... press it... done. Ah, life can go on now.
Yeah, software companies should have "Stand Down" week, where the company goes off crisis mode for 7 days and people go home to sleep.
Someone you trust is one of us.
Along with allowing us to test our procedures once a year, in case of a real emergency, our critical systems can be reloaded at their facilities and brought back up until the neccesery repairs/reloads are performed here.
When you're doing a test, or an actual disaster recovery, they also have a full staff of experienced sysadmins to help. This is really valuable as even the most experienced sysadmin doens't get as much practice at disaster recovery as those guys do.
As you can imagine, services like this aren't cheap, but they are aimed at large companies that need this type of protection. They also have a whole host of other services they provide, mostly network monitoring services; this may include testing your site's ability to defend against a hacker attack, but I'm not sure. If you work for a large (or growing) company and don't yet have a company providing these services for you, I highly, highly recommend Comdisco.
--- Rectum?! Damn near killed em'! - Confucius
Then one day, we actaully had a fire, at about 6 pm. Three of us were containing it, and called the QuarterDeck (front office) to sound the alarm, which they did, except they announced that it was a drill!
Sooo, the people who usually run the drill (officers=managers) called the QuarterDeck and told them to cancel it because there was no drill for the day.
Needless to say, we spent a VERY LONG time on the phone before the QuarterDeck got the story right, and the fire crew finally arrived.
Moral of the story: Don't get so caught up in doing drills that you miss the actual fire!
Never never never smoke crack before geometry class!
Marcus Ranum gave an interesting talk on intrusion detection systems and security including physical threats at ALS last year. I'd also recommend secrets and lies by Schneier. It also takes an interesting look at physical security issues. As for crisis week the last one I can think of was Y2K but that wasn't really a mock up type thing. The only other crisis preparedness we were trained for was 'fire'.
Of course telling people that you're going to have a simulated crisis is not very effective at all. It just has to happen without warning or the workers are definitely going to be prepared.
:-)*
One day I came in to work and I was told that the CVS server went down. The support staff knew exactly when it went down because NetSaint sent messages to their phones.
I'm not normally support/admin, but I have experience in it so I jumped in to help. Here is what we did:
- Went to the console and tried to boot it up. No go
- I booted from a rescue disk and tried to boot it that way. Nope
- Tried to mount the partitions, found that the partition table was gone
- We then split into two different efforts: I mentioned gpart (guesses lost partition tables) and started running it with various options while the other team began rebuilding the server from backups
- gpart didn't work so I just partitioned it again with the original settings (I've done that successfully before on a home computer)
- That didn't work, but the replacement server was ready by then so we plugged it into the network
Once the backup server was up the head of development announced that he had replaced the CVS server's hard drive with a blank one early that morning.
We all wrote reports on what we did and, while we were pissed for a minute ("You WHAT!?!?!?"), the drill was determined to be a success.
I was freaked out mainly due to the fact that I volunteered to help out... Me and my big mouth
I sysadmin for a government research lab. You'd better believe every week is an IT crisis week. If it's not crackers in China looking for revenge for the embassy accident, it's some dumbfsck college kid trying to telnet past the routers or something.
Those aren't the crises, though (the routers keep those jerks out). The actual crises begin when the logfiles get too big to fit on the backup tape. Then I have to scrounge around to find more tapes, 'cause they won't let me buy any more on the government budget (yes dammit I'd raise my own grandmother's taxes if it means I have money to buy backup tapes), and then I have to decide whether the stuff currently on the tapes can be sacrificed for the holy cause (backups! backups always take priority!). This decision-making process usually requires some caffeine, and the single soda machine within reach charges a freaking dollar for a 20-oz bottle, so there's another twelve or thirteen dollars gone.
Don't talk to me about "planned" crisis week.
You cannot apply a technological solution to a sociological problem. (Edwards' Law)
This is one of the topics covered in the CISSP exam, I think the CISA also has it. Methods for disaster recovery, which are often ignored by many companies. Often I wonder how much a company has prepared for a disaster, via way of anything imaginable, hurricanes, fires, break-ins, etc.
;) ... For those with higher ranking positions I suggest you go out and get the "Information Management Handbook -- Tipton/Krauss" which has tons of informative information regarding safeguarding data, disaster recovery techniques, etc. Its one of the best books I ever bought.
Personally I think companies grow too fast and focus on growing, growing, growing, rarely stopping to take the time to implement measures against disaster recovery.
One of the things we do @ my place is once every other month we have a sit in with beers, pizza, etc., and focus on security via way of games. Why do you need a safe password is based on a guess your co-workers info to see how much we can gather by knowing them to see if we could guess their pw's, we also have a twist on Jeopardy where we use the names obtained from Attrition.org, and make a question about the company, so we could say "yes this company was owned this/last month" in order to make our workers aware of the risks involved on the `net'.
Its better than ramming security down their throats and constantly lecturing people. We also have little twists on dealing with all sorts of issues, voicemail management to avoid having pw's cracked, social engineering games, and makeshift scenarios where someone comes in to social engineer their way into information.
keep us on our toes
Want Root?
I'm glad the editors are posting stuff that piques their interest, but maybe it's time for a bit more editorial creativity? A vocab building class perhaps? Or maybe they should change the site name to "Slashdot: An interesting idea."
[ yes, this is offtopic. It's probably also flaimbait. But I, for one, think it's funny. Or at least intersting. hehehe ]
---
Ah! Another NT domain...
---
Book(n): Utensil used to pass time while waiting for the TV repairman
If you work in IT, you owe it to your company it your coworkers to practice this essential drill. I believe OSHA is considering making it mandatory for all businesses with more than 45mbps of bandwidth total (across all locations).
In order to perform an effective ping flood drill, you'll need every employee in the building to be equipped with the proper ping flood protective gear: two buckets, a mop, a snorkel, and a waterproof flashlight.
The drill should come as a surprise, so employees learn to react quickly and safely in the event of a real ping flood.
To begin the drill, a senior IT staffer should use the in-building paging system (if the building is not so equipped, a megaphone may be substituted).
Announce in a clear, calm voice, "Your attention please! We are currently experiencing a ping flood! All employees to ping flood response stations! This is not a drill!"
IT staffers should walk the building, making sure that employees are using their buckets and mops properly. The most common mistake non-IT staff makes when dealing with a ping flood is to not echo-reply properly. Unless you are practicing an IRC ping flood, people should *not* be saying "PONG!" This is a common panic response among employees, and part of the reason for the ping flood drill.
IT staff should also ensure that everyone in the building is mopping properly, and bailing the buckets out of the window, you may halt the drill. In the drill, of course, there will be no actual pings in the buckets, but it's important to have complete realism. Some buildings may have to have their windows knocked out with a chair or piece of computer equipment. The expense is well worth it in the event of an actual ping flood.
Although an actual ping flood can last for hours, you should limit a ping flood drill to no more than 45 minutes, as exhaustion may set in and render employees unable to deal with a real ping flood, should one occur immediately after the drill.
If you are in IT and not practicing this essential drill, you are negligent and irresponsible. If upper management refuses to allow you to stage ping flood drills, it is your moral obligation to do so anyway. When a real ping flood occurs, they will thank you for it.
Cheers
-b
If I wanted a sig I would have filled in that stupid box.
We don't bother with simulating an IT crisis, we simply allow people to log into the network and do their daily tasks.
main(i){(10-putchar(((25208>>3*(i+=3))&7)+(i ?i-4?100:65:10)))?main(i-4):i;}
Beyond the normal fire drills or chemical spills, a new addition was 'Attack on IT Infrastructure'.
They took away the coffeemaker?
must... have... caffeine... to... code...