Slashdot Mirror
Do You Have Your 'Crisis Week'?
pmbarth asks: "This week, the large company I work at is having a 'Crisis Week', where we simulate different types of problems, and have training on how to deal with them. Beyond the normal fire drills or chemical spills, a new addition was 'Attack on IT Infrastructure'. I was wondering how many other companies out there are actually training their non-IT employees on how to be aware of, and perhaps even counteract these types of issues?" It's an interesting idea, and one can't tell when an extra skill one learns on the job may come in use in a critical situation. Do other companies have something similar? Do you think such drills are particularly effective?
From my experience, "modern" industries and businesses don't take the time to worry about disasters. They either think it won't happen to them, or they'll just naturally handle it.
:)
Older industries, particularly manufacturing, do this fairly regularly. Of course, manufacturing as an activity has been going on for many years, so they've learned the hard way.
Power failure tests, system control failure tests (when your furnace burns itself up because you couldn't regulate it, and you spend 7M$ to rebuild it, I bet you work very hard not to make the same mistake in the future.)
As for IT-attacks, I haven't encountered that, but it makes good sense.
With all of the failure tests you should start with planned failure test, and eventually run "unplanned" (at least from the perspective of some of the employees) test to see how people respond. It's easy to remember where the emergency exits are in a maze of corridors and factory equipment when you know there's going to be a drill
... in any of the companies I've worked for but as a sysadmin, I've always had a policy of "self-regulation", staying late a couple of times a month and purposely taking servers out of service and seeing how long it takes me to get them up, restore files from backup, etc.
That said, when I quit as sysadmin due to "political" reasons I went on holiday for a week and when I got back it took 30 seconds to work out a server had been compromised and was being used to launch DOS attacks on www.microsoft.com. My replacement had spent a week replacing everyones NIC blaming the bad network performance on a faulty network card.
They should replace interesting with a dynamic value that users can moderate. The values should be:
Incredible
Interesting
Mediocre
Redundant
Tiresome
Completely Erronious
Dont Bother
...or at least generate email.
Is there any such week where IT doesn't get attacked? I thought that's why we got paid the big bucks? Outlook viruses, RIP left on on the firewall and gated started, bad firmware, cut fiber, BGP flaps, IIS worms, named worms, DOS attacks, need I go on?
kashani
- Why is the ninja... so deadly?
Gonna be kinda hard. There's only one Slashdot, and it's not sentient. Your revolution will have to wait a few years.
Could be illiterate journalists. We've seen enough of that situation, too.
It's when you pull the plug on your daily XFS CVS compilation and stress testing because someone finds it annoying that the machine reboots every night even if you plucked all the problems this was causing, namely the not-in-the-kernel-because-they-are-binary-only-cra p NVIDIA drivers weren't being recompiled along and the some people left some mp3 players open on a no-longer-existant NFS mount, and then they complain because now there are bugs showing up in that freaking old version of the kernel because the machine no longer rebooted (bugs which incidentally wouldn't have showed up if the recompiles had kept their pace) and someone claims they lost two whole freaking hours of valuable work because of this problem (which I'm sure wouldn't have happened if he hadn't heard of the problem in the first place) and then to calm people down you upgrade the kernel to a current version only to find that hardware thing that had been creeping on the box suddenly shows up in all its glory and some moron that thinks he knows (dick) about the problem because he reads LinuxToday starts giving his unrequested opinion about it. This while all you want to do is some real work. This is a crisis week, and it spans over 14 days and counting.
No, seriously, rant aside, I'd love to have something like this. Maybe people would actually learn to differentiate between it doesn't work *whine* and this particular part of the infrastructure has this particular problem, where that part of infrastructure is something more specific than the network and the problem goes beyond it doesn't do what I want (I'd pay for "it worked ok until I did this"). I'd also love if people were able to spot a problem ("hmm... I type ls --weird-option and it doesn't recognize it anymore") and report it instead of thinking "oh, the planets must be in the wrong position, I'll try again next week" and do zilch about it.
And while day dreaming, can people stop saying "could it be possible that foo and bar have a problem?" if a) they know there's a problem and b) they try to smooth it out because they know it pisses the hell out of me when I hear "foo and bar just don't work". If you are going to say that, say it without the sugar, please.
... then my opinion woul d be that your people aren't practised enough. If people always reacted badly in crisis situations even with training, then why would the military bother training? After all, it doesn't get much more high-tension than seeing the person next to you get blown to bits...
Granted, continuous training isn't always worth the time lost in it, but still, don't dismiss an idea like this out of hand just because.
I work for an architorture firm and we have crisis weeks every week. what we train for is the annual slow friday
Often I wonder how much a company has prepared for a disaster, via way of anything imaginable, hurricanes, fires, break-ins, etc.
Not at all, in a lot of cases companies don't even plan for problems they know they will have. A friend works at a major estate agent here in the UK, when recently they announced large branch closures. Firstly they announced these to the TV stations before their own employees, with a couple of weeks notice before closure. Secondly they failed to come up with any plan regarding what to do with the properties where they managed shorthold renting on behalf of others. Customers were left frantically phoning branches trying to find out this information for days before a decision was actually made.
If multi-billion pound companies can't prepare for obvious short term definite eventualities how can we expect them to have plans for remote eventualities.
Another problem is who reports these things. At a previous (Fortune 50)employer 50+ programmers were left for almost an entire day without power because no-one had the initiative to call in the problem, I (the summer student) eventually sorted out the problem (I have the unfortunate curse of feeling responsible for fixing other's mistakes).
On the subject of breakins, the same company managed to let theives get away with 60+ Sun Workstations and a room-sized MAINFRAME, with full security on watch at the time. How?, I dont know! As far as I can tell, gross incompetence of companies as wholes is rife.
Where I work, every week is crisis week.
---
I am having a crisis week where I work. When I came in on Monday my HD failed to spin up. Which was pretty bad considering it wasn't backed up.
I guess its time to RAID my desktop and pray my FS doesn't crash, since backing it up really isn't an option considering the amount of space on it. I would need to spend 10x on a backup solution. Of course I do backup the 'critical' data onto the server and on zips.
And you better believe they're gonna let you know, often and loud.
No boom today. Boom tomorrow. There's always a boom tomorrow. - Cmdr. Susan Ivanova
When in trouble,
When in doubt,
Run in circles,
Scream and shout.
(Author unknown)
Others, anyone?
The abbreviated Laws of Thermodynamics:
1)You can't win.
2)You can't break even.
"We all do no end of feeling, and we mistake it for thinking." -Mark Twain
My two weeks paid vacation are crisis weeks here.
I'm a doctor in a hospital and we have "crisis" simulations all the time. IT crises are the least of our problems.
:)
Crises to plan for:
"Boeing events" - ?400 injured -can the ambulance/ER/OR cope? How many doctors/nurses can you get to the hospital in 30 minutes on fridays night? I've seen this simulated with 50 patients in a rural hospital. They employed actors to test this, and used the results in to extrapolate to other hospitals. It was great fun, they didn't tell us it was a simulation until we arrived (talk about crying wolf). The ER was overflowing with actors in bandages pretending to die while a surgeon suggested to the study co-ordinator that he could amputate with a hammer since no other equipment was available...
"Power cuts" Power goes and the backup generators fail. Not cool if your in theatre with someone's chest open. The aneathetic machines have good batteries, but the lights have only minutes of power to clamp/close important things.
IT problems are a joke. When the do snapshots ie declare "at 0935 today all computers/phones (which means all xrays,labs results, communications) go, then survey what might have happened" the number of predicted deaths is relatively small. Sure its a major distruption, but not too many people die.
Seriously these "crises" are real issue for places like a hospital. Disasters happen. People MUST plan for them. IT disaster just don't feature compared to earthquakes, big plan crashes, total power failure etc...
Elvis
>They took away the coffeemaker?
We're talking attacks, and you start WW3..
//rdj
No one can understand the truth until he drinks of coffee's frothy goodness.
--Sheikh Abd-Al-Kadir, 1587
Ok, this may be off-colour, but I laughed my ass off! Thanx for a morning pick-me-up.
--
Later...
KangarooBox - We make IT simple!
Working in tech support for an ISP, we have escalation procedures defined for just about every fire/network/telecom outage. It's called a service level interruption procedure, or SLIP for short. We "practice" each and every day :)
-_-
-_-
No, its after "Sexual Harassment Week" right before "Voluntary Leave Week" and "Office Shooting Day".
Someone you trust is one of us.
We did something similar, mailed everyone a .VBS script that went into the registry and disabled their ability to run....VBS scripts.
;-)
It worked well until we did an AV procedure on our Mail store and it efficently deleted them all
~~~~~ BigLig2? You mean there's another one of me?
So of course, your post has been moderated as Interesting. :) You can win for losing, can you?
--Ty
We even thought of not working on fridays but the plan never worked :)
Call it crisis week de chinois. Hackers from a certain eastern nation threatened to attack US government sites over the course of a week. The IT folks a certain government academy fell all over themselves trying to prepare for an onslaught and in the process accidentally disrupted outside connectivity for a day and brought down the students' mail server for several days over finals. Of course, there were no attacks from the outside, but we sure had a week of crisis.
I beleive that the only reason the other people in a company are hired is to alert the IT department when there is an email problem.\ =\=\=\=\=\
=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=
Drill? Where I work, every week is a crisis week.
We all know that crap is king
Give us dirty laundry!
I wouldn't be surprised if these people would give out dangerous information to a clever social engineer.
.exe & .vbs files that was attached to your email. Do not use Outlook express."... And so on...
Yes, that is another issue you could make them aware of. Why not take a day, or even a few hours of a day to educate the emplyees in basic security awareness?
"Do not give out this information to ppl outside the company. Do not click on these
Having drills otoh... I dunno, how would you set them up?
--
"I'm surfin the dead zone
--
"I'm surfin the dead zone
In the twilight, unknown"
The same drill can also be applied to memory leaks. It does not work for core dumps.
The list of things was fairly involved: rebuild the servers from backups and from scratch; switch to an alternate pipe (isdn, dialup) if the primary failed; run through the restart procedures on the critical systems (necessary because you couldn't just power them back up); plus various repair procedures for filesystems, hardware, etc..
The rebuilding of the fileserver was particularly useful. In one case, we realized that though a system was emailing lots of "successful" messages, the backup was useless in recovering the system. I know restoring is the other half of backing up, but at this place, the job was so onerous that it was rarely actually performed.
great - illiterate sysadmins!
.oO0Oo.
they probably can't even read
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
well take a look at security focus
.oO0Oo.
www.securityfocus.com
And see all the holes and exploits available for your system. That should be a start
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
"Oh, and internet access for the whole network is through a single 64K ISDN line."
.oO0Oo.
you are so spoiled. I ran an ISP with 50 users on a 64k line for 6 months
The COUNTRY of Gambia has
International Bandwidth (Kbps): 128
from http://www3.sn.apc.org/africa/index.html
I picked Gambia because it had 64k the last time I looked. They must be well pleased with the upgrade!
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
there probably wouldn't be any electricty anyway but..
.oO0Oo.
put your server in a lead lined case of you're that concerned. People need a Quaker server in times of nuclear war
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
Be careful what you wish for, you just might... oh. Too late.
(Score: -1 Flamebait)
(Score: -1 True)
(Score: -1 Very True)
-- Is "Sig" copyrighted by www.sig.com?
We're doing an unscheduled major code realease to our website today. I think this qualifies as a "crisis week".
Any sufficiently well-organized community is indistinguishable from Government.
The sysadmin probably knows this already, unless he's busy getting somebody's coffee cup out of their CD-ROM drive. Stopping them to complain about it only delays action. :)
every single crisis didnt end up on my desk anyway. Unfortunately of the few people in our company, I am the only one with any technical know how it seems. If we did have a crisis day, it would just consist of me getting phone calls complaining that their email sends are going slow or they cant reach a website, just like an ordinary day. This is a crisis to these people.
fear
Time for some tasty Shiner Bock!
I know people who don't even keep backups!
hmm... for fun I enjoy launching DDoS attacks against 127.87.42.5
I work for an ISP and some of our coustomers practice this. The latest IIS worm owned their SQL server. Their soulition? Remove and reinstall the SQL server and NOT do testing on the rest of the network. Their about to find out the real meaning of this topic. Sigh
The slashdot 2 minute between postings limit: /.'ers since Spring 2001.
Pissing off hyper caffineated
1q2w3e4r5t6y7u8i9o0pqawsedrftgthyjukilo;p'azsxdcf
The only reason you were hired is so people have somebody to alert when there is an e-mail problem.
They were all hired to generate revenue.
Information wants to be anthropomorphized.
Everything is in a crisis as it is.
Seriously, think of the fun a pointy haired manager would have with this!
Check out the Vinny the Vampire comic strip
"It is a greater offense to steal men's labor, than their clothes"
...you mean that you actually have to set aside time for your crises? And that they only last a week?
Need a Perl programmer?
b&
All but God can prove this sentence true.
I'll give a real crisis--I work in a Microsoft shop. Crisis is the damn COM+ $@&# giving me a bad memory error everytime I try to shut it down. When you have to reboot everytime you rebuild a web app to test it, that's serious downtime.
The Blaster Master Fighting for Truth, Justice, and Evil Pie since 1979
oh ya. Typing one charactar at a time with a pen. Have fun!!!
Sanchi
"They said we couldn't do it [Athlon]... but we built it, we shipped it... and we didn't have to recall it." Rich Heye
Yeah. That was the week IT took away my Unix-based mail and made me switch to Microsoft Outlook. Now they warn me not to open attachments that could flim-flam Microsoft Sillyware.
Thanks for the link, I must stay on the lookout for "script kitties," scratch0ring my sofa :-)
no sig.
Yeah, I have my crisis week. They're called finals. They come at the end of every semester. But seriously, I see the value in training employees how to deal with a chemical spill if your business is to work with hazardous materials. But what good will it do to take the sales department away for a week of computer skills that they don't understand or will never use. Don't forget to unplug the cash register on the way out. A better way would be to keep a constant and open dialog between departments so that "Crisis Week" doesn't become a paid in office holiday for some and a 1/4th reduction in commisions for others.
I've hit Karma 50 and gotten a Score:5, Troll... I win!
Awesome idea...I think I will start doing that!
Jaysyn
There is a war going on for your mind.
ditto....
Jaysyn
There is a war going on for your mind.
We have a disaster recovery day once a year where a rep from every team in the division goes off-site and has to conduct tests to make sure they are fully functional without any links to home office (we pull the plugs on the routers). It can be quite an experience every year.
Of course, during our last real disaster (the Chicago Flood), our disaster site was knocked an hour before our home office. Guess they need to be more than a mile apart.
Viv
-----------
Viv
Gmail invites for ip
There are a lot of dot-com employees who have gone through real crisis weeks recently where everything in their company failed, especially finances.
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
If your in the IT industry anyways :)
The first thing Non-IT people should do when they know that the IT structure is being attacked is to NOT go to the sysadmin and say, "My Internet's broken."
Check out Althea for a stable IMAP email client for X. Now with SSL!
> Ya gotta love irony.
coincidence
Last time I checked DLT4 would hold 40 gb uncompressed and ~80 compressed. Thats 187.5 tapes for each full backup. I think they need something a little more dense, personally.
This is a bowel disruptor, and you are just full of shit. - Spider Jerusalem
One crisis week a year, usually around tax time
--
Kiro
They're aware of the issues here. And the non-IT staff counteract as follows....
Phone:RING!!!
Me:What? Yes, I know you can't get your mail, the mail server's down.
Phone: blah,blah...
Me:It'll be up when it's back up, now go away.
lather, rinse, repeat 100X
/*drunk.. fix later*/
--brian
Clue: if the EMP fried your computers, it fried the computers at the store as well.
I was originally hired as a web developer, and walked out on them after one week they told me they wouldn't need me after a couple of months, and then the next week they asked me to prepare and teach a CLASS so that I could show the others how to use FP 2000.
How about that? "We don't need you, but can you show everyone how you do your job before you leave."
...All I can say is that my life is pretty strange...
In this economy, the last thing I will be worried about are some sort of IT drills for my non-IT employees. I would focus on what earns the bread and butter for my company and deal with scenarios on a case-by-case basis. We deal with breakdowns all the time, and don't give a second thought about it. why worry about this one specially?
Now, if we could just convince those knuckle-draggers down in administration who insist on sending everything with a message like "Details on the last Board Meeting - big changes!" and then send it out in MS Word format, complete with macros.
No amount of logic seems to convince these people that "Copy" and "Paste" are a good idea.
political_news.c: warning: comparison is always true due to limited range of data type
Forget the old 1950's nuclear drills where our parents all had to hide under their desks when the air-raid siren sounded. We all know that the true threat from a nuclear attack is the EM pulse, which fries all active electronics in a hundred miles radius! If a nuclear bomb hit NJ tomorrow, it would be 2 minutes before I would be at the computer store to stock up on PC's to replace all the fried ones! Maybe a few extra to trade for valuable life-giving water in the post-apocalyptic wasteland that was once called "America"...
Also, we could stockpile large amounts of vitamins C and E to protect us from all the free radicals smashing apart my fragile cells. I'm submitting the proposal to my supervisor straight away! - Kengineer
Where I currently work we have 35 computers, through 4 hubs going out on a single 56k modem. And we use Wingate!!!! To top it off part of that network is Coax cable, that only works if you stand on one foot in the right hand conrner of the top office (You think I'm joking). At least you run an ISDN.
Acaila
Growing Old is Inevitable; Growing Up is Optional.
i take it you are in the MOB. i was there for 3 years, and know how all that was. it's really fun trying to splice wires in MOPP 4.
"I really must respectfully protest, your reverence," said the acolyte. "We have practised for just such an emergency as -- "
"Yeah, I know all about practising procedures for emergencies," said Lu-Tze. "And there's always something missing."
"Ridiculous! We take great pains to--"
"You always leave out the damn emergency."
(from Thief of Time by Terry Pratchett)
All I ask is a warm bed, a kind word, and UNLIMITED POWER
> In 37 hours, 23 minutes, and 8 seconds, I am gonna be... so fucking drunk
Yeah - I suppose thats a great way to do the finals. ;) Hope you're taking an abstarct art. ;) Interpretive dance, perhaps? :) :)
Nah, it's the moodiness that kills me...
I completely don't understand you? What? What are you talking about? Oh, I see... I'll be back.
*returns an hour later*
I got you some flowers. Let's watch tv together.
Dancin Santa
Interesting story, but I got to it by clicking underneath the Clipper story...
Everyday, I'm part of the lucky ones who actually leave a fixed time.. However, when I'm gone, there is nobody to run the network. Driving home, one day, I start getting alerts via sms complaining that a network link is down, blah blah blah.. Not a problem, I'll simply reroute the traffic when I get home.. right? Next page I get, 2 minutes later, our PDC server has been rebooted, then another page, PDC server has been rebooted again, another page, firewall server has been rebooted, and it just doesn't stop, seems like everything is rebooting.. Of course, by now, I figured that someone who decided to to play sysadmin and thought he could simply reboot half of the servers just so that his internet would work again. So I call the office, which breaks the relaxing mood of being stuck in traffic without any laptop or pda, and gently tell that person to stop f***ing around with the servers and he'll have to live without it. Now, that's a panic moment, the moment when the user thinks he can fix a network problem by rebooting, and ruining my beautiful uptime stats.
what would your company do if the power plant near you blew up and all of your power went out, Just as the Damn near your office also colaped. Who would know about the rushing water of doom coming to take your office with it??
DaveWpW
Because our own guys tend to set off the IDS systems a lot themselves (mostly tripwire), we get daily (especially at 3AM) training on how to combat IT infrastructure security scenerios. =)
I think you need to flash your brain's firmware.
Every couple of months we have a 'Normal Week'.
We attempt to simulate deadlines being met, documentation standards being adhered to, workloads being realistic and sensible management,
I feel this is an invaluable exercise should we be subject to an outbreak of normality at some stage in the future.
However unlikely it may be, it makes sense to be prepared.
<-- You are here.
Hehe, during the easter vacation, I had to go into school to work on the yearbook. We're only a small international school in the South of France, so I brought my own computer to speed up the pace of work. BAD idea. For some reason, the Win NT server decided to pack up its bags and go home, and as the yearbook team (a group of 4) were the only people there, we got the blame. OUCH
Those darned teachers, maybe they should learn to actually use a computer before they try to administer a network running them. I mean, who connects 50 computers together using only 10 Base-T hubs? No switches in sight. Oh, and internet access for the whole network is through a single 64K ISDN line.
Pity this is France, people don't get sued or thrown out of their jobs for stuff like this here.
I bet that a lot of companies don't do nightly backups, rotate the tapes, verify the tapes, maintain the tapes, store them off site, and have up to date security procedures to protect their data. How many of them haven't even updated to the latest version of BIND? Of course, if you're not going to do these things, you might as well practice what you're going to do in case of a disaster. But where as our company will be kicking on generators, and restoring redundant servers at a second facility, most companies will be throwing the dice to see if they will be going out of business as the result of a disaster. I think that you should actually perform, at least once a year, your emergency recovery procedures, just to see if they work.
I have my "Crisis Week." It's called, "finals." In 37 hours, 23 minutes, and 8 seconds, I am gonna be... so fucking drunk
--
I think there is a world market for maybe five personal web logs.
So, when they simulate the network hacking attack crisis, just run down the hall to the data center. Hit the big red switch on the wall.
You will see that the network hacking attack exercise will be successfully ended, and everyone will begin the simulation of what to do when the data center loses power.
If tits were wings it'd be flying around.
Right on, my brother. It's been a couple of years (gee, time flies) but I was trained that it was always better to know where to look things up then to try and know everything. STAN-EVAL just ate that shit up. I wish we could have done more with the continuity binders though, we were always so busy putting out fires and holding the office together we didn't have much time for things, like proper docs and recurring training.
I was a 1W051 (Weather Observer) and every unit I was at always seemed to be running as fast as they could just to stay in one place. I was never able to make much progress trying to make the place better, it was very frustrating and is one of the reasons that I didn't reenlist.
Of course I'm in the same boat with my current job, running as fast as I can just to keep from being swamped. The difference is that I have hope in my current job that it will get better sooner rather than later. Also in my current job I really can make a difference and make things better for myself and the others around me. Sometimes it helps being small.
-- Remember: Wherever you go, there you are!
Then I got to the bottom of the page, and saw this quote:
Your love life will be... interesting.
Ya gotta love irony.
--
1. CEO Fired [How fast can you find a new one?]
2. CFO Fired [How fast can you forge his signature all those PO's you've been wanting.]
3. Stock Devaluation [Speed selling]
4. Stock Devaluation at a dot com [Resume update]
Get a DLT (digital linear tape) and create a cron script to run nightly/weekly according to your discretion. DLT's have dropped since there are other alternatives, so their inexpensive (considering you have a 15tb set up) or you could always have a Clariion purchased for this. Even a Netapp using bzip2 on a sys will do more justice than people realize. Many companies don't have any idea how important a backup plan is until the shit hits the fan and they're shit out of luck.
Its always good to have at least 90 days worth of backups in case something may have been corrupted, whats more is the storage of that data. You don't want to just leave it lying around. Consider renting small storage space for a monthly dump of the tapes or keep em locked up in a secure place such as a safe, or encrypt the tapes so the average joe can't read it should they get their hands on them (the tapes)
I hate even thinking about the idea of going to a new company, and having to deal with this since its a painstaking task, but once its done, its all a matter of following up on things, and making it part of daily/weekly/monthly work. It gets easier once you've done it and gotten it over with.
Want Root?
---
...is identifying how we can pass the crisis-response knowledge on to the next generation of employees. One focus in US military exercises is not only on how each situation is dealt with, but how each person knows what to do in the first place. Rather than simply train each person from scratch then hoping they remember what to do under every single set of circumstances, the US Air Force often creates multi-purpose response checklists and "continuity" binders/folders that contain everything from basic response overviews to detailed information on how to deal with various problems. Saying "I'm not sure but the answer is written right *here*" is very nearly as good as having the answer memorized especially if the answer is available to everyone in the organization.
It's a huge pain in the a$$ to prepare this material but it helps insure that there's always SOMEONE around who knows what to do and there is a source of info to check when the crap hits the fan. A little investment in time to create these documents pays off in a big way, one incident at a time. These are living documents instead of regulations, so they are continually subject to improvement or even disposal if they become obsolete or something better comes along.
This kind of thing ought to work in the corporate world especially in a company where there is a measure of procedural inertia that carries on regardless of who is holding down each particular job.
Sounds like a lot of folks (at least in charge) with too little to do. The environment I'm in is constant crisis. I'd sign on for a 'NonCrisis Day' in a heartbeat.
I find that massaging the back right under the ribs also helps. Also, there is a program for the palm pilot that is helpful in keeping track when the next one will occur.
-no broken link
I for one am tired of all these interesting stories. I demand boring stories on Slashdot!= \=\=\=\=\=\
=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\
Oooh good idea. I know what I'm coding up tomorrow.
Vintage computer games and RPG books available. Email me if you're interested.
Considering how many times I've seen our non-technical employees warn the company about hoaxes and spread email viruses I wouldn't be surprised if these people would give out dangerous information to a clever social engineer.
Rats would be more funny if they could fart.
Actually, my company has a daily drill.
Someone codes something good and shouts something like 1 4M S000 13373 D00DZ!!11 (or something else in 13373 speak), then we beat the ever living crap outta them.
I'd like to see a script kiddie pop their head up in my company.
BTW - This article sounds like a preface to a new BOFH.
Good quote, too many chars. Seriously, the slashdot 120 char limit sucks!
Those people are merely sharing how common crisis mode is.
Even if it is just due to managers believing that you can do with half the staff, or half the hardware, or half the budget you need to get the results they want.
Maybe it was a silly question to begin with, but it sure seems like whoever had the 150 moderator points had a problem with that much unexpected opinion.
Check out the Vinny the Vampire comic strip
"It is a greater offense to steal men's labor, than their clothes"
Well, crisis has a little different meaning in the Milittary. Try codeing in MOPP-4 (full chem) gear. Its not fun. Yes we have the regular bomb threat practice and the runs Tornado Shelter, both practice and real, I'm stationed At Tinker AFB in Oklahoma. Its a very, for lack of a better word, "Interesting" Job.
Sanchi
"They said we couldn't do it [Athlon]... but we built it, we shipped it... and we didn't have to recall it." Rich Heye
I was thinking though, technically most mail that I get falls under the category of "unexpected" :)
dynamo
Well, I have to admit, it's solid in theory, but my experience ( both electronically and physically ) is that folks like to loose all their "cool" in situations of a critial nature ( SysAdmins not excluded ). People like to run during fire alarms ( when they're fot real ), tornados, when their the targets of crackers, etc. I guess drilling probably does give one person a very cushy job though.
My "original" and "personal" expressions go here.
Most likely running under Netware...
political_news.c: warning: comparison is always true due to limited range of data type
"I wouldn't have that code finished until next week, I'm practicing having crisis this week."
OR
"NOooooooooo! this is the third time this week I've split bong water on my keyboard."
--
M0571y H@rml355.
I have brought this up to my superiors several times and then get reminded that we only have 1 link to the "internet" and am told there is no need. They don't understand that attacks can come from within and that even 1 outside connection is all it takes.
I've started putting some info together on possible situations and what should be done but but I'm not a security guru and could use some tips on possible senarios or some form of an example that might wake the management up in this company.
And for those that seem to think you are reading between the lines, thats not an open invite to attack.
Trying to be different, just like everyone else.
This previous "Ask Slashdot" discusses this, and deserves reference on the current subject.
Prevention of social engineering is critical to corperate and personal information security.
EveryDNS. Use it. It works.
AC's need not reply
Is that between 'Major Layoff Week' and 'Sexual Harassement Week' or is just after 'Clueless Management Week'??
--
Je t'aime Stéphanie
You can just imagine the script kiddies already getting busy, tracking down this company and taking advantage of the situation..
".. and now you can observe how easily the intruder is gaining access to our classified documents. But don't worry folks, remember; it's just a simulation!"
-- If no truths are spoken then no lies can hide --
I work on an army base. About a month ago, the entire Installation had our little practice excercises. It lasted one week, and we made it all the way up to Threatcon DELTA (The highest level of threat). During this time there was...
Bombings
Spies Peering into places
Security Checkpoints almost everywhere
Attacks on networks (software AND hardware)
Power outages
Phone outages
All simulated of course. From the Army's standpoint, it was a great training excercise. From the contractor's standpoint, it was a huge hassle, but I learned a lot regardless. That was my experience though.
And it's practiced every month or so. Luckily, I can usually keep track and bring flowers to lighten the mood. I find the best thing to do during crisis week is to just shut up and get out of the way. Let the experts handle most of the issues.
Dancin Santa
I accidentally opened the email. I didn't think IT would send me a virus.
at the failed dot-com i worked for, our employees were the double clicking, attachment getting, outlook using, microsoft sheep that graze the non-technical side the the IT industry. And as training, i as sysadmin would send bogus emails with VBS attachments that just open up a browser and took them to a page on our intranet that said: PLEASE DON'T OPEN VBS OR EXE FILES FROM YOUR EMAIL... then script would email me and i'd go have a chat with the person.... This worked wonderfully! People hate to be humilated...
.cig
For the last two years (at least, I've only been here 3), my company I work for has done presentations on security risks to our IT infrastructure. These presentations have been more of a "How do you prevent X from happening.", but do cater to informing the non-IT workers how to protect our data, and their own. I think they have been good wake up calls, even to me, and I work in an IT environment of the company. But certain issues that the regular Joe brings up in the meetings do get addressed and are taken very seriously by our Security folks. If your company isn't having meetings regarding IT security these days, you're missing the boat!
Attention! E-Mail coming in! It has an evil attachment; keep your eyes away from it! Don't open it, lest civilization as we know it ends forever!
Now try to find the delete key... press it... done. Ah, life can go on now.
Along with allowing us to test our procedures once a year, in case of a real emergency, our critical systems can be reloaded at their facilities and brought back up until the neccesery repairs/reloads are performed here.
When you're doing a test, or an actual disaster recovery, they also have a full staff of experienced sysadmins to help. This is really valuable as even the most experienced sysadmin doens't get as much practice at disaster recovery as those guys do.
As you can imagine, services like this aren't cheap, but they are aimed at large companies that need this type of protection. They also have a whole host of other services they provide, mostly network monitoring services; this may include testing your site's ability to defend against a hacker attack, but I'm not sure. If you work for a large (or growing) company and don't yet have a company providing these services for you, I highly, highly recommend Comdisco.
--- Rectum?! Damn near killed em'! - Confucius
Then one day, we actaully had a fire, at about 6 pm. Three of us were containing it, and called the QuarterDeck (front office) to sound the alarm, which they did, except they announced that it was a drill!
Sooo, the people who usually run the drill (officers=managers) called the QuarterDeck and told them to cancel it because there was no drill for the day.
Needless to say, we spent a VERY LONG time on the phone before the QuarterDeck got the story right, and the fire crew finally arrived.
Moral of the story: Don't get so caught up in doing drills that you miss the actual fire!
Never never never smoke crack before geometry class!
Marcus Ranum gave an interesting talk on intrusion detection systems and security including physical threats at ALS last year. I'd also recommend secrets and lies by Schneier. It also takes an interesting look at physical security issues. As for crisis week the last one I can think of was Y2K but that wasn't really a mock up type thing. The only other crisis preparedness we were trained for was 'fire'.
Of course telling people that you're going to have a simulated crisis is not very effective at all. It just has to happen without warning or the workers are definitely going to be prepared.
:-)*
One day I came in to work and I was told that the CVS server went down. The support staff knew exactly when it went down because NetSaint sent messages to their phones.
I'm not normally support/admin, but I have experience in it so I jumped in to help. Here is what we did:
- Went to the console and tried to boot it up. No go
- I booted from a rescue disk and tried to boot it that way. Nope
- Tried to mount the partitions, found that the partition table was gone
- We then split into two different efforts: I mentioned gpart (guesses lost partition tables) and started running it with various options while the other team began rebuilding the server from backups
- gpart didn't work so I just partitioned it again with the original settings (I've done that successfully before on a home computer)
- That didn't work, but the replacement server was ready by then so we plugged it into the network
Once the backup server was up the head of development announced that he had replaced the CVS server's hard drive with a blank one early that morning.
We all wrote reports on what we did and, while we were pissed for a minute ("You WHAT!?!?!?"), the drill was determined to be a success.
I was freaked out mainly due to the fact that I volunteered to help out... Me and my big mouth
I sysadmin for a government research lab. You'd better believe every week is an IT crisis week. If it's not crackers in China looking for revenge for the embassy accident, it's some dumbfsck college kid trying to telnet past the routers or something.
Those aren't the crises, though (the routers keep those jerks out). The actual crises begin when the logfiles get too big to fit on the backup tape. Then I have to scrounge around to find more tapes, 'cause they won't let me buy any more on the government budget (yes dammit I'd raise my own grandmother's taxes if it means I have money to buy backup tapes), and then I have to decide whether the stuff currently on the tapes can be sacrificed for the holy cause (backups! backups always take priority!). This decision-making process usually requires some caffeine, and the single soda machine within reach charges a freaking dollar for a 20-oz bottle, so there's another twelve or thirteen dollars gone.
Don't talk to me about "planned" crisis week.
You cannot apply a technological solution to a sociological problem. (Edwards' Law)
This is one of the topics covered in the CISSP exam, I think the CISA also has it. Methods for disaster recovery, which are often ignored by many companies. Often I wonder how much a company has prepared for a disaster, via way of anything imaginable, hurricanes, fires, break-ins, etc.
;) ... For those with higher ranking positions I suggest you go out and get the "Information Management Handbook -- Tipton/Krauss" which has tons of informative information regarding safeguarding data, disaster recovery techniques, etc. Its one of the best books I ever bought.
Personally I think companies grow too fast and focus on growing, growing, growing, rarely stopping to take the time to implement measures against disaster recovery.
One of the things we do @ my place is once every other month we have a sit in with beers, pizza, etc., and focus on security via way of games. Why do you need a safe password is based on a guess your co-workers info to see how much we can gather by knowing them to see if we could guess their pw's, we also have a twist on Jeopardy where we use the names obtained from Attrition.org, and make a question about the company, so we could say "yes this company was owned this/last month" in order to make our workers aware of the risks involved on the `net'.
Its better than ramming security down their throats and constantly lecturing people. We also have little twists on dealing with all sorts of issues, voicemail management to avoid having pw's cracked, social engineering games, and makeshift scenarios where someone comes in to social engineer their way into information.
keep us on our toes
Want Root?
I'm glad the editors are posting stuff that piques their interest, but maybe it's time for a bit more editorial creativity? A vocab building class perhaps? Or maybe they should change the site name to "Slashdot: An interesting idea."
[ yes, this is offtopic. It's probably also flaimbait. But I, for one, think it's funny. Or at least intersting. hehehe ]
---
Ah! Another NT domain...
---
Book(n): Utensil used to pass time while waiting for the TV repairman
If you work in IT, you owe it to your company it your coworkers to practice this essential drill. I believe OSHA is considering making it mandatory for all businesses with more than 45mbps of bandwidth total (across all locations).
In order to perform an effective ping flood drill, you'll need every employee in the building to be equipped with the proper ping flood protective gear: two buckets, a mop, a snorkel, and a waterproof flashlight.
The drill should come as a surprise, so employees learn to react quickly and safely in the event of a real ping flood.
To begin the drill, a senior IT staffer should use the in-building paging system (if the building is not so equipped, a megaphone may be substituted).
Announce in a clear, calm voice, "Your attention please! We are currently experiencing a ping flood! All employees to ping flood response stations! This is not a drill!"
IT staffers should walk the building, making sure that employees are using their buckets and mops properly. The most common mistake non-IT staff makes when dealing with a ping flood is to not echo-reply properly. Unless you are practicing an IRC ping flood, people should *not* be saying "PONG!" This is a common panic response among employees, and part of the reason for the ping flood drill.
IT staff should also ensure that everyone in the building is mopping properly, and bailing the buckets out of the window, you may halt the drill. In the drill, of course, there will be no actual pings in the buckets, but it's important to have complete realism. Some buildings may have to have their windows knocked out with a chair or piece of computer equipment. The expense is well worth it in the event of an actual ping flood.
Although an actual ping flood can last for hours, you should limit a ping flood drill to no more than 45 minutes, as exhaustion may set in and render employees unable to deal with a real ping flood, should one occur immediately after the drill.
If you are in IT and not practicing this essential drill, you are negligent and irresponsible. If upper management refuses to allow you to stage ping flood drills, it is your moral obligation to do so anyway. When a real ping flood occurs, they will thank you for it.
Cheers
-b
If I wanted a sig I would have filled in that stupid box.
We don't bother with simulating an IT crisis, we simply allow people to log into the network and do their daily tasks.
main(i){(10-putchar(((25208>>3*(i+=3))&7)+(i ?i-4?100:65:10)))?main(i-4):i;}
Beyond the normal fire drills or chemical spills, a new addition was 'Attack on IT Infrastructure'.
They took away the coffeemaker?
must... have... caffeine... to... code...