Slashdot Mirror
Planning for Survivable Networks
Priscilla Oppenheimer writes "Annlee A. Hines' book Planning for Survivable Networks, is quite a page-turner. Yes, that's surprising for a technical book, but I found it to be true. I was fascinated by the stories of real companies (Lehman Brothers, the Wall Street Journal, and others) that survived the 9/11 attack and resumed business quickly. There are also stories from other disasters, both man-made and natural, and information on companies that were not able to quickly resume business. The author summarizes the stories with explanations of what went right and what went wrong, with advice on developing your own disaster recovery plan." Read on for the rest of her review.
Planning for Survivable Networks
author
Annlee A. Hines
pages
320
publisher
Wiley Publishing, Inc.
rating
10
reviewer
Priscilla Oppenheimer
ISBN
047123284X
summary
Designing networks that can recover from natural and unnatural disasters
As Hines explains, Lehman Brothers had headquarters in Tower 1, as well as in 1,2,3 World Financial Center (across the street from the WTC towers). Lehman moved to a backup recovery location and performed cash-management functions the same day as the attack. The company was online trading fixed-income securities by the next day. They had 400 traders online when the NYSE reopened Monday, 9/17.
The Wall Street Journal (WSJ) published the story of its own recovery and Hines used that as source material for her book. WSJ had an extensive disaster recovery plan, based on lessons learned in the 1990 power blackouts in New York. After the blackouts and a subsequent fire in the emergency generator room, WSJ decided that it would never again depend on just one location being operational. WSJ opened other offices that could perform some of the necessary tasks to bring out a paper. Geographical diversity of resources seems to be a key to success.
When the 9/11 terrorists attacked the buildings across the street from WSJ's main offices, senior managers called for an evacuation, knowing that they could still produce the paper. The Wall Street Journal managed to publish a full newspaper with eyewitness accounts of the tragedy the next day.
Hines' writing is easy to follow. Although she delves into some technical details, with the requisite IP and TCP header depictions that you will find in so many networking books, the book can easily be read by managers and business people. Planning for Survivable Networks has many factual tidbits about disasters of all sorts, and although these are interesting, the primary benefit of reading the book is to gain an understanding of the characteristics of companies that sustained business after a disaster compared to companies that did not.
As Hines says, the companies that survived disasters all had disaster recovery plans in place. The plans were activated by decisive managers, who also promptly got their people out of harm's way. (If people don't survive, it won't matter much if systems survive.) Another point she makes is that the managers had to be adaptable. Not everything went according to plan, and it shouldn't be expected that it will.
The book opens with the author being rocked by a terrorist-caused explosion herself. She wasn't present for the 9/11 attackers. Rather, the bombing she survived occurred at Ramstein Air Base in Germany, 20 years before. A retired Air Force officer, she has dealt with threats all over the world for many years. Her direct command and control experience has taught her many lesson, which she shares with the reader in Planning for Survivable Networks.
Probably one of the most useful chapters, Chapter 11, "The Business Case," offers advice on presenting to management a case for a network continuity plan. According to the back cover, Hines has taught economics at a community college, and I would say that experience helped her explain the many costs involved in having a disaster recovery plan, including fixed, variable, direct, and indirect costs. She also explains the expected value of having a plan and how to sell that to management.
I recommend this book as an informative discussion of how companies can ensure business and technology continuity in a world with hackers, terrorists, natural disasters, and human error. It's a practical book, but also a surprisingly uplifting book, considering its technical content. I truly enjoyed reading about the adaptable human spirit that enabled managers and workers to keep their businesses going after the 9/11 attacks.
As Hines explains, Lehman Brothers had headquarters in Tower 1, as well as in 1,2,3 World Financial Center (across the street from the WTC towers). Lehman moved to a backup recovery location and performed cash-management functions the same day as the attack. The company was online trading fixed-income securities by the next day. They had 400 traders online when the NYSE reopened Monday, 9/17.
The Wall Street Journal (WSJ) published the story of its own recovery and Hines used that as source material for her book. WSJ had an extensive disaster recovery plan, based on lessons learned in the 1990 power blackouts in New York. After the blackouts and a subsequent fire in the emergency generator room, WSJ decided that it would never again depend on just one location being operational. WSJ opened other offices that could perform some of the necessary tasks to bring out a paper. Geographical diversity of resources seems to be a key to success.
When the 9/11 terrorists attacked the buildings across the street from WSJ's main offices, senior managers called for an evacuation, knowing that they could still produce the paper. The Wall Street Journal managed to publish a full newspaper with eyewitness accounts of the tragedy the next day.
Hines' writing is easy to follow. Although she delves into some technical details, with the requisite IP and TCP header depictions that you will find in so many networking books, the book can easily be read by managers and business people. Planning for Survivable Networks has many factual tidbits about disasters of all sorts, and although these are interesting, the primary benefit of reading the book is to gain an understanding of the characteristics of companies that sustained business after a disaster compared to companies that did not.
As Hines says, the companies that survived disasters all had disaster recovery plans in place. The plans were activated by decisive managers, who also promptly got their people out of harm's way. (If people don't survive, it won't matter much if systems survive.) Another point she makes is that the managers had to be adaptable. Not everything went according to plan, and it shouldn't be expected that it will.
The book opens with the author being rocked by a terrorist-caused explosion herself. She wasn't present for the 9/11 attackers. Rather, the bombing she survived occurred at Ramstein Air Base in Germany, 20 years before. A retired Air Force officer, she has dealt with threats all over the world for many years. Her direct command and control experience has taught her many lesson, which she shares with the reader in Planning for Survivable Networks.
Probably one of the most useful chapters, Chapter 11, "The Business Case," offers advice on presenting to management a case for a network continuity plan. According to the back cover, Hines has taught economics at a community college, and I would say that experience helped her explain the many costs involved in having a disaster recovery plan, including fixed, variable, direct, and indirect costs. She also explains the expected value of having a plan and how to sell that to management.
I recommend this book as an informative discussion of how companies can ensure business and technology continuity in a world with hackers, terrorists, natural disasters, and human error. It's a practical book, but also a surprisingly uplifting book, considering its technical content. I truly enjoyed reading about the adaptable human spirit that enabled managers and workers to keep their businesses going after the 9/11 attacks.
You can purchase the Planning for Survivable Networks from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
first post!
lsaib ei afo boea jblid bie.
trollin'
woohoo!
go read a book or something
at first posts.
My book on this subject is one page long.
Page 1: Don't let Slashdot link to you.
â¦as I would prefer death to running a network these days.
from the article:
"probably one of the most useful chapters, chapter 11, "the business case," offers advice on presenting to management a case for..."
in light of the current economy, i find this particular chapter arrangement particularly funny.
ed
In the wake of the tragedy of 9/11, what lessons can a System Administrator learn? This article highlights the bitter lessons of 9/11, and helps you prepare for such disasters.
1) Have a good, tested backup plan.
There are companies that were housed in the WTC that went out of business on 9/11, and there were companies that were housed in the WTC that were open for business in a day or two. A safe, tested backup policy can make all the difference. Be sure to have adequately sized tape drives for backup. Come in on a weekend or two and recreate a server from backup tapes, so you know how long it takes. Document everything, and prepare for growth. If you find that you've outgrown your backup capacity, upgrade now while the horror of 9/11 is still fresh on those controlling the pursestrings.
1a) Have offsite backup.
Keeping backup tapes in the storage closet is no longer good enough. You need backup tapes stored offsite. The ideal offsite storage is an abandoned coal mine or limestone mine in Apalachia (the locals work cheap, the state governments are willing, and the locals aren't sophisticated to know what to do with 4mm dat tapes). If your company doesn't have the cash for that, offer to move into a bigger house or apartment, with one room being locked and fireproofed to store backup tapes. Of course, you will need a modest stipend to pay for your larger quarters.
1b) Offsite backup machines
While many of the WTC affected companies were donated PC's by the PC manufactuers, you may not have that luxury, especially if your disaster is not photogenic enough, or terrorist related. It makes sense to have a few high end workstations and a server in your house, for immediate use in case of disaster (until then, can you say LAN party?).
2) Have a disaster plan
Think of the worst things that can happen to your company, and work out a response. Powe loss, flooding, plain crash, riots, anything that will disrupt your company.
If your company doesn't have a disaster preparedness officer, get one. And who else has experience with sudden failures and crashed, but you? Once you've assumed the mantle of Disaster co-ordinator, brainstorm every possible disaster (Tom Clancy noel's become a business expense), and work out a response. Have monthly drills, so the peons get used to your authority.
Along with this added responsibility, you should lobby for a larger salary, and a better sounding title. May I suggest Vice-President for Disaster Planning.
3) Be aware of who you hire
Your biggest danger is from a terrorist on the inside, be careful of whom you hire. It would be nice to only hire Americans, but in the land where Wrestling biographies are on the top ten bestsellers list, Americans who can handle a computer are becoming rarer and rarer. The few that are compentent with computers tend to have attitudes, if not allowed to waste 4 hours a day surfing the web for pornography and weblogs, they get whiny and cantankerous. So, I give you a rundown of various foriegners; who to hire, who to avoid.
Europeans: Western Europeans have been spoiled by their socialist nanny-state, and will be loathe to work more than 35 hours a week. Eastern Europeans are very smart, and work cheap, but usually come with organized crime ties. What you make up on cheap salaries you lose on disappearing laptops. If you have a good source of old laptops, a Russian may be your best choice.
Arabs: Have the lessons of 9/11 taught you nothing? Al Qeada sleeper agents have been in the US for years, do you want to take the chance that the cheap camel jockey you hire is possibly a terrorist? Point any Arabic applicants down the road to the Yemenese-owned mini-mart, and save a copy of their resume for the FBI.
Asians: You're getting better. They work cheap, they have a good, tireless antlike work ethic, they're smart, and they're
"Disaster recovery" is the biggest load of shit. If I had been a 9/11 survivor, the LAST thing on my mind would have been getting my projects back on track!
People need to get their damned priorities straight. If you lose your job because you'd rather spend time with family or just enjoy life, so be it. Jobs can be replaced. Time cannot.
Why didn't they mention the survival of slashdot in the face of countless disasters. The great troll strike of 2002 comes to mind! The revival of beowulf jokes, the lawsuit from Nat Portman and the hot grits famine that followed were all destructive but /. survived. Slashdot is able to survive just about any disaster whether in Soviet Russia or at home, and for that it should be commended!
Slashdotter are stupid and biased.
I can't wait for the comments on this one. Someone will try to be all cool and enlightened, in order to achieve a +5, Insightful. That will be the highlight of their day. And then someone will spend ten minutes thinking of some wisecrack or pun to get them a Funny. They'll then show it to everyone else and wonder aloud if they could take Leno's job. Someone else will Google around for a minute, grab the first semi-relevant link, and post it hoping to get an Informative. Then someone will spin a fake yarn about their disaster, and get pegged for Interesting.
I post pretty often here on Slashdot, usually not as an AC. But sometimes you just see the futility in it all.
Your services are wanted at 20:00 EST (Midnight UTC, 01:00 British time, 05:30 India time) for a massive uber trolling of Taco's irc channel. Bring your ascii art, your nastiest bots and proxies and cause as much chaos as you can before you get kicked. Make Wipo troll proud!
Seems like that chapter is required reading these days.
Asuka was staying out at Hikari's place tonight, so it was the perfect time to put my plan in motion. Kaji-kun had got me the stuff. He didn't ask me what I needed it for.
Misato always bugged me to go to the fridge and get her more beer. I took the opportunity to drop the small white tablet in the top of the can when I opened it in the kitchen. The dumb slut downed it in one as she always did. She had thanked me and called me "Shinji-chan". I'd show her just how much of a 'little Shinji' I was tonight.
The rohypnol was really fast to work. Misato complained of having a headache then she was fast asleep draped over the couch, her bathrobe falling open and revealing one of her big, bouncy tits. I got a hard-on straight away.
I grabbed her by the arms, dragged her into her bedroom and dumped her on the bed. God damn, she looked fucking gorgeous when she was unconscious. Striping off, I grabbed her long purple hair and began to play with it, rubbing it on my 12 inch cock and raping it around. I nearly cummed straight away.
I pulled her bathrobe open and exposed her massive F cup tits. Even though they were big they stuck straight up in the air. I grabbed hold of them and felt how big and soft they were. Lying next to her I spent about half an hour kneading her tits. Even though Misato was out cold, she began to moan in excitement. This was the first time I'd ever touched a womans body and I was as hard as steel.
I took her long fat nipple into my mouth and began to suck it. Misato groaned. I got a pleasant surprise when warm sweet milk flooded into my mouth. So that explains why her tits were so big. I suckled her and suckled her, wanting to get every drop from both breasts, twisting at biting at her nipples to make them flow more as I was getting more and more excited until I ended up shooting my load all over her smooth tummy.
I scooped up my hot white jizz with my fingers and held it to Misato's mouth. She started sucking all off. It was true - she was a total slut.
I looked at her pussy. It was smooth and shaved. I could smell her excitement from what I had done before. I spread her legs and slid 2 fingers into her cunt. She was good and wet. I licked her cum from my fingers, she tasted great.
My cock hadn't even gone down a single bit from before, so I used it to impale her cunt. I slid it in right up to the end. Misato arced her back and moaned again and again. Her inner muscles squeezed me and I began to fuck her as hard as I could.
"Misato, you fucking slut. You like being raped by me, don't you? Don't you?" she started to orgasm as I fucked her so hard that her pussy began to bleed and her juice squirted out of her.
I was in heaven. She'd led me on so much, now here I was, fucking her brains out and she couldn't do anything about it. I screamed as I cummed, dumping one of my biggest wads of jizz into the depths of her cunt, filling her womb and making it squirt out of her. as I did so I squeezed her tits as hard as I could causing milk to spurt into the air.
I wiped my dick clean on her hair making sure I let some cum drip on her face. Just to think, I could fuck the head of NERV operations whenever I wanted and she would not be able to stop me. She wouldn't even know who had raped her. she wouldn't think it was me, Shinji-chan. She might even get pregnant too cause there was no way that I was going to wear a condom.
I still had a whole bag of rohypnol left, so there was lots more fun to be had. I thought that I might take her up the ass next time. Asuka better be careful how she talks to me too.
As I went off to my bed and left her asleep on hers, Pen Pen watched me leave. He looked very happy and there was some blood on his feathers from when he had had a nosebleed. I wonder if he would like to join in fucking Misato next time?
THE END
"Surviving Slashdot" by Oliver Clozoff
"Surviving Slashdot" Illstrates how to build a corporate network that accepts large numbers of incoming connections from stories posted at Slashdot.org, while still allowing employees to make network connections that they need. Techniques covered include round-robin DNS with different servers in different geographical locations, multiple HTTP servers with load balancing, and smooth transition over to a volume web host. like Conxion or cNet at a moment's notice without significant downtime. Other Anti-Slashdotting tactics also discussed.
Do not look into laser with remaining eye.
Pure genius... gold I say gold!
"Planning for Survivable Networks has many factual tidbits about disasters of all sorts..."
I wonder if that's included.
When SARS hit earlier this year our disaster recovery planning team was faced with a situation they hadnâ(TM)t anticipated: potential quarantining of large numbers of staff with critical business-continuity functions.
The building and computer systems would be physically secure, but staff would not allowed into the workplace.
So there was a scramble to survey everyoneâ(TM)s job function and set up broadband and VPN access from home if needed.
Esteem isn't a zero sum game
In case of disaster:
1) Start a riot, and punch people trying to stop you from looting.
2) Run in the street without wearing any pants.
3) Have sex with a mare.
4) ???
5) Profit!
...VaL going under? Because it certainly looks like that is going to happen.
Their trading floor might've been up in no time, but speaking as someone who worked with the Lehman Brothers in WTC on 9/11, I can say some of their other divisions weren't as lucky.
The team I was on lost 2 months worth of work, because it wasn't backed up on a remote site. The version control servers were at WTC.
If it wasn't for a single developer, who had made an unauthorized copy of the project on a floppy, we would've lost much more than just 2 months.
Proletariat of the world, unite to kill terrorism
In Soviet Russia, I ruled you
Rather, the bombing she survived occurred at Ramstein Air Base in Germany, 20 years before.
I happened to be at Rammstein the day after the bombing mentioned. The transmission from the car got blown over the top of a four-story building (other parts didn't quite make it through the building). Quite a powerful bomb that killed and hurt many people. I think it eventually got pinned on the Red Army Faction.
The fun part was I was returning a Siemens teletype to the maintenance depot there, and the other guy in the VW pickup with me had forgotten his military ID (he had left it in his field jacket back at our base). So here we are pulling up to the main gate with this huge wooden crate in the back, and only one of us has any ID. We were lucky they didn't strip search us on the spot.
Chip H.
Unban me from the chatroom, dweebs.
And I'm guessing your pathetic moderators will just mod me down. Your moderators have the identification number (ID) of 10-T. (aka ID-10-T = idiot)
Every major corporation should remember this: Data is more important than lives. Even if 3000 people die in a terrorist attack, you'd better have backups of your data so that you can open for business the next morning.
Tarsnap: Online backups for the truly paranoid
if it can't be recovered from the on-site week+ old backup, then we close the doors (if the doors are still there) and file for chapter (7, 11, 13, whatever the lawyer suggests)
They are the best prepared for a disaster, by the virtue of being required to be open on the fourth day. Ever since the stock market crash, banks have exactly 3 day to recover from ANY disaster and open the doors or the federal government will step in and take over. The fines for failing to uphold any of the fed reg's is ENORMOUS. Both BofA and WellsFargo have used their plans successfully in the past. BofA in both SF during the quake, and in LA during the riots, and Welss Fargo's main headquarters burned. A good Contingency Operations Program is VERY EXPENSIVE, and requires many things beyond the obvious. Do your sales people have all their numbers in a rolodex on their desk, will they be able to function without it ?
errr....umm...*whooosh* *whoosh* Is this thing on ?
If it wasn't for a single developer, who had made an unauthorized copy of the project on a floppy,
I ask this question only half-jokingly:
Was s/he fired?
The coolest voice ever.
Run down on what I learned from 9-11.
Were constantly under attack on some front, hey I knew this in my Marine corps days, some attacks are just worse than others.
What YOU should have learned from 9-11.
Dont take life for granted, your a freaking SysAdmin, A programmer, a Techie or god forbid some kind of manager that can be replaced. Work when your at work, back shit up and when you leave work, leave work, dont take it with you if your gone tomorrow, someone will notice, in a week there will be a new face in the crowd to replace you.
You never really know when your gonna be part of some F-ed up shit that is going to happen. Go surfing, get a Girlfriend, get a life outside of work.
The most important disaster you should be planning for is your own, is this mentioned in the book?
corporations don't spend on disaster recovery anymore. They blew it on Y2K, its been all down-hill since.
disgruntled IT schlub.
"would you like fries with that ?"
FAILIN'
And the phrase "camel jockey" is not perjorative? Where I live, this kind of invective can potentially be "racial hatred". Incitement to racial hatred is a criminal offence
Leaving aside the blatant racism, I don't get why so many people think this post is funny; it seems very lame to me.
I mentioned this just the other day - don't forget the Emergency Pants!
You can learn a lot from Sluggy Freelance.
Don't trust President George W. Bush et al.
Cheers,
W00t
...when disaster strikes, don't forget your towel.
Everything backed up to *a* floppy?? What do you guys do all day anyway?
My daddy built the atom bomb!
The Survivable Network Technology program at the Software Engineering Institute (part of Carnegie Mellon University) describes in detail what "survivable network" actually means. The author [of the book in the
In fact, a quick google on "survivable network" turns up several hits (on the first page) from the SEI.
(Disclaimer: I used to work at the SEI, but in a different area.)
I want to drag this out as long as possible. Bring me my protractor.
Hey! Great book. Anyone have a BitTorrent link?
$40 at Barnes and Noble
$28 at Amazon
Eschew Obfuscation
There was an interesting article in Nature a while back... said that networks like the Internet, which are very tolerant of faults in links and nodes, are not so tolerant of intentional attacks on nodes with high connectivity.
here's the ref. for the curious:
Albert A, Jeong H, Barabasi AL, Error and attack tolerance of complex networks Nature 406:378-382, 2000
have interviewed the slashdot crew on how they handled there system when everybody in the world was hitting them during 9/11. Sure, its a secondary effect of the actually event, but some news agencies SAs could use this for future events.
The Kruger Dunning explains most post on
...Planning to Survive Your Network?
Good post- kudos!
Manipulate the moderator system! Mod someone as "overrated" today.
When a disaster actually occurs, and your well thought out and tested disaster plan makes the whole operation a sucess, celebrate with a fine vintage.
Don't forget to keep a redundant backup copy of a corkscrew as well
"dope will get you through times of no money better than money will get you through times of no dope"
you can get it at www.bookpool.com for $21.95 + shipping
Thanks to VERITAS Software's NetBackup http://www.veritas.com may companies were saved in the Towers coming down.
Ross Anderson, professor at Cambridge University has some works on this including Programming Satan's Computer (PDF) which looks at cryptographic protocols being attacked by being deployed on hostile system. Such as Satellite TV decoders which rely on smartcards which are in the posession of the attacker / customer.
The Tamper Lab is pretty impressive too.
Making your system realible in the present of the hostile attacker or on a hostile system is very hard, well nearly impossible.
#1: OK, small nitpick: Lehman's HQ was in WFC3, NOT in WTC1. However, it did have presence in all 4 buildings mentioned.
:(
#2: While thoretically Lehman was migrated more-or-less OK (we did have off-site backups, backup datacenter, etc...), in practice the only thing that saved them was the working-to-death of IT people in the next week.
Many of backups were made on the same-site servers. Restores were difficult, obviously. (read: almost impossible in some cases).
Many servers didn't have decent failover h/w in the backup datacenter. Hint: the datacenter was increased by over 100% in 4 days, based on my visual estimates while carrying servers up there).
FYI, I was "blessed" with starting off with a 24-hour shift, and then pulling 12-hour night shifts for over a week. Considering the fact that 9/12/01 was my 1-month wedding anniversary and that both Mrs. and myself were in WTC1 when the plane flew into it, one can see how I was a bit upset at the management, ESPECIALLY since my own application failed over with no problems - i'd rather have spent more time with her.
What did I get for all that effort? Yay! A plaque, with an image of WTC. Nice gesture, Mr. CEO!
-DVK
"The right to figure things out for yourself is the only true freedom everyone shares. Go use it"-R.A.Heinlein
It's more oriented towards small businesses and ISPs without the resources to build complete backup sites a few thousand miles away.
Tech Public Policy stuff
On the other hand, we're not so good at planning for the "ice storm kept everybody home" problem. I have once had a meeting cancelled because of snow in Silicon Valley - the critical customer lived in Santa Cruz, and there were six inches of snow on the mountain top, on a road which is unpredictable in good weather.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I would have modded you down to zero for encouraging an irrational fear of guns... you expect brownie points for political correctness on slashdot?
Tech Public Policy stuff
No, but I do exepect something that over-paranoid admins are gonna mistake for advice get modded something a little less than a 4... The parent post seems to have attracted a few Flamebait and Troll mod points since I posted. :)
1) Just because you have a disaster plan doesn't put your company in the clear. You've got to put it into action and make sure that this plan will be ready to go at a moment's notice.
2) You've got to test the plan/Backups pierodically.
3) During 9/11 in NYC, the only portable communication devices that worked in the Twin Towers were Blackberry devices.
4) A Remote, out of state, location for a backup datacenter is a good thing.
5) If you need justification for Management for putting together a disaster plan, say this "Which will cost more, putting together a Disaster Plan or repairing a companies reputation as a result for not having one?
Dolemite
_______________________
Save the World! Use a Quote!
"2) You've got to test the Backups pierodically."
Dammit, backups aren't there to be used. They're there for our ISO-9000 certification! Whaddya mean you want to retrieve data?
Oops, best become AC again...
"I mentioned this just the other day - don't forget the Emergency Pants!"
Coffee machine on a UPS