Slashdot Mirror
Planning for Survivable Networks
Priscilla Oppenheimer writes "Annlee A. Hines' book Planning for Survivable Networks, is quite a page-turner. Yes, that's surprising for a technical book, but I found it to be true. I was fascinated by the stories of real companies (Lehman Brothers, the Wall Street Journal, and others) that survived the 9/11 attack and resumed business quickly. There are also stories from other disasters, both man-made and natural, and information on companies that were not able to quickly resume business. The author summarizes the stories with explanations of what went right and what went wrong, with advice on developing your own disaster recovery plan." Read on for the rest of her review.
Planning for Survivable Networks
author
Annlee A. Hines
pages
320
publisher
Wiley Publishing, Inc.
rating
10
reviewer
Priscilla Oppenheimer
ISBN
047123284X
summary
Designing networks that can recover from natural and unnatural disasters
As Hines explains, Lehman Brothers had headquarters in Tower 1, as well as in 1,2,3 World Financial Center (across the street from the WTC towers). Lehman moved to a backup recovery location and performed cash-management functions the same day as the attack. The company was online trading fixed-income securities by the next day. They had 400 traders online when the NYSE reopened Monday, 9/17.
The Wall Street Journal (WSJ) published the story of its own recovery and Hines used that as source material for her book. WSJ had an extensive disaster recovery plan, based on lessons learned in the 1990 power blackouts in New York. After the blackouts and a subsequent fire in the emergency generator room, WSJ decided that it would never again depend on just one location being operational. WSJ opened other offices that could perform some of the necessary tasks to bring out a paper. Geographical diversity of resources seems to be a key to success.
When the 9/11 terrorists attacked the buildings across the street from WSJ's main offices, senior managers called for an evacuation, knowing that they could still produce the paper. The Wall Street Journal managed to publish a full newspaper with eyewitness accounts of the tragedy the next day.
Hines' writing is easy to follow. Although she delves into some technical details, with the requisite IP and TCP header depictions that you will find in so many networking books, the book can easily be read by managers and business people. Planning for Survivable Networks has many factual tidbits about disasters of all sorts, and although these are interesting, the primary benefit of reading the book is to gain an understanding of the characteristics of companies that sustained business after a disaster compared to companies that did not.
As Hines says, the companies that survived disasters all had disaster recovery plans in place. The plans were activated by decisive managers, who also promptly got their people out of harm's way. (If people don't survive, it won't matter much if systems survive.) Another point she makes is that the managers had to be adaptable. Not everything went according to plan, and it shouldn't be expected that it will.
The book opens with the author being rocked by a terrorist-caused explosion herself. She wasn't present for the 9/11 attackers. Rather, the bombing she survived occurred at Ramstein Air Base in Germany, 20 years before. A retired Air Force officer, she has dealt with threats all over the world for many years. Her direct command and control experience has taught her many lesson, which she shares with the reader in Planning for Survivable Networks.
Probably one of the most useful chapters, Chapter 11, "The Business Case," offers advice on presenting to management a case for a network continuity plan. According to the back cover, Hines has taught economics at a community college, and I would say that experience helped her explain the many costs involved in having a disaster recovery plan, including fixed, variable, direct, and indirect costs. She also explains the expected value of having a plan and how to sell that to management.
I recommend this book as an informative discussion of how companies can ensure business and technology continuity in a world with hackers, terrorists, natural disasters, and human error. It's a practical book, but also a surprisingly uplifting book, considering its technical content. I truly enjoyed reading about the adaptable human spirit that enabled managers and workers to keep their businesses going after the 9/11 attacks.
As Hines explains, Lehman Brothers had headquarters in Tower 1, as well as in 1,2,3 World Financial Center (across the street from the WTC towers). Lehman moved to a backup recovery location and performed cash-management functions the same day as the attack. The company was online trading fixed-income securities by the next day. They had 400 traders online when the NYSE reopened Monday, 9/17.
The Wall Street Journal (WSJ) published the story of its own recovery and Hines used that as source material for her book. WSJ had an extensive disaster recovery plan, based on lessons learned in the 1990 power blackouts in New York. After the blackouts and a subsequent fire in the emergency generator room, WSJ decided that it would never again depend on just one location being operational. WSJ opened other offices that could perform some of the necessary tasks to bring out a paper. Geographical diversity of resources seems to be a key to success.
When the 9/11 terrorists attacked the buildings across the street from WSJ's main offices, senior managers called for an evacuation, knowing that they could still produce the paper. The Wall Street Journal managed to publish a full newspaper with eyewitness accounts of the tragedy the next day.
Hines' writing is easy to follow. Although she delves into some technical details, with the requisite IP and TCP header depictions that you will find in so many networking books, the book can easily be read by managers and business people. Planning for Survivable Networks has many factual tidbits about disasters of all sorts, and although these are interesting, the primary benefit of reading the book is to gain an understanding of the characteristics of companies that sustained business after a disaster compared to companies that did not.
As Hines says, the companies that survived disasters all had disaster recovery plans in place. The plans were activated by decisive managers, who also promptly got their people out of harm's way. (If people don't survive, it won't matter much if systems survive.) Another point she makes is that the managers had to be adaptable. Not everything went according to plan, and it shouldn't be expected that it will.
The book opens with the author being rocked by a terrorist-caused explosion herself. She wasn't present for the 9/11 attackers. Rather, the bombing she survived occurred at Ramstein Air Base in Germany, 20 years before. A retired Air Force officer, she has dealt with threats all over the world for many years. Her direct command and control experience has taught her many lesson, which she shares with the reader in Planning for Survivable Networks.
Probably one of the most useful chapters, Chapter 11, "The Business Case," offers advice on presenting to management a case for a network continuity plan. According to the back cover, Hines has taught economics at a community college, and I would say that experience helped her explain the many costs involved in having a disaster recovery plan, including fixed, variable, direct, and indirect costs. She also explains the expected value of having a plan and how to sell that to management.
I recommend this book as an informative discussion of how companies can ensure business and technology continuity in a world with hackers, terrorists, natural disasters, and human error. It's a practical book, but also a surprisingly uplifting book, considering its technical content. I truly enjoyed reading about the adaptable human spirit that enabled managers and workers to keep their businesses going after the 9/11 attacks.
You can purchase the Planning for Survivable Networks from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
My book on this subject is one page long.
Page 1: Don't let Slashdot link to you.
â¦as I would prefer death to running a network these days.
from the article:
"probably one of the most useful chapters, chapter 11, "the business case," offers advice on presenting to management a case for..."
in light of the current economy, i find this particular chapter arrangement particularly funny.
ed
"Disaster recovery" is the biggest load of shit. If I had been a 9/11 survivor, the LAST thing on my mind would have been getting my projects back on track!
People need to get their damned priorities straight. If you lose your job because you'd rather spend time with family or just enjoy life, so be it. Jobs can be replaced. Time cannot.
Why didn't they mention the survival of slashdot in the face of countless disasters. The great troll strike of 2002 comes to mind! The revival of beowulf jokes, the lawsuit from Nat Portman and the hot grits famine that followed were all destructive but /. survived. Slashdot is able to survive just about any disaster whether in Soviet Russia or at home, and for that it should be commended!
Slashdotter are stupid and biased.
Seems like that chapter is required reading these days.
"Surviving Slashdot" by Oliver Clozoff
"Surviving Slashdot" Illstrates how to build a corporate network that accepts large numbers of incoming connections from stories posted at Slashdot.org, while still allowing employees to make network connections that they need. Techniques covered include round-robin DNS with different servers in different geographical locations, multiple HTTP servers with load balancing, and smooth transition over to a volume web host. like Conxion or cNet at a moment's notice without significant downtime. Other Anti-Slashdotting tactics also discussed.
Do not look into laser with remaining eye.
"Planning for Survivable Networks has many factual tidbits about disasters of all sorts..."
I wonder if that's included.
When SARS hit earlier this year our disaster recovery planning team was faced with a situation they hadnâ(TM)t anticipated: potential quarantining of large numbers of staff with critical business-continuity functions.
The building and computer systems would be physically secure, but staff would not allowed into the workplace.
So there was a scramble to survey everyoneâ(TM)s job function and set up broadband and VPN access from home if needed.
Esteem isn't a zero sum game
Their trading floor might've been up in no time, but speaking as someone who worked with the Lehman Brothers in WTC on 9/11, I can say some of their other divisions weren't as lucky.
The team I was on lost 2 months worth of work, because it wasn't backed up on a remote site. The version control servers were at WTC.
If it wasn't for a single developer, who had made an unauthorized copy of the project on a floppy, we would've lost much more than just 2 months.
Proletariat of the world, unite to kill terrorism
In Soviet Russia, I ruled you
I've been involved with disaster recovery plans since 1993 in Houston (hurricane seasons, a propensity for flooding). Most reputable companies down here have viable plans including offsite call centers, daily backups to servers/db's offsite, etc..
I have to relate a funny story though. I wrote code for a large bank with a few offices in downtown Houston. As tropical storm Allison approached (you may have seen pictures of the aftermath), we started sending people home. Unfortunately, the shortsighted management had placed two offsite databases IN HOUSTON for data and call center recovery. The last I saw of our particular network administrator was him loading the physical DB server into his truck in hopes that he could get it home and upstairs. The two DR sites both flooded and we lost those servers. Needless to say, that manager is no longer employed with .
Saying Android is a family of phones is akin to saying Linux is a family of PCs.
Ha! I love the political incorrectness. Seriously, isn't the whole idea planning and an second location? Really, when disaster hits, whether external (terrorism) or internal (hard disk failure), is the person responsible for these systems in a frame of mind to create a plan?
I've watched my 24/7 server choke and die. I had a fever and still got things up and running in less than 8 hours. Why? A plan. I knew where it was and where all my manuals and documentation were.
Just because a server is small and easy to set up doesn't mean it should be treated any less than a mainframe should. Let me say that again, because this is why this is a topic of discussion: treat your servers like mainframes were treated 20 years ago.
--- "1.21 Jigawatts!" -Doc
Rather, the bombing she survived occurred at Ramstein Air Base in Germany, 20 years before.
I happened to be at Rammstein the day after the bombing mentioned. The transmission from the car got blown over the top of a four-story building (other parts didn't quite make it through the building). Quite a powerful bomb that killed and hurt many people. I think it eventually got pinned on the Red Army Faction.
The fun part was I was returning a Siemens teletype to the maintenance depot there, and the other guy in the VW pickup with me had forgotten his military ID (he had left it in his field jacket back at our base). So here we are pulling up to the main gate with this huge wooden crate in the back, and only one of us has any ID. We were lucky they didn't strip search us on the spot.
Chip H.
if it can't be recovered from the on-site week+ old backup, then we close the doors (if the doors are still there) and file for chapter (7, 11, 13, whatever the lawyer suggests)
You need your own column on /. This is one of the funniest posts I've read in a very long time...
And at this hour, the parent post is modded 80% Funny and 20% troll, with a bonus point for coming from a high-karma user. Good work mods, I'm glad nobody's mistaking it for insightful or informative.
Sorry, as much as you try to sugar coat it, this advice is too smart for its own good. Being a racist in your hiring practices is illegal, and keeping a gun or two in the server room is more likely to get you blamed for contributing to a death when a workplace fight gets out of hand than ever being shot at a terrorist. There is such a thing as being too secure for your own good.
Some common-sense preperations are needed, but too many become wasteful, and many bring their own daily problems. Remember, when disasters by definition are rare. Balance your risk tolerance with your tolerance for annoyance in your daily operations... far too many people forget to do that.
That, my friend, is one of the funniest damn things I've read in a long time. That's getting printed and going onto a wall somewhere.
Vintage computer games and RPG books available. Email me if you're interested.
They are the best prepared for a disaster, by the virtue of being required to be open on the fourth day. Ever since the stock market crash, banks have exactly 3 day to recover from ANY disaster and open the doors or the federal government will step in and take over. The fines for failing to uphold any of the fed reg's is ENORMOUS. Both BofA and WellsFargo have used their plans successfully in the past. BofA in both SF during the quake, and in LA during the riots, and Welss Fargo's main headquarters burned. A good Contingency Operations Program is VERY EXPENSIVE, and requires many things beyond the obvious. Do your sales people have all their numbers in a rolodex on their desk, will they be able to function without it ?
errr....umm...*whooosh* *whoosh* Is this thing on ?
1)Absolutely, and I would go even farther, in prep for Y2K a friend of mines father had a worst case cenario drill during off hours, they disconnected all incoming WAN links simulating all other datacenters or the telco failing to catch a critical error resulting in total loss of conectivity, some of his collegues found out that their systems had interdependencies that they had not accounted for that could have resulted in problems had there been an actual disruption.
1a)or if you can afford it have offsite backups AND offsite replication
1b)You may not need to have them already purchased there are companies like on of my former employers (then G.E. Capital I.T. Solutions now just G.E. I.T.S) that will sign contracts to have a certain number of each of your machine types available along with either space or mobile trailers to house them.
2)Duh =)
3) Ummm, whatever
4) again duh =)
4a) already standard practice most places unfortunatly
4b) I leave civil defense to the goverment, it's one of the few things they should do.
4c) I forsee this as being a bad thing the next time layoffs happen.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Yeah, okay. Some people died, so let's make sure the people that didn't are also out of work. Great strategy.
If it wasn't for a single developer, who had made an unauthorized copy of the project on a floppy,
I ask this question only half-jokingly:
Was s/he fired?
The coolest voice ever.
Run down on what I learned from 9-11.
Were constantly under attack on some front, hey I knew this in my Marine corps days, some attacks are just worse than others.
What YOU should have learned from 9-11.
Dont take life for granted, your a freaking SysAdmin, A programmer, a Techie or god forbid some kind of manager that can be replaced. Work when your at work, back shit up and when you leave work, leave work, dont take it with you if your gone tomorrow, someone will notice, in a week there will be a new face in the crowd to replace you.
You never really know when your gonna be part of some F-ed up shit that is going to happen. Go surfing, get a Girlfriend, get a life outside of work.
The most important disaster you should be planning for is your own, is this mentioned in the book?
corporations don't spend on disaster recovery anymore. They blew it on Y2K, its been all down-hill since.
disgruntled IT schlub.
"would you like fries with that ?"
I mentioned this just the other day - don't forget the Emergency Pants!
You can learn a lot from Sluggy Freelance.
...when disaster strikes, don't forget your towel.
Everything backed up to *a* floppy?? What do you guys do all day anyway?
The Survivable Network Technology program at the Software Engineering Institute (part of Carnegie Mellon University) describes in detail what "survivable network" actually means. The author [of the book in the
In fact, a quick google on "survivable network" turns up several hits (on the first page) from the SEI.
(Disclaimer: I used to work at the SEI, but in a different area.)
I want to drag this out as long as possible. Bring me my protractor.
I don't love racism. Don't be silly. I love the humor of the comments.
--- "1.21 Jigawatts!" -Doc
$40 at Barnes and Noble
$28 at Amazon
Eschew Obfuscation
Please tell us your age, sex, race and circumstances of your upbringing, and we in the know will explain to you a) why it's funny; b) why you don't "see it".
There are no karma whores, only moderation johns
There was an interesting article in Nature a while back... said that networks like the Internet, which are very tolerant of faults in links and nodes, are not so tolerant of intentional attacks on nodes with high connectivity.
here's the ref. for the curious:
Albert A, Jeong H, Barabasi AL, Error and attack tolerance of complex networks Nature 406:378-382, 2000
have interviewed the slashdot crew on how they handled there system when everybody in the world was hitting them during 9/11. Sure, its a secondary effect of the actually event, but some news agencies SAs could use this for future events.
The Kruger Dunning explains most post on
You need to exercise that brain some more. The sarcasm in the original post wasn't that subtle.
Good post- kudos!
Manipulate the moderator system! Mod someone as "overrated" today.
When a disaster actually occurs, and your well thought out and tested disaster plan makes the whole operation a sucess, celebrate with a fine vintage.
Don't forget to keep a redundant backup copy of a corkscrew as well
"dope will get you through times of no money better than money will get you through times of no dope"
Thanks to VERITAS Software's NetBackup http://www.veritas.com may companies were saved in the Towers coming down.
Ross Anderson, professor at Cambridge University has some works on this including Programming Satan's Computer (PDF) which looks at cryptographic protocols being attacked by being deployed on hostile system. Such as Satellite TV decoders which rely on smartcards which are in the posession of the attacker / customer.
The Tamper Lab is pretty impressive too.
Making your system realible in the present of the hostile attacker or on a hostile system is very hard, well nearly impossible.
#1: OK, small nitpick: Lehman's HQ was in WFC3, NOT in WTC1. However, it did have presence in all 4 buildings mentioned.
:(
#2: While thoretically Lehman was migrated more-or-less OK (we did have off-site backups, backup datacenter, etc...), in practice the only thing that saved them was the working-to-death of IT people in the next week.
Many of backups were made on the same-site servers. Restores were difficult, obviously. (read: almost impossible in some cases).
Many servers didn't have decent failover h/w in the backup datacenter. Hint: the datacenter was increased by over 100% in 4 days, based on my visual estimates while carrying servers up there).
FYI, I was "blessed" with starting off with a 24-hour shift, and then pulling 12-hour night shifts for over a week. Considering the fact that 9/12/01 was my 1-month wedding anniversary and that both Mrs. and myself were in WTC1 when the plane flew into it, one can see how I was a bit upset at the management, ESPECIALLY since my own application failed over with no problems - i'd rather have spent more time with her.
What did I get for all that effort? Yay! A plaque, with an image of WTC. Nice gesture, Mr. CEO!
-DVK
"The right to figure things out for yourself is the only true freedom everyone shares. Go use it"-R.A.Heinlein
It's more oriented towards small businesses and ISPs without the resources to build complete backup sites a few thousand miles away.
Tech Public Policy stuff
On the other hand, we're not so good at planning for the "ice storm kept everybody home" problem. I have once had a meeting cancelled because of snow in Silicon Valley - the critical customer lived in Santa Cruz, and there were six inches of snow on the mountain top, on a road which is unpredictable in good weather.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I would have modded you down to zero for encouraging an irrational fear of guns... you expect brownie points for political correctness on slashdot?
Tech Public Policy stuff
No, but I do exepect something that over-paranoid admins are gonna mistake for advice get modded something a little less than a 4... The parent post seems to have attracted a few Flamebait and Troll mod points since I posted. :)
1) Just because you have a disaster plan doesn't put your company in the clear. You've got to put it into action and make sure that this plan will be ready to go at a moment's notice.
2) You've got to test the plan/Backups pierodically.
3) During 9/11 in NYC, the only portable communication devices that worked in the Twin Towers were Blackberry devices.
4) A Remote, out of state, location for a backup datacenter is a good thing.
5) If you need justification for Management for putting together a disaster plan, say this "Which will cost more, putting together a Disaster Plan or repairing a companies reputation as a result for not having one?
Dolemite
_______________________
Save the World! Use a Quote!
If you're DR plan is relying on DAT tapes, well then you don't have a DR plan, you're relying on Lady Fortune, and she's a mighty fickle helper.
??? = sell the looted goods -- finally, one even I can figure out
Think global, act loco
"I mentioned this just the other day - don't forget the Emergency Pants!"
Coffee machine on a UPS