Slashdot Mirror
Explosion At ThePlanet Datacenter Drops 9,000 Servers
An anonymous reader writes "Customers hosting with ThePlanet, a major Texas hosting provider, are going through some tough times. Yesterday evening at 5:45 pm local time an electrical short caused a fire and explosion in the power room, knocking out walls and taking the entire facility offline. No one was hurt and no servers were damaged. Estimates suggest 9,000 servers are offline, affecting 7,500 customers, with ETAs for repair of at least 24 hours from onset. While they claim redundant power, because of the nature of the problem they had to go completely dark. This goes to show that no matter how much planning you do, Murphy's Law still applies." Here's a Coral CDN link to ThePlanet's forum where staff are posting updates on the outage. At this writing almost 2,400 people are trying to read it.
9000::7500?
So I guess a "customer" in this case is a company or business, not an individual? Unless many of the individuals have several servers each.
Electricity is a fickle mistress, one moment she's gently caressing your genitals through gingerly applied electrodes the next she's blowing up your data centers.
... for posting frequent updates to the status of the outage.
Lesson learned: don't store dynamite in the power room.
At this writing almost 2,400 pelople are trying to read it. Posting it on slashdot should help speed it up.
-- -- Warning. Do not stare directly at the sun.
I wonder what the dollar value of the repairs will run? I'm sure insurance covers this kind of thing, but I'd love to see hard figures like in one of those mastercard commercials: Structural damage: $15000 Melted hardware: $70000 Halon refill: $however much halon costs Real-Life Slashdot effect: Priceless
People are like slinkies; useless but fun to watch when you push them down the stairs
And then they put it on the front page of Slashdot.
It was Sunday, June 1, 2008. Xeon, my children, just don't belong in some places.
(About the only thing missing from this real-world version of the story is a YouTube video of a halon fire suppression system going off. Damn ozone-protection regs :)
Clearly this is bad karma resulting from all their years of human rights violations....especially Tiananmen Square...oh wait--
Careful What You Wish For....
have that can explode like this? All I can think of are all those cheap electrolytic caps. They really do put on quite a show, don't they? Put the transformer up on the roof, ok?
What?
I blame Kevin Hazard.
What does the scouter say about the servers affected? IT'S OVER 9000!!!!
At this writing almost 2,400 people are trying to read it
and as of this posting, make that 152,476.
Sacred cows make the best burgers.
So who's missing from Al Gore's Internet? Who do we know who's hosted on ThePlanet?
Being in the power systems engineering biz, I'd be interested in some more information on the type of building (age, original occupancy type, etc.) involved.
To date. I've seen a number of data center power problems, from fires to isolated, dual source systems that turned out not to be. It raises the question of how well the engineering was done for the original facility, or the refit of an existing one. Or whether proper maintenance was carried out.
From TFA:
electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding their electrical equipment room. Properly designed systems should never result in any fault to become uncontained in this manner.Have gnu, will travel.
schlockmercenary.com is down. apparently they lost.
Schlock Mercenary, the popular webcomic, as well as most of the Blank Label Comics collective is down. Schlockmercenary.com now points to a holder site, and Sunday's comic is on the Livejournal community at http://schlocktroups.livejournal.com./
--
# Canmephians for a better Linux Kernel
$Stalag99{"URL"}="http://stalag99.net";
The only thing that I can imagine that could've caused an explosion in a datacenter is a battery bank (the data centers I've been in didn't have any large A/C transformers inside). And even then, I thought that the NEC had some fairly strict codes about firewalls, explosion-proof vaults and the like.
I just find it curious, since it's not unthinkable that rechargeable batteries might explode.
mr c
"Physics is like sex. Sure, it may give some practical results, but that's not why we do it." - R. Feynman
Kudos to them for their timely updates as to system status. Having their status page listed on /. doesn't help them much, but I
was encouraged to see a Coral Cache link to their status page. In that light, here's:
a link to the Coral Cache lofiversion of their status page:
I am wondering what UPS/Generator Hardware was in use?
Where would the "failure" (Short/Electrical Explosion) have to be to cause everything to go dark?
Sounds like the power distribution circuits downstream of the UPS/Generator were damaged.
Whatever vendor provided the now vaporized components are likely praying that the specifics are not mentioned here.
I recall something about Lithium Batteries exploding in Telecom DSLAMs... I wonder if their UPS system used Lithium Ion cells?
http://www.lightreading.com/document.asp?doc_id=109923
http://tech.slashdot.org/article.pl?sid=07/08/25/1145216
http://hardware.slashdot.org/article.pl?sid=07/09/06/0431237
Clearly these Sony batteries had to be replaced one way or another...
They should of also had two separate demarcation points for power as well, with a trow switch on both sides of the backup to have physical separation from the farm and the grid, only to be connected up when something like this happens. When you have that many servers, it's the only thing that makes sense.
Further, the 9,000 servers were physically, geographically, isolated enough from the power supply (which is what exploded) to be protected. We know this to be the case because we read the article and headline and understood them and they indicate that the 9,000 servers were not blown up.
To put it another way, only the power supply was damaged by the explosion, the servers were not. Probably there was no way to isolate the power from its own explosion. The servers, however, we protected.
So, in summary, the 9,000 servers were not blown up. Only the power.
The power is off due to the explosion but there servers themselves are A-OK.
"Sacrifice for the good of The State" - The State
We run the largest social networking site for family gatherings and saw all our domains and email services go down as it happened. The issue was indeed a power transformer that exploded and caused the fire. Thankfully no one was hurt and no servers were damaged including FamilyReunion.com, but the fire department ordered all power to the data center cut to reduce the possibility of subsequent fire hazard.
I have 5 servers. Each of them is in a different city, on a different provider. I had a server at The Planet in 2005.
I feel bad for their techs, but I have no sympathy for someone who's single-sourced, they should have propagated to their offsite secondary.
Which they'll be buying tomorrow, I'm sure.
Never thought I'd see that headline.
I like right across the street from one of The Planet's Dallas data centers, so when I saw this article, I was like "So why wasn't I woken by an exploding generator?" Makes sense now.
Of course, I still have to go to work on Wednesday now, too. Bah.
No, the power was off because the fire department told them to shut it off (during an investigation, I assume). The explosion was in a high power conduit - I'm sure it severed all the lines inside the conduit itself. This is one of those things that couldn't easily be avoided at a single site. But, if your server is of any importance, you do have a colo, right?
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
My main server is located there and it's killing me waiting for it to come back up again. The abuse some people are getting though for Planet customers for not having 'switched to their backup data centres' is amazing. Some of us are small fry, we can't afford to run multiple hosting infrastructures.
Data Center Knowledge has a story on the downtime at The Planet, summarizing the information from the now Slashdotted forums. Only one of the company's six data centers was affected. The Planet has more than 50,000 servers in its network, meaning that one on five customers are offline.
Fortunately I'm hosted at the Dallas facility, and this event was at the Houston one.
Hunt your preferred prey at Aliens vs Predator MUD. Join the war at avpmud.com port 4000
http://www.vcdquality.com
Really? What about a little known thing called colocation?
At least with colocation, if the building gets blown up by terrorists, the servers are still running somewhere else.
Fight Spammers!
its not the 'no name' hosting resellers who host at the planet. no name resellers do not employ an entire server, they just use whm reseller panel that is being handed out by a company which hosts servers there.
Read radical news here
If so, that could explain the cause of the explosion...
The view was horrible and the smell was even worse; Julie severely regretted becoming a proctologist.
-Maurice
FixingTheWeb.com Helping to keep the bad guys out...
Sorry for replying to myself, I don't think I made my post clear; the backup power is not on (the mains was blown to bits), because the fire department told them to shut it off.
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
Everyone please let me know when iyfwrestling.com is back up and running!
You never expect irony, do you?
Want to be a professional wrestler? Visit www.iyfwrestling.com
@iyfwrestling
If this were DreamHost there would be a few flippant words in the official statement but pages and pages of photos...
-1 not first post
Andy
They need to build the building out of what ever they build the servers out of.
Because Hackers can turn your computer into a BOMB!
GAAH! MY PRINTER IS ON FIRE!!! PUT IT OUT! PUT IT OUT!
This morning I was wondering what has happened with Darklyrics.com
Turns out they were hosted on ThePlanet!
http://toolbar.netcraft.com/site_report?url=http://www.darklyrics.com
so let me get this straight.. the servers were /not/ blown up? ;)
This is just another example of someone writing a critical response without reading the story.
Well done.
What what I remember from Uni the coils of wire in a transformer want to be straight. When a transformer has power flowing through it the coils can exert some fairly serious pressure. Big transformers tend to be encased in concrete for this reason. Maybe there was a short, a big current flowed through the secondary coil and the force was enough to over come some weak restraints.
Can anyone give a less arm wavey description of this? Or have I misunderstood?
The power to the server was never lost, and I didn't even find out about it till a couple of days into it.
-- these are only opinions and they might not be mine.
I once caught a production exchange server on fire due to a faulty wire connected to one of the DLT drives.
The motherboard was scorched so bad that when you tapped it burnt flakes that used to be the transistors fell off.
I knew something was up as soon as I smelled burning. The smoke pretty much gave it away, though.
I don't work there any more.
Have a squat over at the hobo house.
And more to the point, the rest of the backup power systems were taken offline at the request of the fire brigade.
It's a common feature to have power shut off in the event of a fire. The Fire Service don't want to be hosing down live cabling after all. It's also why you shouldn't use lifts. Everyone thinks it's "in case the fire reaches the lift". It's actually 'cos the power is likely to be cut off at any moment (the office I work in cuts the power after 3 minutes)
The power is off due to the explosion but there servers themselves are A-OK. Brought to you by the Department of Redundancy Department.
Turn up the power
This is the hour
From every tower
A million watts of love
There comes a time when
You need a good friend
But all that you have
Is that glowing screen
You know you could fly
Your hate a run high
But you've been squeezed in
To that same old scene
You know what I mean
Turn up the power
This is the hour
From every tower
Shout it from above
Turn up the power
This is the hour
From every tower
A million watts of love
By turning that switch
You're finding your niche
And you could tell them
Where to put the advice
You should get back in
It's time to jack in
We'll help you hack in
To that glowing life
You won't have to think twice
Turn up the power
This is the hour
From every tower
Shout it from above
Turn up the power
This is the hour
From every tower
A million watts of love
.
Circumcision is child abuse.
You're thinking of The World. See http://www.theworld.com/about/internet.shtml.
ThePlanet has 5 or more datacenters. The cost and complexity of doing a full blown physically separated 2N power system at every datacenter is far more expensive than taking the chance of having to issue a credit against an SLA. Not to mention that when a fire is involved, the fire department has full authority and may instruct you to cut all power anyway - they are coming in to an unknown situation and won't risk their own people just because you say the other power system is isolated.
Another issue is the complexity of a full blown 2N power system is likely to cause more outages due to human error during routine maintenance over an N+1 system. Complete 2N power systems from grid and backup sources all the way to the servers with no single point of failure (transformers, wiring, switching, PDUs, UPSs, etc.) are enormously complex and expensive, so it's not "the only thing that makes sense". I assure you issuing a one-day pro-rated credit to all your customers is cheaper.
The Planet buried the news about this in their forum instead of publishing something in the service logins or on the home page. Customers could not log in to the system to see what was wrong because this went down as well. Only after midnight (6 hours later) did they email out details to affected servers. We were lucky we had customers more attentive who informed us a few hours after it happened and found out through the online sales chat which was trying to get us to uy a new server when we visited the site. Very effective use of personnel.
ETA has continuously been pushed back. Now that the SLA is maxed at 100% credit they seem to be in no rush to get back online. We get a few hundred dollar credit next month (as long as they don't claim force majeur) while our client who's server is affected will lose over $5000 in sales.
Service has suffered ever since the merger with EV1 and this is just another such example.
Everyone loves firemen, right? Not me. While the guys you see in the movies running into burning buildings might be heroes, the real world firemen (or more specifically fire chiefs) are capricious, arbitrarty, ignorant little rulers of their own personal fiefdom. Did you know that if you are getting an inspection from your local firechief and he commands something, there is no appeal? His word is law, no matter how STUPID or IGNORANT. I'll give you some examples later.
I'm one of the affected customers. I have about 100 domains down right now because both my nameservers were hosted at the facility, as is the control panel that I would use to change the nameserver IPs. Whoops. So I learned why I need to obviously have NS3 and ND4 and spread them around because even though the servers are spread everywhere, without my nameservers none of them currently resolve.
It sounds like the facility was ordered to cut ALL power because of some fire chief's misguided fear that power flows backwards from a low-voltage source to a high-voltage one. I admit I don't know much about the engineering of this data center, but I'm pretty sure the "Y" junction where AC and generator power come together is going to be as close to the rack power as possible to avoid lossy transformation. It makes no sense why they would have 220 or 400 VAC generators running through the same high-voltage transformer when it would be far more efficient to have 120 or even 12VCD (if only servers would accept that). But I admit I could be wrong, and if it is a legit safety issue...then it's apparently a single point of failure for every data center out there because ThePlanet charged enough that they don't need to cut corners.
Here's a couple of times that I've had my hackles raised by some fireman with no knowledge of technology. The first was when we switched alarm companies and required a fire inspector to come and sign off on the newly installed system. The inspector said we needed to shut down power for 24 hours to verify that the fire alarm would still work after that period of time (a code requirement). No problem, we said, reaching for the breaker for that circuit.
No no, he said. ALL POWER. That meant the entire office complex, some 20-30 businesses, would need to be without power for an entire day so that this fing idiot could be sure that we weren't cheating by sneaking supplimentary power from another source.
WHAT THE FRACK
We ended up having to rent generators and park them outside to keep our racks and critical systems running, and then renting a conference room to relocate employees. We went all the way to the country commmissioners pointing out how absolutely stupid this was (not to mention, who the HELL is still going to be in a burning building 24 hours after the alarm's gone off) but we told that there was no override possible.
The second time was at a different place when we installed a CO alarm as required for commercial property. Well, the inspector came and said we need to test it. OK, we said, pressing the test button. No no, he said, we need to spray it with carbon monoxide.
Where the HELL can you buy a toxic substance like carbon monoxide, we asked. Not his problem but he wouldn't sign off until we did. After finding out that it was illegal to ship the stuff, and that there was no local supplier, we finally called the manufacturer of the device who pointed out that the device was void the second it was exposed to CO because the sensor was not reusuable. In other words, when the sensor was tripped, it was time to buy a new monitor. You can see the recursive loop that would have devloped if we actually had tested the device and then promptly had to replace it and get the new one retested by this idiot.
So finally we got a letter from the manufacturer that pointed out the device was UL certified and that pressing the test button WAS the way you tested the device. It took four weeks of arguing before he finally found an excuse that let him safe face and
-- I wonder which will go down in history as the bigger failure: the War on Drugs or the War on Filesharing
ahahaha
sweet job reading troll. you didn't even need to read all those big words. here's a bunch of tiny words (they happened to be at the end of his post so you couldn't miss them)
> So, in summary, the 9,000 servers were not blown up. Only the power.
yeah, I know, I'm being a prick right now, but I've got karma to burn and people who don't read piss me off
I have developed a truly marvelous proof of this comment, which this signature is too narrow to contain.
Last message on the linux console before the explosion:
lp0 printer on fire!
Invalid Checksum. Retrying.
"The power is off due to the explosion but there servers themselves are A-OK."
Physically OK maybe... lets see how many of them come back up when the power is restored ^ ^
They supposedly had a "short in a high-volume wire conduit." That leads to questions as to whether they exceeded the NEC limits on how much wire and how much current you can put through a conduit of a given size. Wires dissipate heat, and the basic rule is that conduits must be no more than 40% filled with wire. The rest of the space is needed for air cooling. The NEC rules are conservative, and if followed, overheating should not be a problem.
This data center is in a hot climate, and a data center is often a continuous maximum load on the wiring, so if they do exceed the packing limits for conduit, a wiring failure through overheat is a very real possibility.
Some fire inspector will pull charred wires out of damaged conduit and compare them against the NEC rules. We should know in a few days.
soooo...
what about all the servers that were blown apart?
It's over NINE THOUSAAAAAND!
I wish I had mod points...I think this is the first time I ever wanted to mod those 5 words up.
[b.belong('us') for b in bases if b.owner() == 'you']
YouTube's home page is returning "Service unavailable". Is this related? (Google Video is up.)
> ...they claim redundant power...
How the hell could they claim redundant power with only one power room?
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
I remember seeing a series of photos on a website a few years ago showing the remains of a transformer outside a commercial office building that housed another datacenter. Unfortunately I forget which company it was (I want to say Hurricane Electric but I'm not 100% sure). Those photos were pretty impressive. After the fire department put the fire out there wasn't much of anything left on the concrete slab where the transformer once was...
But surely there must've been some way to keep the servers from being damaged in the explosion?
In related news, I was wondering why I wasn't getting much spam today and my sites didn't have strange spiders hitting them.
Open Source Java DAO Generator
You sure it wasn't...over 9,000?
Time it takes to get a new transformer (assumming a supply house has one laying around in their warehouse and are willing to do business on a Sunday): 1 day
Time it takes to get a new switchboard: a few weeks usually (although possibly 2 or 3 days at the earliest, assumming GE/SquareD/CutlerHammer have one laying around in their warehouse and the building doesn't have any special needs)
Time it takes to order all new copper wire and pull it in and terminate it (assumming no problems are encountered with the way the pipe is run, etc): probably about 3 to 5 days at the earliest (remember, their main board exploded, the wire ends are more-than-likely all burnt up and the wire is now worthless)
Time it takes to rehang/fireseal three walls: dunno, could take a while. sheetrockers are funny like that, LOL.
But in all seriousness, it sounds like ThePlanet are downplaying this. They are estimating stuff coming online today, when it looks like sometime next weekend (at the earliest (and assumming they're not trying to rig stuff up)).
yhbt. yhl. hand.
could be nice to remove the link.. so we (clients) cand be up to date.
We put up a large 2 car barn at the back of the parking lot to house snow removal equipment and for staging storage.
The outside had to have cement board siding, and the inside had to have TWO layers of 5/8 inch firecode drywall. Why? Because the fire house, which was next door, might be delayed in hooking up to a hydrant that was 30 feet from the building.
Whatever.
umm.... the initial article on Slashdot misspells people.... but I think the article on SE is more informative then this one on SlashDot...
http://www.syndicatedelitist.com/2008/06/theplanets-h1-datacenter-encounters.html
Usually the server blows up *after* a link is posted on Slashdot...
Not very fun being a developer on a game site that has been down for around 24 hours now...
"I feel a great disturbance in the Force, as if thousands of servers suddenly cried out in NAGIOS and were suddenly silenced. I fear something terrible has happened."
Blank Labels WebComics are hosted there, and this catastrophe could be the very first thing in over eight years straight to cause Howard Taylor to not update his daily comic Schlock Mercenary.
Maybe we deserve this world ?
Time it takes to get a new transformer (assumming a supply house has one laying around in their warehouse and are willing to do business on a Sunday): 1 day
Time it takes to get a new switchboard: a few weeks usually (although possibly 2 or 3 days at the earliest, assumming GE/SquareD/CutlerHammer have one laying around in their warehouse and the building doesn't have any special needs)
Time it takes to order all new copper wire and pull it in and terminate it (assumming no problems are encountered with the way the pipe is run, etc): probably about 3 to 5 days at the earliest (remember, their main board exploded, the wire ends are more-than-likely all burnt up and the wire is now worthless)
Time it takes to rehang/fireseal three walls: dunno, could take a while. sheetrockers are funny like that, LOL.
But in all seriousness, it sounds like ThePlanet are downplaying this. They are estimating stuff coming online today, when it looks like sometime next weekend (at the earliest (and assumming they're not trying to rig stuff up)).
I've been in DC when the power's dropped. It's surprising how physical it is when it gets quiet. I imagine the initial bang and then the silence broken by the alarms was quite an experience.
Count me as one affected. They've been great about notifying us and posting about it though. Thankfully my site is just one I run for fun, and not a business site. Sheesh.
I'm one of their customers, and it takes more than a single instance in 5 years of hosting to make me switch. That said we'll see how long it takes to get things back up. Unfortunately *both* my dns servers are in that DC, I thought they were in physically distant locations... so much for ass-um-ing things...
MP3 Search Engine
The fact is in the real world there is no such thing as 100% guarantee, datacenters no matter how well designed can, do and will go down, and it doesn't mean there is a design flaw or that another datacenter is superior.
YouTube is back up, after a few minutes of outage, so it was probably a different problem.
as if they haven't been through enough with the explosion and fire and all... you just had to rub it in and slashdot their forum as well... kudos!
You don't think enough... therefore you better not be!
This is rough... Crucial time for our game and -WHAMMO- the server is dead for a day and a half minimum.
My servers dropped off the net yesterday afternoon, and if all goes well they'll be up and running late tonight. At 1700PST they're supposed to do a power test, then start bringing up the environmentals, the switching gear, and blocks of servers.
My thoughts as a customer of theirs:
1. Good updates. Not as frequent or clear as I'd like, but mostly they didn't have much to add.
2. Anyone bitching about the thousands of dollars per hour they're losing has not credibility to me. If your junk is that important, your hot standby server should be in another data center.
3. This is a very rare event, and I will not be pulling out of what has been an excellent relationship so far with them.
4. I am adding a fail over server in another data center (their Dallas facility). I'd planned this already but got caught being too slow this time.
5. Because of the incident, I will probably make the new Dallas server the primary and the existing Houston one the backup. This is because I think there will be long term stability issues in this Houston data center for months to come. I know what concrete, drywall, and fire extinguisher dust does to servers. I also know they'll have a lot of work in reconstruction ahead, and that can lead to other issues.
For now, I'll wait it out. I've heard of this cool place called "outside". maybe I'll check it out.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
The lesson you should be taking from Murphy's Law is not "Shit Happens". The lesson you should be taking is that you can't assume that an unlikely problem (or one you can con yourself into thinking unlikely) is one you can ignore. It's only after you've prepared for every reasonable contingency that you're allowed to say "Shit Happens".
While it may be the fire dept that is erroneously preventing them from bringing up their back-up power, it's part of a poor disaster recovery plan to not engage with the fire dept, electric co, etc. before a disaster happens, so that everyone is on-board with your disaster recovery plans and that you have the ability to implement that plan.
The explosion was isolated to the power room. The servers are fine, the backup generators and batteries are fine. The servers should have been back online if they had a good disaster recovery plan. The whole point of disaster recovery is being able to handle a disaster. You can't say "oh there was a disaster, you can't help that". This is exactly what their plan should have been able to handle. The power room goes offline. It shouldn't matter if it was because of an explosion, a fire, equipment failure or being beamed into outer space.
It also shouldn't matter who is telling them to keep the power off. Part of the disaster recovery plan should have been making sure local authorities allowed them to carry it out. Fine, they have to shut off all power when firemen are in there with hoses. I understand that. But once the fire is out your plan should allow you to bring up backup power. It didn't. So I don't see how they can call themselves a "World Class Data Center". Part of what they sell and what customers expect is disaster recovery. And there are data centers that can provide this.
ThePlanet is pretty cheap compared to datacenters like NAC that have more redundancy and security. But ThePlanet wants to advertise that they are just as good. Now they were caught with their pants down when there was actually a disaster and their disaster recovery plan failed.
Open Source Java DAO Generator
There's no reason to use the forum software when they've locked the thread and are only using it to disseminate information. A Pentium one running lighttpd serving a static html page would be sufficient to handle the flood of requests.
The creature is driven by rage, and pursued by an investigative reporter. The creature is wanted for a murder he didn't commit. David Banner is believed to be dead, and he must let the world think that he is dead, until he can find a way to control the raging spirit that dwells within him.
Yup, he is a complete fucking moron.
You may also be interested in a pretty positive write-up from SANS about ThePlanet's response and handling of the situation thus far.
Somebody didn't get their Emergency Prepardness Merit Badge. This is what you get when you don't hire a Boy Scout!.
When I signed up for hosting you are given the propaganda speech on redundancy, security and all the rhetoric. The bottom line is: THEY WERE NOT PREPARED FOR THIS. The data center engineers dropped the ball. I have seen facilities that have monitors on the transformers. They could have and should have known before the transformer hit "Critical Mass," shut it down and switched to emergency backup. In this day and age it is posssible. Somewhere was a post on a "Gear jamming" that caused the transformer to blow." Anyone every seen a gear in a transformer?
Run to the local pet shop and buy all the hamsters and hamster wheels and get me back online.
Assumption is the mutherF#@&3R of all!
Same here, been happy with them for 3+ years, but this looks like we haven't seen the end of it, man I just need to get access to my box to move everything out to a server in Dallas !
So what time is it anyway...1800 and no news so it looks like 1700 power up didn't do much good.
This was taken from the data center security cameras.
http://www.youtube.com/watch?v=esymRl_0C2s
We moved three racks of servers out of ThePlanet last Thursday. Timing is everything.
Building an orphanage into the server room was unforgivable as well. All those children who were burnt crisp when the entire datacenter erupted in a huge mushroom cloud...
USE HOT GRITS WITH STATUE OF NATALIE PORTMAN (NAKED AND PETRIFIED)
Many of those same customers bitching about thousands of dollars per hour lost, not only do not have redundancy in another data center, they don't have redundancy on the only server they have. No RAID, no backups, no idea what ssh is or if it's a windows server they don't even know what remote desktop is, they're too dumb to hire an admin to manage their server... I could go on, but the most important point is:
The vast majority of those bitching customers have the world's most POINTLESS and STUPID websites. The internet is better off for having those sites down.
How are you going to handle the failover to Houston? Round-robin DNS? Or just very low TTL on the root nameservers so if something goes wrong you can update the nameservers and have them point elsewhere?
Oh and that reminds me, make sure your nameservers are spaced out as well. Learned that one the hard way.
-- taking over the world, we are.
I feel like crying. A whole weekend without it.
I have no social life. It's true.
Camping on quad since 1996.
I have about 100 sites hosted there right now that are offline, but none worth so much that a day or two of downtime will affect me so much. The worst part is getting phone calls from people who I host. and try to explain in as many ways as possible that No, I can not personally go to texas and fix it, and that there is absolutely no alternate way for them to get their email right now.
Other data centers I've been to have been smaller and primarily designed to host mainframes for large corporations. Multiple power coming in to two different rooms, multiple backup generators, adequate ups, etc.
I'm no expert in data centers, but like I previously mentioned, if someone is making claims of 100% uptime I would expect them to have some reasonable way of backing that up. They didn't. Their power room caught on fire. They didn't have a second power room that could be used and they couldn't bring backup power online.
Electrical systems fail, and sometimes catastrophically. There was a transformer that blew up on a utility pole directly across the street from me. The whole house shook. If that was in an enclosed room I can picture walls being blown down. There was also an underground fire in the wiring at one point. In both cases power was brought back online relatively fast.
I don't care what anyone says. This is poor performance compared to their marketing claims. It looks like they couldn't bring up their back-up systems because they didn't work with the local authorities when they came up with that plan.
I'm not saying ThePlanet sucks. But I wouldn't call it "world Class" and I doubt anyone's "100% uptime" claims. There is simply no amount of security / redundancy that can be done at a single location that will provide 100% uptime Someone should tell ThePlanet to stop marketing something that is so impossible then.
Open Source Java DAO Generator
Anyone know if this is why www.schlockmercenary.com is down?
From the status update thread... "Today at approximately 5:45 p.m., a transformer in our H1 data center in Houston caught fire, thus requiring us to take down all generators as instructed by the fire department. All servers are down." I read this as the fire department ordering them to kill *all* the power for safety reasons, rather than the explosion knocking the whole thing out.
What I do isn't just a web site, its also a pbx and some other stuff.
The client software that does the automation is easy. I wrote it to handle the need for a failover server so it will just try the other if the first one fails.
The PBX failover is easy, the DID provider will route to both, only one will pickup sooner. If the second does catch a call, it will have a flag on it that lets it know if the primary is still up and in that case will try to transfer over the call.
The Web Site is actually the least important part of the process for me, and I'll likely handle that with a low ttl on that one particular address.
The inbound mail is easy because I use Postini and it is good at failover.
The data is stored in a database that knows how to sync in near real time between the servers, and on disk as files. I use unison to keep the file directories in sync within just a few minutes.
Overall, I think it should work just fine. There are more elegant solutions -- and more expensive. This one will work for me.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
Well my server has been offline and even though I use it for personal use. I am considering ordering another server from a different hosting company and restoring from my remote backups and repointing my DNS and canceling my service with The Planet. I'm glad I don't host my DNS with The Planet for this and other reasons.
Also has this happened before at this data center. Wasn't this data center owned by Rackshack? Because thats who I signed up with originally before they were bought by The Planet. Here is a link from 2003 about a transformer exploding at Rackshack's Houston data center:
http://www.carrierhotels.com/wiredspace/archives/000010.html
Is this the same data center?
First, that time was an estimate -- a target. Second, even if the initial power test passes, it will take hours to bring up the a/c systems, the switches, and the routers.
:-)
The initial draw from each new bank of gear to be given power will be very high so it will need to go slow.
The battery systems (be they on each rack or in large banks serving whole blocks) will try to charge all at once. If they're not careful, that'll heat those new power lines up like the filaments in a toaster. Remember, the battery plan they have was built with the idea that they'd be used very briefly during transition to generator power -- not drained down all at once.
Only once all the switches and routing gear is back up can they start updating the network paths (do they use BGP for this -- that's not my area of expertise) so that peering data starts flowing.
Only once the network is all up and stable (no small task on a site with dozens of high end peering points) can they even start doing banks of servers.
Its also probably that each bank of servers will needs its own new power lines (and eventually replaced conduit) in the distribution center that was destroyed.
Bank by bank they'll have to bring up all these servers, each of which will draw its maximum load during boot as disks are scanned and checked.
Most of these servers probably haven't been shut down in months or years. Some drives may not spin up due to tired motors that can run fine but spinning from cold is just too much now. Other servers may have boot configuration problems undiscovered since the machines have been running without reboot for a long time -- linux ones anyway
This isn't something out of Young Frankenstein where they'll yell across the room "throw za main svitch!" and a watch the lights dim briefly while 9000 servers boot up with the deafening sound of system beeps. If they did try such a thing -- as if such a thing were possible -- it would immediately blow at least another transformer if not more.
Think about it. 9000 servers @ an average of what, 300 watts, plus the networking gear, plus the air conditioning, plus charging all those batteries....you're talking megawatts.
Without a Mr. Fusion or Harry Mudd stumbling in with some chicks wearing dilithium crystal jewelery this is going to take a while.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
The explosion may have tried hard to stop it, but the Schlock site is already back up (abet a minimal version of the site). So he still hasn't missed a daily update.
The extreme example is of course a DC in the WTC but "our power room went BOOM" is a good nth level version
( on or about 09/16/2001)
irate customer: Why is my site down AND WHY DID IT TAKE SO [redactd]LONG FOR YOU [redacted] TO GET ON THE [redacted]PHONE
support tech: well sir as you should know our primary DataCenter and primary support center was located in Tower 1 of the World Trade center [tone type=subzero chilling]You May have heard on the news THAT SAID BUILDING NO LONGER EXISTS [/tone] Thank You for calling please see our website for updates , Good Bye (click)
Any person using FTFY or editing my postings agrees to a US$50.00 charge
2 servers down. Burn.
my first impression would be something battery related. If a short trips something that shuts off your incoming AC, it kicks you over to batteries and generators. If something is then reset and brings you back online a little bit later, your hardware switches back to AC and all the batteries start charging. If the electrical fault wasn't really FIXED, (think sparks spraying from a nearby electrical box) but merely tripped something that you reset, then it can set off a hydrogen explosion from the H and O the batteries are dumping out while being recharged. THAT would require you to take things totally offline to fix since it's the point where your redundant power sources converge.
The support forum posts were not as heavy on detail as I would like to have seen, but better than about any I have ever seen under such circumstances. (something is always better than nothing) Looks like a transformer went out and did some structural damage. Probably not so much of an explosion. If you've ever seen a substation transformer go, that's probably about what happened here.
Their main concern besides getting power restored seems to be to repair networking equipment. Hard to say how that was damaged, it may have power spiked their routers and switches. (could have been other related causes - physical damage or got soaked with transformer coolant when it vented) At any rate, hazmat and firemen in general don't like working on live wires so they basically told them we don't care if you can turn some of it back on, you're going to leave it all off until we're done. Looks like they made good use of that time to gather replacement hardware and build an action plan. At this point it appears that they've been given the OK to get in there and start replacing hardware and fixing power.
There is a good video of a substation problem on youtube. This isn't necessarily what happened here, but you get the idea. Not really an explosion so much as a fire.
I work for the Department of Redundancy Department.
connects to the real world
Not to mention no no number of redundant power systems helps you at all when the fire department orders the power off.
Next, after an explosion due to an electrical fault blows 3 walls down, you have a lot of checking to do before just powering the redundant system up unless you'd like more smoke and fire.
Look, when I go into a building in gear and carrying an axe and an extinguisher, breathing bottled air, wading through toxic smoke I couldn't give crap number one about your 100 sites being down.
I have a crew to protect. In this case, I'm going into an extremely hazardous environment. There has already been one explosion. I don't know what I'm going to see when I get there, but I do know that this place is wall to wall danger. Wires everywhere to get tangled in when its dark and I'm crawling through the smoke. Huge amounts of currents. Toxic batteries everywhere that may or may not be stable. Wiring that may or may not be exposed.
If its me in charge, and its my crew making entry, the power is going off. Its getting a lock-out tag on it. If you wont turn it off, I will. If I do it, you won't be turning it on so easily. If need be, I will have the police haul you away in cuffs if you try to stop me.
My job, as a firefighter -- as a fire officer -- is to ensure the safety of the general public, of my crew, and then if possible of the property.
NOW -- As a network guy and software developer -- I can say that if you're too short sighted or cheap to spring for a secondary DNS server at another facility, or if your servers are so critical to your livelihood that losing them for a couple of days will kill you but you haven't bothered to go with hot spares at another data center then you sir, are an idiot.
At any data center - anywhere - anything can happen at any time. The f'ing ground could open up and swallow your data center. Terrorists could target it because the guy in the rack next to yours is posting cartoon photos of their most sacred religious icons. Monkeys could fly out of the site admin's [nose] and shut down all the servers. Whatever. If its critical, you have off site failover. If not, you're not very good at what you do.
End of rant.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
What about all the losses? (sells, ads incomes, lost of opportunities, etc.)
What a shame for a service that advertise REDUNDANCY deceiving all its customers... we should initiate a massive legal measures.
2 days already for my server and my 10 sites offline. What a crap service.
In that case, it cannot be advertised at all. Go ahead and install your 1024N power system and nuclear powered UPS and put it in a nuclear bunker. Fill the room with argon to make fires impossible. But if you can't promise that an asteroid will never ever strike the bunker, you better not advertise 100% uptime.
Of course, if you read the fine print, they probably have something in there about not being responsible for actions of civil authorities...such as the fire dept. ordering them to shut down all power.
I'm all for stricter requirements for truth in advertising, but honestly, it's not like this is a common problem they're having. Talk is dirt cheap. It's easy to CLAIM you could put together a system that could sail right through this sort of problem, but another matter to actually DO it. You'll never be 100% sure it'll work until something like this actually happens.
I should think only a tiny fraction (maybe .1%) of the customers are in the thousands of dollars per hour of loss category.
Most little online stores hardly break the tens of dollars per hour.
If your data is THAT important, multiple hot swap hosts in different colo's are the answer.
The common site selling earmuffs, or hosting java crapplets doesn't need that redundancy, nor can they properly pay for it.
ants are causing a lot of damage to electrical devices in Texas: http://urbanentomology.tamu.edu/ants/exotic_tx.cfm
new letter/phrase: hex-u means "www"
They don't let you use halon anymore these days. Back in '06 when my company was upgrading its datacenter, we had a similar fire issue. We had just gotten all of the servers moved into the new racks, and everything was running nicely for a few weeks.
Well, what do you know? The UPS blew up. Due to improper assembly of the UPS, one of the main cables was stretched and chafed along the chassis. Imagine, let's say, an array of 12 car batteries in series in a parallel arangement of 12 (144 total of course). The cable finally wore through late at night and arc gouged several inches of 12-gauge sheet metal chassis before it finally shorted and destroyed the batteries (from what I heard).
It was a very nasty mess, and half of the UPS was this blob of burnt-out circuitry. No fun. My company opted to spend several thousand dollars (over 10k IIRC) for a small FM-200 system.
My guess is that something like this happened to ThePlanet, but it was most likely a chemical like FM-200 that is less environmentally harmful than halon, not as cool nonetheless.
Anyway, the incident at my company was completely avoidable, and I would wager to guess that the one at ThePlanet was competely avoidable as well. I am just glad that ThePlanet lost my business about a year ago for price gouging a loyal customer after three years of service.
One power room doesn't seem to qualify as "fully redundant power system" or a "complete redundant power management system" that "assures 100% uptime".
With only one power room, you have to wonder how thoroughly and how often they perform maintenance on that equipment.
While a transformer explosion might be rare it is not uncommon. I don't think it's too much to expect for a data center that talks about "100% uptime", "fully redundant power systems" without "a single point of failure" to go offline because of a fire in the power room.
Open Source Java DAO Generator
The two latest press releases from The Planet:
"The Planet Announces Server Blowout Sale!"
and
"The Planet's Jeff Lowenberg Named Data Center Manager of the Year"
aaah good times. I'll be here all week, folks.
No, really, I'll be here all week, reloading. Checking for updates. Wishing I'd backed up my database more often.
You forget that part of your job description is to protect property. It can't always be running in to save the cute blond kid who got left behind in the panic to evacuate the home.
And yes, that means that sometimes firefighters are expected to risk their life to save just a building and prevent that fire from spreading to other empty buildings. If there wasn't a point where we expected firefighters to take an acceptable risk, we would just evacuate a town when a fire started, wait for everything to burn down in absolute safety, then rebuild.
And yet, I personally have seen a fire break out at a self-storage facility and watched four firecrews sit there and watch the entire structure (with I remind you countless family's personal possessions and sentimental collectibles) burn to the ground. Why? Because the fire chief thought there might be one person there storing ammunition and he wasn't going to risk anyone to stop the fire.
To me that's complete cowardice. If that was the job some firefighters think they are supposed to do, I want my taxes back. I can hire a $8/hr security guard to cordone off an area and wait for a fire to burn itself out. We pay someone $98K plus benefits to actually try to stop the fire even if there is a eensy chance that something could hurt him.
You point out that anything can happen at any time. Ditto for firefighters. You could get into an accident on the way to the scene. So why not just stay there in the safety of the firehouse if there are no lives at stake?
Electricians have to work with LIVE power on a routine basis. Nobody apparently gives a shit if an electrician has to risk death because they don't want to have to power down their racks while he replaces a fuse or adds a new circuit. And I have electrician friends that have the scar tissue to prove that it doesn't always turn out well.
Risk management. Do I want to see a firefighter die so that my customer can get his email today instead of tomorrow? Of course not. But do I think it makes any sense to have 9000+ people and all the systems they may depend upon go down because the fire chief doesn't think his employees can resisted clutching every exposed wire they run across?
I obviously never expected that an entire facility with redundant power supplies would lose power. Whoops, yes, my bad but I assure you I can't be the only one who never thought this was a likely possibility. I've never seen "Fireman independant power systems" advertised as a selling feature when shopping for server hosting. Maybe someone will see a market opportunity, I don't know.
But I expect that people who are paid to minimize damage to property will MINIMIZE damage, and not throw the baby out with the bathwater. The problem was clearly with high voltage. So cut the feed from the power company, case closed. Shutting down the redundant power generators that are DOWNSTREAM from the problem? That's idiotic. It's right up there with the voodoo and witchcraft method of personal protection.
Neither of us are right, dude. But the point is that I've never heard of generators being ordered off when the whole point of having generators is to provide power when the main fails. So this completely bugs me and I'm skeptical it has anything to do with erring way too far on the side of caution. Otherwise, what's next:
"Say chief, I know we have all these people in this hospital on life support, but someone left a cigarette in a trash can and I'd sure as heckfire feel safer if we just shut the whole things down"
-JoeShmoe
.
-- I wonder which will go down in history as the bigger failure: the War on Drugs or the War on Filesharing
"Did you catch the article about Google's datacenters the other day? Clearly they recognize that fact and design around it."
I wonder, http://www.informationweek.com/news/storage/showArticle.jhtml?articleID=202400961 how does a 'data center' in a box go down, when it's 'power room' explodes?
complicated electrical devices, especially where varying current can cause undesirable operation of the device, are the kinds of electrical devices that make a big bang when they go up in smoke, the conventional data center can put these parts far away from the server, so even with 3 walls going down no servers were harmed... but if it's all tightly integrated into a 'box' what happens to all the servers and the data?
i suppose if the thing is as big as a semi trailer, it could have a blast barrier, between servers, and power unit... otherwise, a data center in a box is a potentially less safe method of implementing a data center than the conventional approach.
https://www.gnu.org/philosophy/free-sw.html
mod parent up!
You sir, don't know what you're talking about. Reaching for ridiculous examples of someone doing their job wrong doesn't change that.
Our S.O.G. (standard operating guidelines) are actually very specific about risk.
We will risk our lives to save a human life.
We will take reasonable risk to save the lives of pets and livestock.
We will take minimal risks to save property.
Sorry, but your building isn't worth the risk of my crew. That's reality.
Don't you DARE tell me what is and isn't bravery or cowardly until you put 50 pounds of gear on and crawl into a pitch black house that's burning over your head.
Don't you DARE tell me that you think you understand the difference between saving the blonde girl and saving your computer server.
This isn't TV World. This is the real world. Fire on TV doesn't look like real fire. You know why? Because a real house on fire doesn't look like anything but pitch black and that makes for lousy TV.
Get over yourself and go volunteer at your local fire department. 86% of the men and women in this country who will risk their lives for yours are volunteers. We could use your help if you have the guts for it. We'll teach you what you need to know -- and we'll keep you as safe as we can so you can go home to your family when its done.
Your examples are stupid and insulting to the 800,000 brave men and women who volunteer to risk death in the most painful way possible to save your sorry butt.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
The Planet's video tour of why this wouldn't happen is up and working. Click on "Take the tour", which has many data center pictures. I like the "100% uptime" part.
It turns out they didn't have all the redundancy they said they had. Their central server management system and the DNS servers for those hosts were all in that data center. So customers couldn't get in and switch the DNS to another location for hours.
They now claim to have the server management system back up.
Andrew, your comment is one of the most asinine I've encountered on slashdot.
DON'T YOU DARE sound self-righteous when an observer is making his point cogently and politely.
I'm sick of these obey-authority fools telling us what we are allowed to criticize and not criticize.
We all respect fire departments. That's a given. The fact that you should attack the messenger undermines the credibility of any point you have made.
I applaud your response, CFD. It's the responsibility of the data center management to ensure redundancy in the event of fire, flood, earthquake, tornado or curious squirrel. In this case, there was an explosion and fire. I've seen fire up close when we invited the local volunteer fire department to use an old house as training. Once they set the fire, the entire thing was engulfed in 26 seconds. They intended to enter and practice saving occupants but it was 60 year old, dry wood and just went poof, so they just waited a minute and turned the hoses on. It smoldered for about 24 hours and was a pile of ash.
In this datacenter, there are all kinds of things that could smolder and cause secondary fires if the generators were turned on and something unknown happened (i.e. short from inside?). Plus, don't firefighters need to check for hot spots? Isn't that easier if power's off? Don't get me started on structural stability, either... 3 of 4 walls collapsed. That has to count as added risk.
So yes, if ThePlanet was willing to take the risk that their building was destroyed by earthquake, they can accept 24 hours downtime at the insistence of the fire chief. Redundant data centers for critical operations; acceptable tactical losses for whatever doesn't have redundancy. Murphy's law happens, and nothing is truly redundant. If their 9,000 customers expected full redundancy, those customers will need to re-evaluate what exact kind of redundancy they're getting. Not everyone needs multi-datacenter stability which is horrifically expensive. After reading this story, I'll be getting a second server for my 19 domains, on a different provider in a different city. Just in case.
Since it was posted AC the second time, it may have been someone else pretending to be him. You never know...
Otherwise, I'm curious as to the probability of whether he meant that (and completely missed Hijacked Public's point), or is just trolling.
Why OpalCalc is the best Windows calc
As previously committed, I would like to provide an update on where we stand following yesterday's explosion in our H1 data center. First, I would like to extend my sincere thanks for your patience during the past 28 hours. We are acutely aware that uptime is critical to your business, and you have my personal commitment that The Planet team will continue to work around the clock to restore your service. As you have read, we have begun receiving some of the equipment required to start repairs. While no customer servers have been damaged or lost, we have new information that damage to our H1 data center is worse than initially expected. Three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed. There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running. We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we're expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production. Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues. We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do. We understand that you will be due service credits based on our Service Level Agreement. We will proactively begin providing those following the restoration of service, which is our number priority, so please bear with us until this has been completed. I recognize that this is not all good news. I can only assure you we will continue to utilize every means possible to fully restore service. I plan to have an audio update tomorrow evening. Until then, Douglas J. Erwin Chairman & Chief Executive Officer
30% off web hosting. Coupon code "SLASHDOT".
As much as I agree with minimizing safety risk, his main point (in my opinion) was:
"So cut the feed from the power company, case closed. Shutting down the redundant power generators that are DOWNSTREAM from the problem?"
In other words, surely if the backup power *to* the servers is kept on, and just the primary power is turned off (i.e. from the power company), then that's surely 99.999% safe? After all, the room for the main power is seperate from the backup power's room.
Maybe I'm missing something, and the power can be leaked from the backup power to the servers, and then finally to the broken main power setup. I doubt that, but at least they should ask the server's technicians if that's even theoretically possible.
Why OpalCalc is the best Windows calc
Mind you that the fact that ThePlanet own 5+ datacenters lends credit to what you're saying.
Put your servers in two geographically-isolated datacenters, and you'll be considerably more protected against virtually any sort of calamity that could occur to your servers.
There are so many things that a datacenter simply cannot be 100% prepared for. Would you really blame the provider if a plane crashed into their building?
It's far cheaper to simply colocate in 2+ locations than it is to prepare for every single event that can possibly occur, no matter how remotely unlikely it may be.
-- If you try to fail and succeed, which have you done? - Uli's moose
Wonder if there is a really stupid ass moderation for people who mod without a clue?
The point is that the this type of thing could be avoided with a colocated system. Not that ThePlanet would be doing it with every one of their systems? The point is if the owner of the system hosted by ThePlanet felt if the expense to colocate was worth it to avoid downtime, it could have been done.
firefighter != electrician
Electricians don't work in areas where an explosion just happened and some wires may or may not be live.
Both, electricians and firefighters would shut down the power. Electricians may take their time to understand the situation. Firefighter have no time. So in case of fire and shit, firefighters will demand power is shut down before going in, especially in cases of ELECTRICAL FIRE!!
How stupid can you be? Electricity was what fueled the explosion and/or fire! So you shut it off! All of it. You don't know what is its state, so you shut it down. You know, electricity "flows" from one end of a transformer to another *both* ways. And hell, I would not trust some relay switch with my life after an explosion. Hell, the railing or stairs or support column could be live and who the hell knows what is live and what is not??
You are like some idiot bitching that there their natural gas supplies were cut because there was an underground pipe explosion 2 miles down from your house.
At first I thought you must know me to use my name, then I realized it was just a cheap trick of looking at my email address or profile.
You know how I knew? Because nobody who knows me would make the mistake of calling me an obey-authority anything. You've got the wrong fool on that one.
The arrogant insult dripping through the trolls post to which I responded deserved all the ridicule and self righteousness it contained.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
bah! the feds, the cia, the chinese, the....
who would use a short!!
We're under attack!
Declare War on [insert free power source here]
Hackers have long memories. It works both ways.
While it sounds like a reasonable approach at first, it makes assumptions that I can't make as an officer on scene.
1. It assumes that the only problem is with the original transformer. When I arrive on scene I don't know what the problem was -- even if you tell me you do know, I can't believe it. I also don't know what the secondary problems are.
2. Feeding power into a building that has been physically damaged is very very dangerous. We're not talking about a transformer "failing to work" we're talking about something that blew the walls off the room it was in.
3. We already know that things didn't go the way they were supposed to. Something failed. Some safety plan didn't work. We have to assume that we're dealing with chaos until proved otherwise.
So, as a fire officer I arrive on scene and have a smoke filled building with reports of an explosion and MAYBE a report that everyone is out. I need to go in and find out what happened, if anything is still burning or in immediate danger, and if anyone is still in side. To do that safely, the first thing I want to do is secure the power to the building (shut it off) as well as any other utility feeds (oil, steam, liquefied petroleum or natural gas).
The gear I carry -- even the radio -- is designed to never create even the tiniest spark in its operation. We call it "intrinsically safe". Its one of a great many precautions we take.
We go in to a place like this not knowing the equipment, not knowing its condition.
My final proof point --
If in fact The Planet had powered up their generators, they'd have fried a lot more stuff and caused more fire. The may have destroyed their chances of salvaging the grid within 48 hours at all. Why? It turns out (we now know) that the force of the initial explosion moved three walls in the power distribution center more than a foot (I heard 3 feet I think) off their base. This tore out electrical connections, cables, conduits and power switches. Just now, after 28 hours, they've figured out how to get power to the servers on the second floor, but for the first floor servers they're having to rig up a line from the generators to that floor and it will take until tomorrow to do that. Why? Because the electrical connections from that distribution room to the first floor servers are destroyed. They're going to be running 3000 servers on the first floor off those generators for a week while they get the equipment to rebuild the connectivity to the main distribution room.
What does this prove?
1. It proves the fire marshal was right in not allowing them to feed power in their.
2. It proves that when that big dumb fireman you see (who may be a volunteer who's also a network guy and software developer with an IQ above 95% of the world) may in fact have a good reason for the way they do things on scene.
Look, as a firefighter I don't set out to ruin someone's day. I set out to keep them safe. If that sounds paternalistic, well, It is paternalistic. It very much feels that way. In my small town, its how I feel. I wonder ever time I walk into a building, how I would protect MY PEOPLE in this building if a fire broke out or a hazmat incident started or whatever. You can't help it, its what you're trained to do.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
You are probably thinking of auto insurance. Yes, it usually goes up when used. The reason is because when you use it, it is usually because you did something that changed your risk level. If you get in an accident, that makes you a higher risk. Continue to get in accidents, you are a higher risk still. Thus the companies want more money. It's all based on risk calculation. That's also why they want more money when you are under 25. Statistically speaking, young people are a much higher risk of accidents.
Well with building insurance, that's not the case. You aren't really a significant risk factor. Risk is instead calculated of of things like what kind of structure it is, how far it is from the fire department, what it's used for, what it contains (that determines what they are on the hook for) etc. So when something happens, unless it was because of a previously unknown risk factor, your rates don't necessarily change. Nothing changed with regards to risk.
Insurance is really all just risk based. They take the probability of having to make a payout and the amount of said payout vs time and come up with a rate. If something changes the risk, the rate will change as well, but if not then it doesn't change. It isn't as though your one single payout is of any significance to their overall operation.
Also, the idea of "Just pay for it yourself," is extremely silly. It smacks of someone who's never owned something of any significant value. The reason behind insurance is that you CAN'T just pay for it yourself. For example I have insurance on my house. The reason is that if I lost it, I can't afford to replace it. I don't have a couple hundred grand just lying around in the bank. That's the point of insurance. You are insuring that if something happens that you can't afford, someone will pay for it. The insurance company is then, of course, that it isn't likely to happen and they get to keep the money.
someone mod this up please.
:(
I'm in DC1 at the planet, I'm down and I'm not pointing any fingers. I'm pretty sure they did a reasonable job of setting up their systems in such a way that the chances of this happening was small to begin with, and when it did happen they seem to have things under control as much as possible.
There are a lot of 'armchair' specialists and complainers around here and all I would like to say to them is we'll see how it goes when *you* operate 5 large datacenters for years. Accidents do happen (and by their nature are caused by the unforeseeable), how you deal with them is what matters.
And fire fighters lives are more precious than *any* amount of hardware.
The only person I blame for not verifying if what I thought was redundant DNS in two locations is me, and I really thought I had it set up that way
MP3 Search Engine
For example someone like Newegg.com probably has a redundant data centre. Reason being that if their site is down, their income drops to 0. Even if they had the phone techs to do the orders nobody knows their phone number and since the site is down, you can't look it up. However someone like Rotel.com probably doesn't. If their site is down it's inconvenient, and might possibly cost them some sales from people who can't research their products online, but ultimately it isn't a big deal even if it's gone for a couple of days. Thus it isn't so likely they'd spend the money on being in different data centres.
You are also right on in terms of type of failure. I've been at the whole computer support business for quite a while now, and I have a lot of friends who do the same thing. I don't know that I could count the number of servers that I've seen die. I wouldn't call it a common occurrence, but it happens often enough that it is a real concern and thus important servers tend to have backups. However I've never heard of a data centre being taken out (I mean from someone I know personally, I've seen it on the news). Even when a UPS blew up in the university's main data centre, it didn't end up having to go down.
I'm willing to bet that if you were able to get statistics on the whole of the US, you'd find my little sample is quite true. There'd be a lot of cases of servers dying, but very, very few of whole data centres going down, and then usually only because of things like hurricanes or the 9/11 attacks. Thus, a backup server makes sense, however unless it is really important a backup data centre may not.
after an explosion you simply can not assume that the original wiring diagram is still matching reality. Any discrepancy between the two translates in to a serious elevation of the risk...
In other words, what you think is a 'dead' wire could easily be a live one because one or more cables that used to be insulated are now connected.
before having inspected the situation and seen that things are good you're better off not risking powering up.
In fact, now that some of the dust has cleared up it seems that the damage was in fact much more serious than was assumed initially, and powering up the servers using emergency generators would not have mattered one little bit (whether it would have worked or not, or even made matters worse is another matter).
MP3 Search Engine
It's over 9000!!!!!
never before was 'anonymous coward' more appropriate.
if you're sick of these 'obey authority' fools telling you what you are allowed to criticize and not criticize I suggest you set up your own commercial firefighting service under your new and enlightened guidelines.
If you can get so much as 1 single person working for you under those guidelines I'll be very amazed.
Attacking the messenger is perfectly acceptable if the messenger tells you how to do your job for you and what risks you should take to save their property. No amount of property is worth the life of a firefighting crew.
Btw, we lost three firefighters in a flashover nearby recently, they were in fact 'just' trying to save some property.
MP3 Search Engine
Remember the Computer on Fire function in the BeOS kernel?
How will it be affected by the extended fireball and what are the ramifications of a positive or negative response.
or if it still looks like this.
I wonder if that's why World of Warcraft is down?
So hstntx1 exploded... That datacenter is located in Greenpoint (Guns point), Houston.
I click the link and it DOES bring up the page. Unfortunately, since it is a cached copy of the page, it is sometimes out of date. I.e., there have been updates to the actual page that are not reflected in what the Coral Cache copy displays. :/
As of this writing (Monday morning, 06/02/08), it appears that ThePlanet Datacenter folks have created a NEW STATUS PAGE to lessen the load on their servers:
Insurance companies around here have generally looked at two things when I was applying for home insurance:
a) Have I been a previous customer of them or their affiliates (discount points)
b) Have I been a customer of other companies *without a claim* (discount points based on claimless time)
You can get further points by things such as having security bars on windows (anti-theft), a home alarm system, fire extinguishers, properly placed fire alarms, etc etc.
Your overall discount is then based on your final number of points.
So, while your "base rate" doesn't change much based on this, the final price can vary quite a bit depending on which discounts you're eligible for, with claims-free time being one of the factors in this.
The point I think everyone is missing is.. If their power is down then who are they using to post the updates. I think everyone affected by this outage should use that hosting co as their provider. lol!
IT'S OVER 9000!!!11
Hack the Planet!
Showing my age
Did they find Earth? Tell them to turn around. We'll likely blow them outta the sky before they had a chance to talk.
Given that they shut down under orders from the fire department and that an explosion that knocked 3 walls down didn't damage the batteries, I'd guess they have more than one power room. They have a fair portion of machines powered up now. Since it would take longer than that clean up after an exploded transformer and install a new one in it's place, that room must not have been a single point of failure.
Honestly, it doesn't matter how well isolated your power systems are from each other, if there is a fire, particularly one with explosions, the fire department WILL order all power shut down. If you don't do it, they will (perhaps not so nicely). You will not be permitted to restore power at all until they are satisfied that it is perfectly safe. They will not be the least bit concerned with your uptime. Their priorities will be 1. nobody gets hurt and 2. no more fires.
Believe me, I am well past sick and tired of advertising that is synonymous with a big fat lie, but I don't think this is an example of it.
(With all due respect to the brave men and women of the fire department, couldn't resist)
Ray: "Everything would have been fine if dickless here hadn't shut off the main power grid!"
Walter Peck: "These men caused an explosion!"
Mayor: "Is this true?"
Peter Venkman: "Yes, it's true... this man has no dick."
--Coming up with something clever... please wait...
Guess you look a little like a fool, eh? Not only did you design a single point of failure into your system with DNS, but you ranted against not turning on the generators as being, "right up there with the voodoo and witchcraft method of personal protection," and it turns out that powering up the generators would have indeed made things worse.
Unless the data center was designed to explode, clearly we were in a situation where things were not operating as designed. Trying to figure out what's safe in the middle of an event where you're already in unknown territory isn't very easy.
This has been handled very poorly. I don't understand the "praise" for communication.
1. The first notification went out 6hrs after the incident.
2. The update messages didn't provide any real info - just "we're continuing to work on restoring power."
I found out about the plan to get the 2nd floor up and running on another site hours before the Planet communicated that. There were messages that "more important" customers were having their servers moved to the second floor to be brought back up sooner, as well as other message boards of "important customers" having their machines moved to other locations. Aren't all customers important?
3. The tech staff was virtually useless - precanned responses.
When I first learned of the issue, I asked about switching machines or changing DNS and the tech support person said I was taking a gamble as they'd be up by noon Sunday. I switched DNS as I didn't want my sites down any longer.
The next day when I asked again whether it was better to just get a new machine, they agreed, but the best part is their offer was for a more expensive machine with less features than the one I just got 2 weeks ago. They wouldn't even match the specs of the machine or the price - what kind of customer service is that?
Whoever out there thinks The Planet is handling this great is crazy.
At this point as it closes in on 2 days downtime with really no vote of confidence in being fully restored, they probably could have packed all the servers up and moved them to other locations during the same time period.
I too will dump these guys as soon as I get all the data off.
Anyone with recommendations on server solutions should post them to these forums.
Open Source Java DAO Generator
Ever tried to do that? Unless you are a life critical operation like an intensive care unit they aren't likely to be all that interested and anything they might agree to is non-binding.
As for running on backup generators, as long as they are willing to sustain that long enough to replace the transformer, what's the problem? They have demonstrated the ability and willingness to provide power in the event that the transformer and power room are blown to bits.
Once a quarter when they claim they test their backup systems they should have a fire engine standing by and have an inspector or chief there too and walk them through what's happening. That way if there's a real disaster, the fire dept either payed attention and knows you're backup system is isolated from the main power system or you've showed them some sort of competence during your drills that they believe you if they didn't pay attention. As for running on backup generators, as long as they are willing to sustain that long enough to replace the transformer, what's the problem? They have demonstrated the ability and willingness to provide power in the event that the transformer and power room are blown to bits. The problem is they didn't have a fully redundant power system without a single point of failure like they claim. It took them more than 40 hours to start powering systems up. This is not what most people would expect from a "world class data center" that's capable of reliably handling your IT infrastructure.
Open Source Java DAO Generator
Hm, the fire department is always poking around our power equipment. Hope they are actually looking for something, or know what to look for, or are they just poking around for the hell of it?
Xconomy was one of the sites hosted at "H1," as The Planet calls it. After waiting all day Sunday to see whether we'd be back up on Monday, we decided to move the site back over to Media Temple, our previous hosting provider, at least temporarily. (Ironically, one of the reasons we left Media Temple in the first place was that they couldn't handle the traffic when our flying car stories got slashdotted.) We published a post about our experience with the outage this morning.
I'm not going to fellate you because you are a firefighter, sorry. There is no such thing as a sacred elephant and there are pros and cons to everything.
As a previous posted accuratly points out, we have been forbidden from taking our own actions when we consider the risks and costs to be worth it. I would have no problem if the choice to hire a firefighting service and understand that the more the risk, the more it will cost me. What I have a problem with is there being a monopoly of only ONE legally allowed provider, and then having that provider refuse service.
I can't stand the way if you criticize troops or firemen, you are considered flamebait. It's a volunteer service. I know plenty of macho asshats who sign up specifically because they know they are basically going to be beyond reproach if they make it in.
People risk their lives every day. I already pointed out that the average electrician (who makes a journeyman's wages of about $25K) faces more danger and risk in his daily job than a firefighter may face in a week. Hell, someone manning a counter at a convenience store in a bad neighborhood has a chance at getting shot.
You aren't special. You aren't magical. You aren't beyond reproach. You don't have a monopoly on heroism. I can point to countless tales of ordinary people who risked their lives or even died trying to pull someone from a canal or even a burning building.
In fact, I have more respect for the average joe who does it than the person who collects a paycheck to do it.
And if you don't like it, then I expect you to support the privitization of social services so that if some chief decides that the 0.000001% of death is an unacceptible risk for saving property, I can pick up the phone and call someone who will.
PS for SOG that are "very specific" about risk, you have some pretty vague and unspecific terms like "reasonable" and "minimal"
-JoeShmoe
.
-- I wonder which will go down in history as the bigger failure: the War on Drugs or the War on Filesharing
OK so it sounds like an easy way to kill everyone at a major metropolitan hospital is to set off a smoke bomb in the main eletrical room because then they will unquestioningly shut down ALL ELECTRICITY including the backup generators that are keeping people on life support alive.
What, that's stupid, you say? Well so is your blanket assertion.
And you are only confirming what I said... firefighter != electrician which is why I used the term IGNORANT. Regardless of the facts and 20/20 hindsight in this particular scenario, the question remains...is it considered standard operating procedure at ANY hosting facility to shut down all power, INCLUDING backup power, regardless of the size or nature of the threat?
Or was this an accurate and analyzed response and not just a knee-jerk reactions. Because otherwise I hope the terrorists don't learn we are ten smoke bombs away from having our entire telecommunications infrastructure turned off to "avoid risking firefighters"
-JoeShmoe
.
-- I wonder which will go down in history as the bigger failure: the War on Drugs or the War on Filesharing
Because you probably wouldnt accept what I said, here is an article with some interesting stats:
http://money.cnn.com/2006/08/16/pf/2005_most_dangerous_jobs/index.htm
I don't see firefighters on that list. But I do see electrical workers.
Who's waving the flags for them?
-JoeShmoe
.
-- I wonder which will go down in history as the bigger failure: the War on Drugs or the War on Filesharing
Actually the fact that they have to run off generators shows they did not have N+1 redundancy in there power feed. Reading though the lines it looks like they had a single power feed into the building then generators/UPS after that. I've designed and built "world class" data centers and data centers used by hosting providers don't get the two confused. Hosting is cost centric business in general and insurance is cheaper than gear. Now the funny bit about world class data centers is they are still expected to fail. Services running out of them are hosted at 2 locations if at all possible and a primary and backup set of gear per location. It's all rather expensive but thats how you get to 5 nines or better. Nothing will help you if you have a single DC and the fire trucks roll in and start cutting power.
No sir I dont like it.
Its really sad and funny.
By the way, Last year I earned less than $2000 as a firefighter. We're volunteers (or in the case of most, the term is sort of a misnomer, we're paid a minimal amount of money to keep some legal requirements met by our tows).
Here's some numbers for you - I believe these are a year or two out of date, but I'm not going to look for newer:
Of US firefighters, ~300,000 are full time career firefighters, while ~800,000 are your neighbors who have regular jobs and respond when called.
Of US fire departments, 86% are all call-responders (volunteers) while 92% have at least half call-responders. Last I knew, FDNY had at least one call-responder station out on Long Island, but that may be out of date.
I did not and do not ask to be held up as a shining knight of irreproachable perfection. It wouldn't fit well anyway. I did ask that you not ridicule and insult an honorable vocation and the people who, like me, spend hundreds of hours a year training to be aware of how to deal with emergency situations ranging from a toaster oven fire to a train derailment with toxic chemicals or a data center fire with massive hazards.
So no, I'm no Bruce Willis. I'm a network guy, a business owner, and a software developer -- and a volunteer firefighter who spends as much time training for that field as in computers and technology. You may be surprised, but both are extremely technical fields.
Your statements do not accurately reflect the real danger of the situation in general, nor did they reflect a solid understanding of this incident in particular. You seem to think this was a transformer outside the building and that generator power could be applied through the generator at minimal risk to the firefighters and the workers in the building. That just isn't the case here, and usually isn't.
Finally, you are prevented from doing things which can cause you harm in cases where -I- am obligated to save you, in cases where you endanger other people (including me), or in cases where you risk damage to other people's property.
At this particular scene, the walls of the room containing this transformer were blown several feet from where they'd been. Virtually all the power conduits and lines in that room were totally mangled. It took 28 hours for the electricians to repair enough of that wiring and infrastructure for the second floor power to begin to be restored -- with generators. I'm told a few hours of that was spent waiting for the return of the fire marshal to inspect and certify the work as safe. It has taken almost 20 more hours to create a temporary new power distribution infrastructure to manage the power to the first floor. I'm hearing reports that people's 1st floor servers are starting to come online -- several hours ahead of expectations as laid out last night. The second floor will be on regular power soon if its not now, but the first floor will have to run on those backup generators for a week, while the entire power grid for that floor is re-built from scratch before it can be connected permanently.
So, it seems the fire marshal was right. Also, it seems that it is very unlikely if he had not required it be kept off line that TP would have brought the generators up without first inspecting the damage, and as soon as they had they'd have known it would be impossible.
Your frustration at having your servers off line has lead to you declaring that they are worth more to you than the lives of human beings. That, I find disgusting.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
SOG's are guidelines and not procedures or rules for reasons like what we're arguing about.
Take any specific incident and we can pick apart its details -- especially in the light of day with more facts. A firefighting crew - arrives on scene knowing only that there's something really wrong. You may have heard "explosion" or "fire" but often what you hear en-route is dead wrong. You don't know what caused the explosion. You don't know what exploded. You don't know if people are trapped or injured. You don't know if the riped and stripped wiring is carrying enough voltage to kill you when you touch a chair leaning against a wire you didn't see. You don't know if the explosion was from a gas leak and there's more gas leaking now. You don't know what toxic chemicals are in whatever blew up. These days, we're also trained to think explosions may be bombs. In that case you may have a secondary bomb -- people who set bombs like to make a small first one that draws police and fire, then a large second one that kills them.
You go into a dark building carrying every kind of tool you can carry to deal with whatever variety of broken thing you may find. One group is doing a search for people. One group is doing a search for fire - or other hazards. Fire may look out but be in the walls, or overhead in the drop ceiling. You can't tell.
In Price William County, a very well trained crew entered a house where fire was visible on the outside back wall. It was before 7am, there were cars in the garage and nobody on the front lawn to say if anyone got out yet. They made the second floor and found temperatures not over 90 degrees and a light haze in the air. We call that a tenable environment so we search. They got down the hallway when the fire dropped down on them from the attack space. One of the two up stairs go t out, the other didn't. The reports say the temperature at face height in that upstairs hallway went from 90 degrees to over 700 degrees in a few seconds.
I do my work in a small town hundreds of miles away. Still, we studied that incident like we study any other where men are killed. In most cases, they're killed because something got a lot more dangerous than it looked -- even to trained firefighters -- very very fast.
This is why I ask you not to insult firefighters by pretending to know what is and isn't dangerous without the years of training and practice that go with it.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
If its me doing the poking, I probably have a good idea what I'm looking at -- and actually that would make it harder to do my job as a firefighter because I'd be too interested in the stuff.
What they're looking for at the basics of good electrical line management. No blocked vents, no exposed wiring. No extension cords through walls (common), no extension cords used as permanent wiring (common), no extension cords coiled up and flowing power (heat builds up and they catch fire), no chains of power strips -- and so on.
Also, are the fire doors operational and not blocked open? Are the sprinkler or other fire suppression systems in order? Are exit signs lit and accurate? Are emergency battery lighting units charged and ready?
They don't care if you stuff doesn't work. They care if you get trapped because the fire code stuff isn't right.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
sigh. best practice DNS is to have the servers on distinct networks -- you had your two servers hanging off the same switch didn't you?
RE fire chief you are also wrong. All fire suppression systems are linked to the power system and kill all power when the fire suppression system is triggered (so that the fire fighters are not electrocuted while fighting the fire). The tech you talked to on the phone that gave you the info your rant is based on likely knows very little about how the data center _facilities_ are operated (they only support the software). The NoC and data center are very different places and the techs have different skills and experience that staff them.
some pics a friend sent me from his ventrilo providers site who hosts with the planet... looks pretty intense
http://www.pure-voice.net/forums/viewtopic.php?t=608
There was more disruption than to the servers in H1. NewsBlaze.com is in another datacenter, but it seems both NS1 and NS2 are in the same place, in H1. So even though our webserver equipment wasn't involved, we were down too. Depending on DNS cache times, traffic slowly dropped off to just a trickle and then eventually nothing. And it took them a long time to get that fixed, a lot longer than it should have. They never apeared to listen to what I told them when I called - and I tried hard to get the support staff to pass the message along. This is one of the problems with big companies - they don't listen to their customers at the times they most need to. We couldn't have been the only ones in that situation. It would be interesting to know how many servers outside H1 were affected. Thank you to the few slashdotters who visited the NewsBlaze story before the server became inaccessible. I'll be writing more about this. The Planet Houston Data Center Goes Up in a Puff of Smoke
Daily News http://newsblaze.com
What single point failed and took everything down? They had TWO failures, one was the explosion and the other was a shutdown order from the FD.
Considering that there was an electrical explosion, the latter is only to be expected. Had they somehow not done that, someone (quite probably you) would be griping about how they risked sending 9000 servers and a dozen employees up in flames just to help their stats.
Funny thing about explosions, things move around as a result (cables, conduits, walls and parts thereof). There WAS after that a non-zero chance that a new hazard developed and that it could have threatened human life. No amount of redundancy would have avoided that. This isn't a spacecraft, it's a data center. It's not like the threat to life from a shutdown was as great or greater than the threat of not shutting down.
Feed into the building, no they did not, but so what? Into the servers, yes they did, as advertised.
HEY! Thanks for that! I just recently stopped making an explicit link in posts here because they seemed to automagically turn into a link anyway. I just figured it was some /. feature.
No idea how long it'd take for me to discover it was my
linkification
Firefox
addon
unless you had pointed it out! Thanks again!
The point is redundant power feeds is a fairly cheap and common practice. They did not seem to have that in there world class data centers. They put themselves up against IBM datacenters and the three of those I've worked at all have n+2 power feeds into the building from multiple substations.
No sir I dont like it.
Life support gear is, as I understand it, built with battery powered redundancy and regularly tested. I don't work with that gear, but I believe it is the case.
I also believe that there are some circuits in hospitals which are specially labeled and are not shut off unless absolutely proved critical or already damaged. In these cases, a lot of money is spent building safety conduits for their cabling and other precautions so that they can handle major damage to the building without becoming a hazard.
Hospital emergencies are their own unique events and there are pre-plan documents and procedures in place for dealing with exactly the issues you describe.
Finally, I would point out that many firefighters are in fact electricians. You see, even career firefighters are not paid well, and most have a second job. Of those, a majority are in the construction and or contracting trades. It is a good fit for them.
People misunderstand the role of a firefighter thinking we just show up and put water on things that are hot. Surely that's the fun part of the job.
In reality, we also have to be many many other things. We have to be truck drivers (you ever drive fast in a 40,000 pound truck carrying a thousand gallons of moving liquid?). We have to be experts in building construction. We have to understand electrical work. We have to be certified in hazmat operations. More recently, we also have to be certified in NIMS (National Incident Management System) which allows us to inter operate using the same language and procedures. We have to be experts at high and low angle rope rescue, confined space rescue, below ground rescue, first responder medical support, mechanical rescue (man vs. meat grinder), traffic control, flood control, crowd control, tree removal, bees, wasps, & snakes, vehicle rescue (we don't take patients from cars, we remove cars -- in chunks -- from around patients) and anything else risky or scary you might want help with.
Even plumbing and water supply -- Just imagine showing up on a scene without a fire hydrant for miles, and being able to organize a tanker shuttle, dump tanks, pumpers, and lines to supply more than 1000 gallons of water a minute within 5 minutes of arriving on scene. That's enough water to fill your pool faster than you can fill your bathtub.
A firefighter crew is a small group of men very much like Macguyver (not as smart maybe but better equiped) with every kind of tool imaginable that they can carry around with them (especially on a heavy rescue unit). You put these guys into ANY situation and within seconds they'll organize around a safe plan for getting to the best possible resolution with the least risk to life and property.
I think the only ignorance I see here, is that which you are demonstrating in your examples.
I'll give you credit for one thing, however. When you state that you are not going to fellate me, you are 100% correct. No matter how nicely you ask.
The problem with quotes on the internet, is that nobody bothers to check their veracity. -- Abraham Lincoln
I was just speaking with Mark Causa, a forums admin of his, this weekend in fact!
(Kevin Hazard's their "SUPER ADMIN" in fact).
(It was in regards to a "IPS Driver Error" I was CONSTANTLY seeing on a posting of mine there, in an attempt to update/edit it, on THEPLANET's forums (in regards to securing Windows))...
WoW! I was trying to point them to security issues too... & they were VERY helpful guys too, trying to help ME out (& going overboard imo in some ways)
I was also today, in fact, prior to seeing this - going to note they were being listed as a site that had problems with hacker/cracker types abusing them as well, per one of these sites:
http://www.castlecops.com/
http://mtc.sri.com/
http://www.spamhaus.org/sbl/latest.lasso
http://www.phishtank.com/
(or, one of the numerous others I look @ daily, like SANS, PacketStorm, etc.)
They were listing theplanet as being abused etc. the past few weeks now in fact, by hacker/cracker/spammer types.
APK
P.S.=> I doubt this is due to "hacker/crackers" though, personally... just bad setup in the server room! apk
I understand that, but I don't see where it matters one whit so long as they have some means to provide power with the main transformer blown up (in this case diesel).
What that amounts to is they decided that the extra diesel would cost them less than the second grid tie. Perhaps they were right, perhaps not, but their duty to the customer is to somehow (barring orders civil authorities) be able to provide power to customers when the primary is down. They are doing that now.
Did you even bother to read the comment you just replied to? It wasn't the same person as the author comment you originally ranted to; on the contrary, he agreed with you.
Your first post was quite reasonable. This one made me think you're an asshat who thinks he's entitled to decide everything that might remotely affect him, and *not* just in your job.
So sorry to do this, but it must be done....
IT'S OVER NINE THOUSAND!!!