I have found that Open Source has become an important market force pushing proprietary software vendors harder to create more reliable, more compelling, and overall 'better' products.
The ecosystem has evolved fairly well in many cases. IBM is an interesting case study. They've spent what to most of us would be considered a dozen fortunes (and is to IBM a pittance) supporting the Eclipse platform. They still have many full time IBM employees who are dedicated to working on this entirely free and open source project. They do this because it is to IBM's benefit for that project to exist and to be a good tool. By donating resources, IBM gets input into how it goes together. IBM then uses that open source project as part of some of its solutions which support their other proprietary software. Its working well for everyone so far as I know.
The third party tools market can be good or bad depending on many things. Open Source is a very positive market force in getting rid of the over priced garbage put out by a couple of mid-skilled programming hacks in a couple of afternoons. I find a real scourge of these attached to the Microsoft development community. It has always seemed as though every idiot and his brother has some neat little tool they badly cobble together and then sell for a ridiculous price and don't support. For those tools, Open Source is a death sentence -- and we're all the better for it.
Open source isn't replacing the software vendor. High quality vendors still do exist and are thriving. The games industry, operating systems (yes, I love Linux too - but its not going to replace all desktops any time soon) vertical markets, and other very highly specialized or highly complex and long term development projects are well served by the professional software industry. They require highly skilled professional programmers -- many of whom learn and practice their skills in the much more dynamic and creative -- and more important, peer critical -- open source arena.
1st -- it isn't just "fire out" its "scene is safe" which may include hazmat cleanup or other hazards.
2nd -- Sometimes we have to stay and keep people out for a police investigation. Not often, they usually do that but we have lots of fancy lighting and such that can help.
3rd -- Often, we do much more. For example, in a house fire we may have put a hole the roof to vent, pips may have burst or broken (or their solder melted, as is frequent), windows are broken, etc. For a home, we usually stay and try to get boards up to keep the weather out. We usually also stay (in winter) to get a plumber on scene to get the heat going where possible, or the rest of the pipes drained when it isn't. That keeps them from freezing and creating more damage.
4th -- sometimes we have to stay on scene overnight for a fire watch, especially if the alarm systems are damaged or if there was a very large fire and we're watching for hotspots.
5th -- Once we're finally done with the "Emergency" part, the law in most states requires that the structure be inspected and occupancy requirements are met just as if it were new construction.
While "YOU" may want to go back into "YOUR" building, that's a long way from a public building or a building where employees have to work. You can't just order your staff back into a building that hasn't got an occupancy certificate or is dangerous -- or tell them they don't have to go, but then they're fired.
"Forcing myself upon the buildings owners" is really just a rather nasty way of saying you don't like the permitting, building code, and other such laws.
A fire or explosion essentially makes any previous occupancy permit or code enforcement inspection essentially unreliable. In some states they become instantly invalid. In all states, they should. OSHA would never let you put people back to work in a building that can't pass code inspection. That's basic workers rights.
The word "Owner" is the problem. I seriously doubt the "Owner" of that data center was on scene demanding to be allowed to personally start repairs in HIS building.
Life support gear is, as I understand it, built with battery powered redundancy and regularly tested. I don't work with that gear, but I believe it is the case.
I also believe that there are some circuits in hospitals which are specially labeled and are not shut off unless absolutely proved critical or already damaged. In these cases, a lot of money is spent building safety conduits for their cabling and other precautions so that they can handle major damage to the building without becoming a hazard.
Hospital emergencies are their own unique events and there are pre-plan documents and procedures in place for dealing with exactly the issues you describe.
Finally, I would point out that many firefighters are in fact electricians. You see, even career firefighters are not paid well, and most have a second job. Of those, a majority are in the construction and or contracting trades. It is a good fit for them.
People misunderstand the role of a firefighter thinking we just show up and put water on things that are hot. Surely that's the fun part of the job.
In reality, we also have to be many many other things. We have to be truck drivers (you ever drive fast in a 40,000 pound truck carrying a thousand gallons of moving liquid?). We have to be experts in building construction. We have to understand electrical work. We have to be certified in hazmat operations. More recently, we also have to be certified in NIMS (National Incident Management System) which allows us to inter operate using the same language and procedures. We have to be experts at high and low angle rope rescue, confined space rescue, below ground rescue, first responder medical support, mechanical rescue (man vs. meat grinder), traffic control, flood control, crowd control, tree removal, bees, wasps, & snakes, vehicle rescue (we don't take patients from cars, we remove cars -- in chunks -- from around patients) and anything else risky or scary you might want help with.
Even plumbing and water supply -- Just imagine showing up on a scene without a fire hydrant for miles, and being able to organize a tanker shuttle, dump tanks, pumpers, and lines to supply more than 1000 gallons of water a minute within 5 minutes of arriving on scene. That's enough water to fill your pool faster than you can fill your bathtub.
A firefighter crew is a small group of men very much like Macguyver (not as smart maybe but better equiped) with every kind of tool imaginable that they can carry around with them (especially on a heavy rescue unit). You put these guys into ANY situation and within seconds they'll organize around a safe plan for getting to the best possible resolution with the least risk to life and property.
I think the only ignorance I see here, is that which you are demonstrating in your examples.
I'll give you credit for one thing, however. When you state that you are not going to fellate me, you are 100% correct. No matter how nicely you ask.
If its me doing the poking, I probably have a good idea what I'm looking at -- and actually that would make it harder to do my job as a firefighter because I'd be too interested in the stuff.
What they're looking for at the basics of good electrical line management. No blocked vents, no exposed wiring. No extension cords through walls (common), no extension cords used as permanent wiring (common), no extension cords coiled up and flowing power (heat builds up and they catch fire), no chains of power strips -- and so on.
Also, are the fire doors operational and not blocked open? Are the sprinkler or other fire suppression systems in order? Are exit signs lit and accurate? Are emergency battery lighting units charged and ready?
They don't care if you stuff doesn't work. They care if you get trapped because the fire code stuff isn't right.
SOG's are guidelines and not procedures or rules for reasons like what we're arguing about.
Take any specific incident and we can pick apart its details -- especially in the light of day with more facts. A firefighting crew - arrives on scene knowing only that there's something really wrong. You may have heard "explosion" or "fire" but often what you hear en-route is dead wrong. You don't know what caused the explosion. You don't know what exploded. You don't know if people are trapped or injured. You don't know if the riped and stripped wiring is carrying enough voltage to kill you when you touch a chair leaning against a wire you didn't see. You don't know if the explosion was from a gas leak and there's more gas leaking now. You don't know what toxic chemicals are in whatever blew up. These days, we're also trained to think explosions may be bombs. In that case you may have a secondary bomb -- people who set bombs like to make a small first one that draws police and fire, then a large second one that kills them.
You go into a dark building carrying every kind of tool you can carry to deal with whatever variety of broken thing you may find. One group is doing a search for people. One group is doing a search for fire - or other hazards. Fire may look out but be in the walls, or overhead in the drop ceiling. You can't tell.
In Price William County, a very well trained crew entered a house where fire was visible on the outside back wall. It was before 7am, there were cars in the garage and nobody on the front lawn to say if anyone got out yet. They made the second floor and found temperatures not over 90 degrees and a light haze in the air. We call that a tenable environment so we search. They got down the hallway when the fire dropped down on them from the attack space. One of the two up stairs go t out, the other didn't. The reports say the temperature at face height in that upstairs hallway went from 90 degrees to over 700 degrees in a few seconds.
I do my work in a small town hundreds of miles away. Still, we studied that incident like we study any other where men are killed. In most cases, they're killed because something got a lot more dangerous than it looked -- even to trained firefighters -- very very fast.
This is why I ask you not to insult firefighters by pretending to know what is and isn't dangerous without the years of training and practice that go with it.
By the way, Last year I earned less than $2000 as a firefighter. We're volunteers (or in the case of most, the term is sort of a misnomer, we're paid a minimal amount of money to keep some legal requirements met by our tows).
Here's some numbers for you - I believe these are a year or two out of date, but I'm not going to look for newer:
Of US firefighters, ~300,000 are full time career firefighters, while ~800,000 are your neighbors who have regular jobs and respond when called.
Of US fire departments, 86% are all call-responders (volunteers) while 92% have at least half call-responders. Last I knew, FDNY had at least one call-responder station out on Long Island, but that may be out of date.
I did not and do not ask to be held up as a shining knight of irreproachable perfection. It wouldn't fit well anyway. I did ask that you not ridicule and insult an honorable vocation and the people who, like me, spend hundreds of hours a year training to be aware of how to deal with emergency situations ranging from a toaster oven fire to a train derailment with toxic chemicals or a data center fire with massive hazards.
So no, I'm no Bruce Willis. I'm a network guy, a business owner, and a software developer -- and a volunteer firefighter who spends as much time training for that field as in computers and technology. You may be surprised, but both are extremely technical fields.
Your statements do not accurately reflect the real danger of the situation in general, nor did they reflect a solid understanding of this incident in particular. You seem to think this was a transformer outside the building and that generator power could be applied through the generator at minimal risk to the firefighters and the workers in the building. That just isn't the case here, and usually isn't.
Finally, you are prevented from doing things which can cause you harm in cases where -I- am obligated to save you, in cases where you endanger other people (including me), or in cases where you risk damage to other people's property.
At this particular scene, the walls of the room containing this transformer were blown several feet from where they'd been. Virtually all the power conduits and lines in that room were totally mangled. It took 28 hours for the electricians to repair enough of that wiring and infrastructure for the second floor power to begin to be restored -- with generators. I'm told a few hours of that was spent waiting for the return of the fire marshal to inspect and certify the work as safe. It has taken almost 20 more hours to create a temporary new power distribution infrastructure to manage the power to the first floor. I'm hearing reports that people's 1st floor servers are starting to come online -- several hours ahead of expectations as laid out last night. The second floor will be on regular power soon if its not now, but the first floor will have to run on those backup generators for a week, while the entire power grid for that floor is re-built from scratch before it can be connected permanently.
So, it seems the fire marshal was right. Also, it seems that it is very unlikely if he had not required it be kept off line that TP would have brought the generators up without first inspecting the damage, and as soon as they had they'd have known it would be impossible.
Your frustration at having your servers off line has lead to you declaring that they are worth more to you than the lives of human beings. That, I find disgusting.
While it sounds like a reasonable approach at first, it makes assumptions that I can't make as an officer on scene.
1. It assumes that the only problem is with the original transformer. When I arrive on scene I don't know what the problem was -- even if you tell me you do know, I can't believe it. I also don't know what the secondary problems are.
2. Feeding power into a building that has been physically damaged is very very dangerous. We're not talking about a transformer "failing to work" we're talking about something that blew the walls off the room it was in.
3. We already know that things didn't go the way they were supposed to. Something failed. Some safety plan didn't work. We have to assume that we're dealing with chaos until proved otherwise.
So, as a fire officer I arrive on scene and have a smoke filled building with reports of an explosion and MAYBE a report that everyone is out. I need to go in and find out what happened, if anything is still burning or in immediate danger, and if anyone is still in side. To do that safely, the first thing I want to do is secure the power to the building (shut it off) as well as any other utility feeds (oil, steam, liquefied petroleum or natural gas).
The gear I carry -- even the radio -- is designed to never create even the tiniest spark in its operation. We call it "intrinsically safe". Its one of a great many precautions we take.
We go in to a place like this not knowing the equipment, not knowing its condition.
My final proof point --
If in fact The Planet had powered up their generators, they'd have fried a lot more stuff and caused more fire. The may have destroyed their chances of salvaging the grid within 48 hours at all. Why? It turns out (we now know) that the force of the initial explosion moved three walls in the power distribution center more than a foot (I heard 3 feet I think) off their base. This tore out electrical connections, cables, conduits and power switches. Just now, after 28 hours, they've figured out how to get power to the servers on the second floor, but for the first floor servers they're having to rig up a line from the generators to that floor and it will take until tomorrow to do that. Why? Because the electrical connections from that distribution room to the first floor servers are destroyed. They're going to be running 3000 servers on the first floor off those generators for a week while they get the equipment to rebuild the connectivity to the main distribution room.
What does this prove?
1. It proves the fire marshal was right in not allowing them to feed power in their.
2. It proves that when that big dumb fireman you see (who may be a volunteer who's also a network guy and software developer with an IQ above 95% of the world) may in fact have a good reason for the way they do things on scene.
Look, as a firefighter I don't set out to ruin someone's day. I set out to keep them safe. If that sounds paternalistic, well, It is paternalistic. It very much feels that way. In my small town, its how I feel. I wonder ever time I walk into a building, how I would protect MY PEOPLE in this building if a fire broke out or a hazmat incident started or whatever. You can't help it, its what you're trained to do.
At first I thought you must know me to use my name, then I realized it was just a cheap trick of looking at my email address or profile.
You know how I knew? Because nobody who knows me would make the mistake of calling me an obey-authority anything. You've got the wrong fool on that one.
The arrogant insult dripping through the trolls post to which I responded deserved all the ridicule and self righteousness it contained.
You sir, don't know what you're talking about. Reaching for ridiculous examples of someone doing their job wrong doesn't change that.
Our S.O.G. (standard operating guidelines) are actually very specific about risk.
We will risk our lives to save a human life. We will take reasonable risk to save the lives of pets and livestock. We will take minimal risks to save property.
Sorry, but your building isn't worth the risk of my crew. That's reality.
Don't you DARE tell me what is and isn't bravery or cowardly until you put 50 pounds of gear on and crawl into a pitch black house that's burning over your head.
Don't you DARE tell me that you think you understand the difference between saving the blonde girl and saving your computer server.
This isn't TV World. This is the real world. Fire on TV doesn't look like real fire. You know why? Because a real house on fire doesn't look like anything but pitch black and that makes for lousy TV.
Get over yourself and go volunteer at your local fire department. 86% of the men and women in this country who will risk their lives for yours are volunteers. We could use your help if you have the guts for it. We'll teach you what you need to know -- and we'll keep you as safe as we can so you can go home to your family when its done.
Your examples are stupid and insulting to the 800,000 brave men and women who volunteer to risk death in the most painful way possible to save your sorry butt.
Redundant power they have. Redundant power distribution grids they do not. This is common. The level of certification in redundancy on power for fully redundant grids is (I think) called 2N where they only claim N+1 -- which I understand means failover power. Its more than enough 99.9% of the the time. To have FULLY redundant power plus distribution from the main grid all the way into the building through the walls and to every rack is ridiculously more expensive. At that point, it is more sensible to buy another server at another facility for failover than to spend what it would cost to host a server with that kind of power redundancy -- on top of which, the server itself could still blow up and then where are you?
Look, when I go into a building in gear and carrying an axe and an extinguisher, breathing bottled air, wading through toxic smoke I couldn't give crap number one about your 100 sites being down.
I have a crew to protect. In this case, I'm going into an extremely hazardous environment. There has already been one explosion. I don't know what I'm going to see when I get there, but I do know that this place is wall to wall danger. Wires everywhere to get tangled in when its dark and I'm crawling through the smoke. Huge amounts of currents. Toxic batteries everywhere that may or may not be stable. Wiring that may or may not be exposed.
If its me in charge, and its my crew making entry, the power is going off. Its getting a lock-out tag on it. If you wont turn it off, I will. If I do it, you won't be turning it on so easily. If need be, I will have the police haul you away in cuffs if you try to stop me.
My job, as a firefighter -- as a fire officer -- is to ensure the safety of the general public, of my crew, and then if possible of the property.
NOW -- As a network guy and software developer -- I can say that if you're too short sighted or cheap to spring for a secondary DNS server at another facility, or if your servers are so critical to your livelihood that losing them for a couple of days will kill you but you haven't bothered to go with hot spares at another data center then you sir, are an idiot.
At any data center - anywhere - anything can happen at any time. The f'ing ground could open up and swallow your data center. Terrorists could target it because the guy in the rack next to yours is posting cartoon photos of their most sacred religious icons. Monkeys could fly out of the site admin's [nose] and shut down all the servers. Whatever. If its critical, you have off site failover. If not, you're not very good at what you do.
First, that time was an estimate -- a target. Second, even if the initial power test passes, it will take hours to bring up the a/c systems, the switches, and the routers.
The initial draw from each new bank of gear to be given power will be very high so it will need to go slow.
The battery systems (be they on each rack or in large banks serving whole blocks) will try to charge all at once. If they're not careful, that'll heat those new power lines up like the filaments in a toaster. Remember, the battery plan they have was built with the idea that they'd be used very briefly during transition to generator power -- not drained down all at once.
Only once all the switches and routing gear is back up can they start updating the network paths (do they use BGP for this -- that's not my area of expertise) so that peering data starts flowing.
Only once the network is all up and stable (no small task on a site with dozens of high end peering points) can they even start doing banks of servers.
Its also probably that each bank of servers will needs its own new power lines (and eventually replaced conduit) in the distribution center that was destroyed.
Bank by bank they'll have to bring up all these servers, each of which will draw its maximum load during boot as disks are scanned and checked.
Most of these servers probably haven't been shut down in months or years. Some drives may not spin up due to tired motors that can run fine but spinning from cold is just too much now. Other servers may have boot configuration problems undiscovered since the machines have been running without reboot for a long time -- linux ones anyway:-)
This isn't something out of Young Frankenstein where they'll yell across the room "throw za main svitch!" and a watch the lights dim briefly while 9000 servers boot up with the deafening sound of system beeps. If they did try such a thing -- as if such a thing were possible -- it would immediately blow at least another transformer if not more.
Think about it. 9000 servers @ an average of what, 300 watts, plus the networking gear, plus the air conditioning, plus charging all those batteries....you're talking megawatts.
Without a Mr. Fusion or Harry Mudd stumbling in with some chicks wearing dilithium crystal jewelery this is going to take a while.
What I do isn't just a web site, its also a pbx and some other stuff.
The client software that does the automation is easy. I wrote it to handle the need for a failover server so it will just try the other if the first one fails.
The PBX failover is easy, the DID provider will route to both, only one will pickup sooner. If the second does catch a call, it will have a flag on it that lets it know if the primary is still up and in that case will try to transfer over the call.
The Web Site is actually the least important part of the process for me, and I'll likely handle that with a low ttl on that one particular address.
The inbound mail is easy because I use Postini and it is good at failover.
The data is stored in a database that knows how to sync in near real time between the servers, and on disk as files. I use unison to keep the file directories in sync within just a few minutes.
Overall, I think it should work just fine. There are more elegant solutions -- and more expensive. This one will work for me.
My servers dropped off the net yesterday afternoon, and if all goes well they'll be up and running late tonight. At 1700PST they're supposed to do a power test, then start bringing up the environmentals, the switching gear, and blocks of servers.
My thoughts as a customer of theirs:
1. Good updates. Not as frequent or clear as I'd like, but mostly they didn't have much to add.
2. Anyone bitching about the thousands of dollars per hour they're losing has not credibility to me. If your junk is that important, your hot standby server should be in another data center.
3. This is a very rare event, and I will not be pulling out of what has been an excellent relationship so far with them.
4. I am adding a fail over server in another data center (their Dallas facility). I'd planned this already but got caught being too slow this time.
5. Because of the incident, I will probably make the new Dallas server the primary and the existing Houston one the backup. This is because I think there will be long term stability issues in this Houston data center for months to come. I know what concrete, drywall, and fire extinguisher dust does to servers. I also know they'll have a lot of work in reconstruction ahead, and that can lead to other issues.
For now, I'll wait it out. I've heard of this cool place called "outside". maybe I'll check it out.
...paying $400+ for a 128gb SSD to replace the standard sata drive in my laptop as long as the performance was truly better and the battery life was that much better.
This consultant wanted you to send it to them? I've been a consultant and developer for nearly 20 years. I would NEVER EVER ask for data like that to be sent to me. I wouldn't want to be anywhere near owning that kind of responsibility for someone else's critical data. You couldn't make me take it if you tried.
Your biggest problem, as pointed out by others, isn't the in-transit data but rather what it does once the consultant gets it. If he's so unaware of modern security best practices as to ask you to send it to him, it's fairly a sure bet that his environment and practices are no where near good enough.
..because we won. History is written by the victors of course. Don't misunderstand me -- nothing could make me defend the German army's actions (or those of many of its citizens at the time). I'm only saying that had we lost that war, a different history might look upon the "re-invasion" of Belgium as a war crime.
Under FIPS, not only must the vendor use specific encryption standards -- those standards must be implemented using specific approved code libraries which have gone through an audited security certification process.
In at least one major application that I'm aware of, if you set the system to be "FIPS" compliant, users who have the newest client can't send encrypted data to users who have older versions because even though they can read it just fine because they do support the standard of encryption -- the libraries used on the older client versions wasn't FIPS compliant. Its a nightmare in terms of implementation and transition from version to version.
Clearly, you don't have teenage children. It is not only normal common practice, but it is in fact essential to force them to follow all kinds of practices that you yourself do not follow.
Is there an immediate life/safety threat? If not, we're going to need to set up a decon in the transition from the hot to the cold zone, and that requires at least 3 people - with their backup which makes six. You have to have at least two at a time entering the hot zone, so that's two more. You'll need someone to do incident command, a couple of medics running rehab, and of course at least two people to manage traffic control.
If you plan to get government funds to cover the cost, you'll have to follows NIMS protocols, which means someone has to do the paperwork for budgeting.
You'll need a pair of engine companies standing by with a charged safety line and a backup line.
With all that manpower and flashing lights, you'll need a media savvy public affairs officer too.
Why can't this particular thing - a good thing - get a compliment on its own? It neither excuses nor condemns any other church behavior. I wasn't making a statement about the Church being good or bad in a BIG GIANT sense. I was saying, "hey, good job on this one, Church."
Sad that everyone with their own personal hobby horse has to ride in a have it crap all over everything else regardless of the topic.
Slashdot Mirror
I have found that Open Source has become an important market force pushing proprietary software vendors harder to create more reliable, more compelling, and overall 'better' products.
The ecosystem has evolved fairly well in many cases. IBM is an interesting case study. They've spent what to most of us would be considered a dozen fortunes (and is to IBM a pittance) supporting the Eclipse platform. They still have many full time IBM employees who are dedicated to working on this entirely free and open source project. They do this because it is to IBM's benefit for that project to exist and to be a good tool. By donating resources, IBM gets input into how it goes together. IBM then uses that open source project as part of some of its solutions which support their other proprietary software. Its working well for everyone so far as I know.
The third party tools market can be good or bad depending on many things. Open Source is a very positive market force in getting rid of the over priced garbage put out by a couple of mid-skilled programming hacks in a couple of afternoons. I find a real scourge of these attached to the Microsoft development community. It has always seemed as though every idiot and his brother has some neat little tool they badly cobble together and then sell for a ridiculous price and don't support. For those tools, Open Source is a death sentence -- and we're all the better for it.
Open source isn't replacing the software vendor. High quality vendors still do exist and are thriving. The games industry, operating systems (yes, I love Linux too - but its not going to replace all desktops any time soon) vertical markets, and other very highly specialized or highly complex and long term development projects are well served by the professional software industry. They require highly skilled professional programmers -- many of whom learn and practice their skills in the much more dynamic and creative -- and more important, peer critical -- open source arena.
Re: Once the fire is out --
There are a few steps, actually:
1st -- it isn't just "fire out" its "scene is safe" which may include hazmat cleanup or other hazards.
2nd -- Sometimes we have to stay and keep people out for a police investigation. Not often, they usually do that but we have lots of fancy lighting and such that can help.
3rd -- Often, we do much more. For example, in a house fire we may have put a hole the roof to vent, pips may have burst or broken (or their solder melted, as is frequent), windows are broken, etc. For a home, we usually stay and try to get boards up to keep the weather out. We usually also stay (in winter) to get a plumber on scene to get the heat going where possible, or the rest of the pipes drained when it isn't. That keeps them from freezing and creating more damage.
4th -- sometimes we have to stay on scene overnight for a fire watch, especially if the alarm systems are damaged or if there was a very large fire and we're watching for hotspots.
5th -- Once we're finally done with the "Emergency" part, the law in most states requires that the structure be inspected and occupancy requirements are met just as if it were new construction.
While "YOU" may want to go back into "YOUR" building, that's a long way from a public building or a building where employees have to work. You can't just order your staff back into a building that hasn't got an occupancy certificate or is dangerous -- or tell them they don't have to go, but then they're fired.
"Forcing myself upon the buildings owners" is really just a rather nasty way of saying you don't like the permitting, building code, and other such laws.
A fire or explosion essentially makes any previous occupancy permit or code enforcement inspection essentially unreliable. In some states they become instantly invalid. In all states, they should. OSHA would never let you put people back to work in a building that can't pass code inspection. That's basic workers rights.
The word "Owner" is the problem. I seriously doubt the "Owner" of that data center was on scene demanding to be allowed to personally start repairs in HIS building.
Life support gear is, as I understand it, built with battery powered redundancy and regularly tested. I don't work with that gear, but I believe it is the case.
I also believe that there are some circuits in hospitals which are specially labeled and are not shut off unless absolutely proved critical or already damaged. In these cases, a lot of money is spent building safety conduits for their cabling and other precautions so that they can handle major damage to the building without becoming a hazard.
Hospital emergencies are their own unique events and there are pre-plan documents and procedures in place for dealing with exactly the issues you describe.
Finally, I would point out that many firefighters are in fact electricians. You see, even career firefighters are not paid well, and most have a second job. Of those, a majority are in the construction and or contracting trades. It is a good fit for them.
People misunderstand the role of a firefighter thinking we just show up and put water on things that are hot. Surely that's the fun part of the job.
In reality, we also have to be many many other things. We have to be truck drivers (you ever drive fast in a 40,000 pound truck carrying a thousand gallons of moving liquid?). We have to be experts in building construction. We have to understand electrical work. We have to be certified in hazmat operations. More recently, we also have to be certified in NIMS (National Incident Management System) which allows us to inter operate using the same language and procedures. We have to be experts at high and low angle rope rescue, confined space rescue, below ground rescue, first responder medical support, mechanical rescue (man vs. meat grinder), traffic control, flood control, crowd control, tree removal, bees, wasps, & snakes, vehicle rescue (we don't take patients from cars, we remove cars -- in chunks -- from around patients) and anything else risky or scary you might want help with.
Even plumbing and water supply -- Just imagine showing up on a scene without a fire hydrant for miles, and being able to organize a tanker shuttle, dump tanks, pumpers, and lines to supply more than 1000 gallons of water a minute within 5 minutes of arriving on scene. That's enough water to fill your pool faster than you can fill your bathtub.
A firefighter crew is a small group of men very much like Macguyver (not as smart maybe but better equiped) with every kind of tool imaginable that they can carry around with them (especially on a heavy rescue unit). You put these guys into ANY situation and within seconds they'll organize around a safe plan for getting to the best possible resolution with the least risk to life and property.
I think the only ignorance I see here, is that which you are demonstrating in your examples.
I'll give you credit for one thing, however. When you state that you are not going to fellate me, you are 100% correct. No matter how nicely you ask.
If its me doing the poking, I probably have a good idea what I'm looking at -- and actually that would make it harder to do my job as a firefighter because I'd be too interested in the stuff.
What they're looking for at the basics of good electrical line management. No blocked vents, no exposed wiring. No extension cords through walls (common), no extension cords used as permanent wiring (common), no extension cords coiled up and flowing power (heat builds up and they catch fire), no chains of power strips -- and so on.
Also, are the fire doors operational and not blocked open? Are the sprinkler or other fire suppression systems in order? Are exit signs lit and accurate? Are emergency battery lighting units charged and ready?
They don't care if you stuff doesn't work. They care if you get trapped because the fire code stuff isn't right.
SOG's are guidelines and not procedures or rules for reasons like what we're arguing about.
Take any specific incident and we can pick apart its details -- especially in the light of day with more facts. A firefighting crew - arrives on scene knowing only that there's something really wrong. You may have heard "explosion" or "fire" but often what you hear en-route is dead wrong. You don't know what caused the explosion. You don't know what exploded. You don't know if people are trapped or injured. You don't know if the riped and stripped wiring is carrying enough voltage to kill you when you touch a chair leaning against a wire you didn't see. You don't know if the explosion was from a gas leak and there's more gas leaking now. You don't know what toxic chemicals are in whatever blew up. These days, we're also trained to think explosions may be bombs. In that case you may have a secondary bomb -- people who set bombs like to make a small first one that draws police and fire, then a large second one that kills them.
You go into a dark building carrying every kind of tool you can carry to deal with whatever variety of broken thing you may find. One group is doing a search for people. One group is doing a search for fire - or other hazards. Fire may look out but be in the walls, or overhead in the drop ceiling. You can't tell.
In Price William County, a very well trained crew entered a house where fire was visible on the outside back wall. It was before 7am, there were cars in the garage and nobody on the front lawn to say if anyone got out yet. They made the second floor and found temperatures not over 90 degrees and a light haze in the air. We call that a tenable environment so we search. They got down the hallway when the fire dropped down on them from the attack space. One of the two up stairs go t out, the other didn't. The reports say the temperature at face height in that upstairs hallway went from 90 degrees to over 700 degrees in a few seconds.
I do my work in a small town hundreds of miles away. Still, we studied that incident like we study any other where men are killed. In most cases, they're killed because something got a lot more dangerous than it looked -- even to trained firefighters -- very very fast.
This is why I ask you not to insult firefighters by pretending to know what is and isn't dangerous without the years of training and practice that go with it.
Its really sad and funny.
By the way, Last year I earned less than $2000 as a firefighter. We're volunteers (or in the case of most, the term is sort of a misnomer, we're paid a minimal amount of money to keep some legal requirements met by our tows).
Here's some numbers for you - I believe these are a year or two out of date, but I'm not going to look for newer:
Of US firefighters, ~300,000 are full time career firefighters, while ~800,000 are your neighbors who have regular jobs and respond when called.
Of US fire departments, 86% are all call-responders (volunteers) while 92% have at least half call-responders. Last I knew, FDNY had at least one call-responder station out on Long Island, but that may be out of date.
I did not and do not ask to be held up as a shining knight of irreproachable perfection. It wouldn't fit well anyway. I did ask that you not ridicule and insult an honorable vocation and the people who, like me, spend hundreds of hours a year training to be aware of how to deal with emergency situations ranging from a toaster oven fire to a train derailment with toxic chemicals or a data center fire with massive hazards.
So no, I'm no Bruce Willis. I'm a network guy, a business owner, and a software developer -- and a volunteer firefighter who spends as much time training for that field as in computers and technology. You may be surprised, but both are extremely technical fields.
Your statements do not accurately reflect the real danger of the situation in general, nor did they reflect a solid understanding of this incident in particular. You seem to think this was a transformer outside the building and that generator power could be applied through the generator at minimal risk to the firefighters and the workers in the building. That just isn't the case here, and usually isn't.
Finally, you are prevented from doing things which can cause you harm in cases where -I- am obligated to save you, in cases where you endanger other people (including me), or in cases where you risk damage to other people's property.
At this particular scene, the walls of the room containing this transformer were blown several feet from where they'd been. Virtually all the power conduits and lines in that room were totally mangled. It took 28 hours for the electricians to repair enough of that wiring and infrastructure for the second floor power to begin to be restored -- with generators. I'm told a few hours of that was spent waiting for the return of the fire marshal to inspect and certify the work as safe. It has taken almost 20 more hours to create a temporary new power distribution infrastructure to manage the power to the first floor. I'm hearing reports that people's 1st floor servers are starting to come online -- several hours ahead of expectations as laid out last night. The second floor will be on regular power soon if its not now, but the first floor will have to run on those backup generators for a week, while the entire power grid for that floor is re-built from scratch before it can be connected permanently.
So, it seems the fire marshal was right. Also, it seems that it is very unlikely if he had not required it be kept off line that TP would have brought the generators up without first inspecting the damage, and as soon as they had they'd have known it would be impossible.
Your frustration at having your servers off line has lead to you declaring that they are worth more to you than the lives of human beings. That, I find disgusting.
While it sounds like a reasonable approach at first, it makes assumptions that I can't make as an officer on scene.
1. It assumes that the only problem is with the original transformer. When I arrive on scene I don't know what the problem was -- even if you tell me you do know, I can't believe it. I also don't know what the secondary problems are.
2. Feeding power into a building that has been physically damaged is very very dangerous. We're not talking about a transformer "failing to work" we're talking about something that blew the walls off the room it was in.
3. We already know that things didn't go the way they were supposed to. Something failed. Some safety plan didn't work. We have to assume that we're dealing with chaos until proved otherwise.
So, as a fire officer I arrive on scene and have a smoke filled building with reports of an explosion and MAYBE a report that everyone is out. I need to go in and find out what happened, if anything is still burning or in immediate danger, and if anyone is still in side. To do that safely, the first thing I want to do is secure the power to the building (shut it off) as well as any other utility feeds (oil, steam, liquefied petroleum or natural gas).
The gear I carry -- even the radio -- is designed to never create even the tiniest spark in its operation. We call it "intrinsically safe". Its one of a great many precautions we take.
We go in to a place like this not knowing the equipment, not knowing its condition.
My final proof point --
If in fact The Planet had powered up their generators, they'd have fried a lot more stuff and caused more fire. The may have destroyed their chances of salvaging the grid within 48 hours at all. Why? It turns out (we now know) that the force of the initial explosion moved three walls in the power distribution center more than a foot (I heard 3 feet I think) off their base. This tore out electrical connections, cables, conduits and power switches. Just now, after 28 hours, they've figured out how to get power to the servers on the second floor, but for the first floor servers they're having to rig up a line from the generators to that floor and it will take until tomorrow to do that. Why? Because the electrical connections from that distribution room to the first floor servers are destroyed. They're going to be running 3000 servers on the first floor off those generators for a week while they get the equipment to rebuild the connectivity to the main distribution room.
What does this prove?
1. It proves the fire marshal was right in not allowing them to feed power in their.
2. It proves that when that big dumb fireman you see (who may be a volunteer who's also a network guy and software developer with an IQ above 95% of the world) may in fact have a good reason for the way they do things on scene.
Look, as a firefighter I don't set out to ruin someone's day. I set out to keep them safe. If that sounds paternalistic, well, It is paternalistic. It very much feels that way. In my small town, its how I feel. I wonder ever time I walk into a building, how I would protect MY PEOPLE in this building if a fire broke out or a hazmat incident started or whatever. You can't help it, its what you're trained to do.
At first I thought you must know me to use my name, then I realized it was just a cheap trick of looking at my email address or profile.
You know how I knew? Because nobody who knows me would make the mistake of calling me an obey-authority anything. You've got the wrong fool on that one.
The arrogant insult dripping through the trolls post to which I responded deserved all the ridicule and self righteousness it contained.
You sir, don't know what you're talking about. Reaching for ridiculous examples of someone doing their job wrong doesn't change that.
Our S.O.G. (standard operating guidelines) are actually very specific about risk.
We will risk our lives to save a human life.
We will take reasonable risk to save the lives of pets and livestock.
We will take minimal risks to save property.
Sorry, but your building isn't worth the risk of my crew. That's reality.
Don't you DARE tell me what is and isn't bravery or cowardly until you put 50 pounds of gear on and crawl into a pitch black house that's burning over your head.
Don't you DARE tell me that you think you understand the difference between saving the blonde girl and saving your computer server.
This isn't TV World. This is the real world. Fire on TV doesn't look like real fire. You know why? Because a real house on fire doesn't look like anything but pitch black and that makes for lousy TV.
Get over yourself and go volunteer at your local fire department. 86% of the men and women in this country who will risk their lives for yours are volunteers. We could use your help if you have the guts for it. We'll teach you what you need to know -- and we'll keep you as safe as we can so you can go home to your family when its done.
Your examples are stupid and insulting to the 800,000 brave men and women who volunteer to risk death in the most painful way possible to save your sorry butt.
Redundant power they have. Redundant power distribution grids they do not. This is common. The level of certification in redundancy on power for fully redundant grids is (I think) called 2N where they only claim N+1 -- which I understand means failover power. Its more than enough 99.9% of the the time. To have FULLY redundant power plus distribution from the main grid all the way into the building through the walls and to every rack is ridiculously more expensive. At that point, it is more sensible to buy another server at another facility for failover than to spend what it would cost to host a server with that kind of power redundancy -- on top of which, the server itself could still blow up and then where are you?
Look, when I go into a building in gear and carrying an axe and an extinguisher, breathing bottled air, wading through toxic smoke I couldn't give crap number one about your 100 sites being down.
I have a crew to protect. In this case, I'm going into an extremely hazardous environment. There has already been one explosion. I don't know what I'm going to see when I get there, but I do know that this place is wall to wall danger. Wires everywhere to get tangled in when its dark and I'm crawling through the smoke. Huge amounts of currents. Toxic batteries everywhere that may or may not be stable. Wiring that may or may not be exposed.
If its me in charge, and its my crew making entry, the power is going off. Its getting a lock-out tag on it. If you wont turn it off, I will. If I do it, you won't be turning it on so easily. If need be, I will have the police haul you away in cuffs if you try to stop me.
My job, as a firefighter -- as a fire officer -- is to ensure the safety of the general public, of my crew, and then if possible of the property.
NOW -- As a network guy and software developer -- I can say that if you're too short sighted or cheap to spring for a secondary DNS server at another facility, or if your servers are so critical to your livelihood that losing them for a couple of days will kill you but you haven't bothered to go with hot spares at another data center then you sir, are an idiot.
At any data center - anywhere - anything can happen at any time. The f'ing ground could open up and swallow your data center. Terrorists could target it because the guy in the rack next to yours is posting cartoon photos of their most sacred religious icons. Monkeys could fly out of the site admin's [nose] and shut down all the servers. Whatever. If its critical, you have off site failover. If not, you're not very good at what you do.
End of rant.
First, that time was an estimate -- a target. Second, even if the initial power test passes, it will take hours to bring up the a/c systems, the switches, and the routers.
:-)
The initial draw from each new bank of gear to be given power will be very high so it will need to go slow.
The battery systems (be they on each rack or in large banks serving whole blocks) will try to charge all at once. If they're not careful, that'll heat those new power lines up like the filaments in a toaster. Remember, the battery plan they have was built with the idea that they'd be used very briefly during transition to generator power -- not drained down all at once.
Only once all the switches and routing gear is back up can they start updating the network paths (do they use BGP for this -- that's not my area of expertise) so that peering data starts flowing.
Only once the network is all up and stable (no small task on a site with dozens of high end peering points) can they even start doing banks of servers.
Its also probably that each bank of servers will needs its own new power lines (and eventually replaced conduit) in the distribution center that was destroyed.
Bank by bank they'll have to bring up all these servers, each of which will draw its maximum load during boot as disks are scanned and checked.
Most of these servers probably haven't been shut down in months or years. Some drives may not spin up due to tired motors that can run fine but spinning from cold is just too much now. Other servers may have boot configuration problems undiscovered since the machines have been running without reboot for a long time -- linux ones anyway
This isn't something out of Young Frankenstein where they'll yell across the room "throw za main svitch!" and a watch the lights dim briefly while 9000 servers boot up with the deafening sound of system beeps. If they did try such a thing -- as if such a thing were possible -- it would immediately blow at least another transformer if not more.
Think about it. 9000 servers @ an average of what, 300 watts, plus the networking gear, plus the air conditioning, plus charging all those batteries....you're talking megawatts.
Without a Mr. Fusion or Harry Mudd stumbling in with some chicks wearing dilithium crystal jewelery this is going to take a while.
What I do isn't just a web site, its also a pbx and some other stuff.
The client software that does the automation is easy. I wrote it to handle the need for a failover server so it will just try the other if the first one fails.
The PBX failover is easy, the DID provider will route to both, only one will pickup sooner. If the second does catch a call, it will have a flag on it that lets it know if the primary is still up and in that case will try to transfer over the call.
The Web Site is actually the least important part of the process for me, and I'll likely handle that with a low ttl on that one particular address.
The inbound mail is easy because I use Postini and it is good at failover.
The data is stored in a database that knows how to sync in near real time between the servers, and on disk as files. I use unison to keep the file directories in sync within just a few minutes.
Overall, I think it should work just fine. There are more elegant solutions -- and more expensive. This one will work for me.
My servers dropped off the net yesterday afternoon, and if all goes well they'll be up and running late tonight. At 1700PST they're supposed to do a power test, then start bringing up the environmentals, the switching gear, and blocks of servers.
My thoughts as a customer of theirs:
1. Good updates. Not as frequent or clear as I'd like, but mostly they didn't have much to add.
2. Anyone bitching about the thousands of dollars per hour they're losing has not credibility to me. If your junk is that important, your hot standby server should be in another data center.
3. This is a very rare event, and I will not be pulling out of what has been an excellent relationship so far with them.
4. I am adding a fail over server in another data center (their Dallas facility). I'd planned this already but got caught being too slow this time.
5. Because of the incident, I will probably make the new Dallas server the primary and the existing Houston one the backup. This is because I think there will be long term stability issues in this Houston data center for months to come. I know what concrete, drywall, and fire extinguisher dust does to servers. I also know they'll have a lot of work in reconstruction ahead, and that can lead to other issues.
For now, I'll wait it out. I've heard of this cool place called "outside". maybe I'll check it out.
...paying $400+ for a 128gb SSD to replace the standard sata drive in my laptop as long as the performance was truly better and the battery life was that much better.
...by email.
This consultant wanted you to send it to them? I've been a consultant and developer for nearly 20 years. I would NEVER EVER ask for data like that to be sent to me. I wouldn't want to be anywhere near owning that kind of responsibility for someone else's critical data. You couldn't make me take it if you tried.
Your biggest problem, as pointed out by others, isn't the in-transit data but rather what it does once the consultant gets it. If he's so unaware of modern security best practices as to ask you to send it to him, it's fairly a sure bet that his environment and practices are no where near good enough.
To be defined by AMD. Certifications by AMD. Hardware requirements by AMD. Video requirements and capabilities by AMD.
Anyone see a problem with this becoming a fully accepted cross platform standard?
..because we won. History is written by the victors of course. Don't misunderstand me -- nothing could make me defend the German army's actions (or those of many of its citizens at the time). I'm only saying that had we lost that war, a different history might look upon the "re-invasion" of Belgium as a war crime.
Under FIPS, not only must the vendor use specific encryption standards -- those standards must be implemented using specific approved code libraries which have gone through an audited security certification process.
In at least one major application that I'm aware of, if you set the system to be "FIPS" compliant, users who have the newest client can't send encrypted data to users who have older versions because even though they can read it just fine because they do support the standard of encryption -- the libraries used on the older client versions wasn't FIPS compliant. Its a nightmare in terms of implementation and transition from version to version.
And how many children over the age of 12 do you have?
Hmmm, yeah. You didn't really add the key point which I'd need before taking this seriously. How many teens, honestly, do you have (or had)?
Clearly, you don't have teenage children. It is not only normal common practice, but it is in fact essential to force them to follow all kinds of practices that you yourself do not follow.
You must be in some crappy shop running a years out of date version administered by buffoons then.
Is there an immediate life/safety threat? If not, we're going to need to set up a decon in the transition from the hot to the cold zone, and that requires at least 3 people - with their backup which makes six. You have to have at least two at a time entering the hot zone, so that's two more. You'll need someone to do incident command, a couple of medics running rehab, and of course at least two people to manage traffic control.
If you plan to get government funds to cover the cost, you'll have to follows NIMS protocols, which means someone has to do the paperwork for budgeting.
You'll need a pair of engine companies standing by with a charged safety line and a backup line.
With all that manpower and flashing lights, you'll need a media savvy public affairs officer too.
Hope this helps.
Oh, were you joking?
Why can't this particular thing - a good thing - get a compliment on its own? It neither excuses nor condemns any other church behavior. I wasn't making a statement about the Church being good or bad in a BIG GIANT sense. I was saying, "hey, good job on this one, Church."
Sad that everyone with their own personal hobby horse has to ride in a have it crap all over everything else regardless of the topic.