Ever see them try to spin up a 1 MW class diesel locomotive engine in winter?
Yes. I have 9 1.25 MW Generators and I can tell you that they spin up and close to the buss in 15 seconds.....even in winter. They're totally capable of meeting this need. That's not to say necessarily that this battery application was a bad choice for this town (I don't have enough details to make that determination), but it wouldn't be fair to say that the battery was chosen over a generator due to some start up delay associated with diesel generators.
I know there are lots of people here who hate Howard Stern, but did anyone else hear Richard and Sal prank calling the pizza parlor with Autotune on their voice? Completely ridiculous.
That may be true. Although, this whole thing has me wondering if they're re-thinking their "Green Technology" push. According to this article, they've recently partnered with a European firm that specializes in Green Data Center Technology, most specifically, using "air side economization" cooling techniques (cool the data center with outside air as opposed to mechanical cooling). Now, while I think this is a viable and worthwhile technology and strategy (and before anyone flames me into oblivion for being a naysayer when it comes to becoming more energy efficient), I do have to raise the question as to whether they had fully commissioned this facility. Had they determined at what point(s) supplemental cooling was brought online? Had they tested the mechanical cooling systems to ensure they would come online successfully and in enough time? Did anyone ask any questions relating to these issues?
Just makes me wonder. I hope they share the information so that we can all learn from it.
No, I'm asking how it is that you feel aspects of various cultures present in America are "imposed" on you. I'm not trying to be argumentative, and if I'm coming across that way, I apologize. I'm simply trying to grasp what cultures are feeling forced on you.
Americans believe that they are "free" only because they live in the same country that imposes the same basic culture and ideology on everyone (usually slightly decorated with some crude ethnic/racial flavor but the same at the core). Nevertheless this is not actually freedom
With respect, can I ask what culture it is that you're feeling is being imposed on you and/or all Americans? To be sure, I'm not at all claiming to know what it's like to have grown up or spent any time living anywhere other than the US (I just want to be sure I'm not misleading you). I'm also not at all trying to claim that the US is some Utopian dream of perfect tolerance and harmony. I do, however, feel that the US has quite a varied culture, which is one of the things that (in my opinion) makes our country both great and sometimes contentious. There are absolutely pockets of intolerance all over our country (try being anything other than Christian in the South for example), but to extend this idea to one that blankets the country as a whole is slightly disingenuous (even if it's unintentionally so).
What are you going on about? Mr. Belits appeared to be discussing his feelings about how it is to live in the US, not about his feelings of US influence while in Russia. I'm not claiming to have an awareness of what it's like to live or have lived in Russia at all, but you seemed to just be pissed because there are McDonald's popping up all over the world. Sure, that's something to be pissed about, but it has nothing to do with what I was commenting on.
then because US imposes on me a culture different from my own, while in USSR I at very least had the luxury of having my native culture being forced on myself
What an odd (and really sad) way of looking at life. If you really feel that the US is "imposing" different culture on you, and you feel that your "native culture" was forced upon you, it might be useful to consider what it is that you feel is coming from you yourself. How can your "native culture" be truly yours if it was "forced" on you? How too can exposure to different cultures within the US be construed as "imposed" on you?
That would surprise me. What wouldn't surprise me is if the 29 domains are all linked to the Iranian government. I think this is a ruse, designed to create the illusion that the Iranian government is a) capable enough to pre-emptively strike its "cyber attackers and b) to paint the Iranian government as a victim of attack, as opposed to the attacker.
Yeah, I've seen those reports, and I am curious as to whether or not they have the server-mounted batteries at all of the data centers, or just some of them. You and I are thinking right along the same lines. If indeed they did have the server-mounted batteries, I'd be curious to know why they didn't hold all the load. It seems to me that either the server-mounted battery strategy is less reliable than traditional UPS Systems, or as you suggested, perhaps something happened with the generators (those generators should have spun up and been carrying load within 15 seconds). Either way, something didn't work as designed, and TFA doesn't touch on any of it.
Power losses, including ones that take down the whole data center, are accepted risks and part of the reason they have a redundant data centers and failover procedures.
The failure wasn't that they had a partial loss at a datacenter. The failure was that the impact of that loss wasn't mitigated properly by the systems that were supposed to be in place to do that.
I must respectfully disagree. Power losses that take down the whole data center are definitely NOT accepted risks. The entire reasoning for spending millions upon millions of dollars to have UPS systems, Static Switches, Automatic Throwover Switches, Diesel Generators with thousands of gallons of fuel, etc isn't because you think downtime is acceptable, it's because downtime is not an option.
We almost agree though. I do agree that the failure was in improper mitigation of the risk as opposed to mitigation of the outage once it happened. There is no reason given that explains why 25% of their floor(s) went down, and in a properly run data center (at least a Tier 3 or higher), there is no reason any of the Critical Load should ever go down.
First of all, the "flywheel generators" you're referring to are actually either standalone UPS systems or a part of a DRUPS (Diesel Rotary UPS). Here is some information on one of the leading manufacturers of such equipment.
However, all of this is moot, since even if they had a flywheel setup as you're speculating, it still doesn't explain why 25% of the floor went down. If the equipment was installed, maintained and loaded properly, they should've been able to get to the generators with no problem.
are you really telling me that you believe you and ElectricTurtle are smarter than the combined brainpower set loose by Google for building and maintaining this facility?
No, I'm telling you that I manage a data center, and I know first hand how they work (or in this case, should work). I fail to see an adequate explanation of how this was unavoidable.
I have to say, I think ElectricTurtle is right. If the generators came online as they're claiming, how could it be that 25% of the load dropped during transfer?? There's more to this story than is being told, and instead, they're focusing on how they came back online rather than why they went offline in the first place. I'd be willing to bet you that heads are rolling behind closed doors. If there were properly functioning UPSs in the building (either the large ones or the server-mounted batteries Google sometimes likes), then there shouldn't have been any outage on the transfer to generator.
The otherwise top rated 365 Main [365main.com] facility in San Francisco went down a few years ago. They had all the shizz, multipoint redundant power, multiple data feeds, earthquake-resistant building, the works. Yet, their equipment wasn't well equipped to handle what actually took them down - a recurring brown-out. It confused their equipment, which failed to "see" the situation as one requiring emergency power, causing the whole building to go dark.
I think you made the right decision in changing providers. I remember that story about the 365 outage, and while I am too lazy to look up the details again, I recall it being as you're telling it. To that end, I'd simply say that they most certainly did have the proper equipment to handle the brown out, but obviously not the proper management. If you're having regular (if intermittent) power problems (brown outs, phase imbalances, voltage harmonic anomolies, spikes, etc), just roll to generator, that's what they're there for.
I'm sick of people making the assumption that the operators of the facility were just at the mercy of a power quality issue because they have redundant power feeds and automatic transfer switches. Yes, in a perfect world, all the PLCs will function as designed, and the critical load will stay online by itself. However, it takes some foresight and some common sense sometimes to make a decision to mitigate where necessary. I direct all my guys to pre-emptively transfer to our generators if there are frequent irregularities on both of our power feeds (i.e. during a violent thunderstorm, simultaneous utility problems, etc).
In other words, I'm agreeing with you that the service you received was unacceptable. Along with that (and in rebuttal to the parent post), I'm saying that it's not enough to talk about how they came back from the dead, but why they got there in the first place.
It's really just better to work your life around it than force yourself into unnatural (for you) sleep patterns.
I find smoking weed helps if I need to get to sleep & wake early, otherwise staying awake all night is better than trying to sleep early if I absolutely must be alert and active before noon.
If I may, give this a try. Much healthier, legal, and you're likely to find yourself either sleeping more easily or needing less sleep. Just a personal suggestion (your results may vary).
Are you implying that the Chief Operating Officer for the entire organization was unaware that there were people on the phone? Are you honestly suggesting that because a bridge line was open on the phone, that Mr. Childs had the responsibility to question the COO's authority and judgment? This wasn't some random person asking for the passwords, rather it was a very senior management official asking for passwords to a company owned and operated network resource. Why is this so hard to understand?
Right in the middle of the "don't" list in the City's policy (which is freely available on the web) was "DO NOT DISCLOSE PASSWORDS TO YOUR BOSS".
So, right there, he cannot disclose it and uphold the policy that he was told to uphold.
With all due respect (I'm not trying to be a troll here), the section of the policy that precedes the one you're referring to trumps your argument. That is, you're referring to the section of the policy relating to "user level" passwords (i.e. the ones to get onto the Exchange Email server, the one to log on to a PC, etc). The prior section (section 4.1) states very plainly that all "production system level" passwords were supposed to be stored in a central database, which this person obviously did not do.
To make the argument that the policy relating to user level passwords should apply in lieu of the specific section of the policy that relates to system level passwords is disingenuous. Also, to be fair, by your rationale, Mr. Childs would never have been able to disclose the passwords to anyone at any time. I still am having a hard time believing that you actually think that the Chief Operating Officer for the entire organization would not be someone who would be authorized to have the passwords.
My boss and I get along really well. However, if my boss called me in to his office and told me to tell him my password, my answer would be "no".
Your boss, maybe (big maybe), but the COO for your company?? So if an employee for Microsoft is asked by Kevin Turner for passwords to a company owned and operated network resource, you honestly believe that the answer should be 'no'??
Do you have a problem with reading comprehension?
It is plain from the policy that he should not have disclosed the passwords, despite your claims.
From the actual policy (reiterated for you):
4. Policy 4.1. General
All system-level passwords (e.g., root, enable, NT admin, application administration accounts, etc.) must be changed on at least a monthly basis. All production system-level passwords must be part of the security administered global password management database.
It seems it is you who could brush up on his reading skills. Why is so difficult to understand that this man was required by the policy to keep the passwords in a central database, and he clearly did not? Perhaps you (as an admin) are personalizing this situation, as opposed to my being "invested in his guilt". It seems perfectly plain the guy did not follow protocol.
According to the policy, he was supposed to have the passwords in a global, heavily-protected database. Whether he did or did not is open for debate. I don't know. And that could be a source of trouble to him.
For clarity's sake, here's the section of the policy relating to this type of "system level" password from your linked PDF:
4. Policy
4.1. General
All system-level passwords (e.g., root, enable, NT admin, application administration accounts, etc.) must be changed on at least a monthly basis.
All production system-level passwords must be part of the security administered global password management database.
As you alluded, it would seem pretty clear that Mr. Childs was indeed in violation of this policy. If he had the passwords in the database, as required by the policy, no one would have had to ask him for them.
According to the charges, he did not surrender the passwords to a group of people, many of whom were unknown to him, and many others of whom were clearly not authorized to possess that information
As I've noted in a few other replies (and detailed in this article):
That afternoon Childs "unwittingly" found himself in a surprise meeting in the city's Hall of Justice, where he maintained network facilities. At the meeting were his boss, DTIC Chief Operations Officer Richard Robinson, San Francisco Police Department CIO Greg Yee and human resources representative Vitus Leung
.
I'm having a hard time buying the whole "he didn't think the people were authorized" routine. When the COO, your direct boss, and a rep from Human Resources are there, I think it's time we drop the whole "oh, I don't think you're cleared for that" story.
That afternoon Childs "unwittingly" found himself in a surprise meeting in the city's Hall of Justice, where he maintained network facilities. At the meeting were his boss, DTIC Chief Operations Officer Richard Robinson, San Francisco Police Department CIO Greg Yee and human resources representative Vitus Leung. On the phone were engineers, listening in to confirm whether the passwords he gave were correct.
Slashdot Mirror
I'm betting it looked like a bigger, badder version of this.
Ever see them try to spin up a 1 MW class diesel locomotive engine in winter?
Yes. I have 9 1.25 MW Generators and I can tell you that they spin up and close to the buss in 15 seconds.....even in winter. They're totally capable of meeting this need. That's not to say necessarily that this battery application was a bad choice for this town (I don't have enough details to make that determination), but it wouldn't be fair to say that the battery was chosen over a generator due to some start up delay associated with diesel generators.
Bilbo!!!!
I know there are lots of people here who hate Howard Stern, but did anyone else hear Richard and Sal prank calling the pizza parlor with Autotune on their voice? Completely ridiculous.
Well, someone's a little bitter. Maybe you could lose the attitude, and you might have better luck in your interviews. Just a thought.
That may be true. Although, this whole thing has me wondering if they're re-thinking their "Green Technology" push. According to this article, they've recently partnered with a European firm that specializes in Green Data Center Technology, most specifically, using "air side economization" cooling techniques (cool the data center with outside air as opposed to mechanical cooling). Now, while I think this is a viable and worthwhile technology and strategy (and before anyone flames me into oblivion for being a naysayer when it comes to becoming more energy efficient), I do have to raise the question as to whether they had fully commissioned this facility. Had they determined at what point(s) supplemental cooling was brought online? Had they tested the mechanical cooling systems to ensure they would come online successfully and in enough time? Did anyone ask any questions relating to these issues?
Just makes me wonder. I hope they share the information so that we can all learn from it.
No, I'm asking how it is that you feel aspects of various cultures present in America are "imposed" on you. I'm not trying to be argumentative, and if I'm coming across that way, I apologize. I'm simply trying to grasp what cultures are feeling forced on you.
Americans believe that they are "free" only because they live in the same country that imposes the same basic culture and ideology on everyone (usually slightly decorated with some crude ethnic/racial flavor but the same at the core). Nevertheless this is not actually freedom
With respect, can I ask what culture it is that you're feeling is being imposed on you and/or all Americans? To be sure, I'm not at all claiming to know what it's like to have grown up or spent any time living anywhere other than the US (I just want to be sure I'm not misleading you). I'm also not at all trying to claim that the US is some Utopian dream of perfect tolerance and harmony. I do, however, feel that the US has quite a varied culture, which is one of the things that (in my opinion) makes our country both great and sometimes contentious. There are absolutely pockets of intolerance all over our country (try being anything other than Christian in the South for example), but to extend this idea to one that blankets the country as a whole is slightly disingenuous (even if it's unintentionally so).
What are you going on about? Mr. Belits appeared to be discussing his feelings about how it is to live in the US, not about his feelings of US influence while in Russia. I'm not claiming to have an awareness of what it's like to live or have lived in Russia at all, but you seemed to just be pissed because there are McDonald's popping up all over the world. Sure, that's something to be pissed about, but it has nothing to do with what I was commenting on.
Calm down, dude.
then because US imposes on me a culture different from my own, while in USSR I at very least had the luxury of having my native culture being forced on myself
What an odd (and really sad) way of looking at life. If you really feel that the US is "imposing" different culture on you, and you feel that your "native culture" was forced upon you, it might be useful to consider what it is that you feel is coming from you yourself. How can your "native culture" be truly yours if it was "forced" on you? How too can exposure to different cultures within the US be construed as "imposed" on you?
I wouldn't be surprised if they were CIA fronts
That would surprise me. What wouldn't surprise me is if the 29 domains are all linked to the Iranian government. I think this is a ruse, designed to create the illusion that the Iranian government is a) capable enough to pre-emptively strike its "cyber attackers and b) to paint the Iranian government as a victim of attack, as opposed to the attacker.
For anyone interested here is the link on the NSF page showing the old site and the new facility. Pretty cool (pardon the pun).
Yeah, I've seen those reports, and I am curious as to whether or not they have the server-mounted batteries at all of the data centers, or just some of them. You and I are thinking right along the same lines. If indeed they did have the server-mounted batteries, I'd be curious to know why they didn't hold all the load. It seems to me that either the server-mounted battery strategy is less reliable than traditional UPS Systems, or as you suggested, perhaps something happened with the generators (those generators should have spun up and been carrying load within 15 seconds). Either way, something didn't work as designed, and TFA doesn't touch on any of it.
Power losses, including ones that take down the whole data center, are accepted risks and part of the reason they have a redundant data centers and failover procedures. The failure wasn't that they had a partial loss at a datacenter. The failure was that the impact of that loss wasn't mitigated properly by the systems that were supposed to be in place to do that.
I must respectfully disagree. Power losses that take down the whole data center are definitely NOT accepted risks. The entire reasoning for spending millions upon millions of dollars to have UPS systems, Static Switches, Automatic Throwover Switches, Diesel Generators with thousands of gallons of fuel, etc isn't because you think downtime is acceptable, it's because downtime is not an option.
We almost agree though. I do agree that the failure was in improper mitigation of the risk as opposed to mitigation of the outage once it happened. There is no reason given that explains why 25% of their floor(s) went down, and in a properly run data center (at least a Tier 3 or higher), there is no reason any of the Critical Load should ever go down.
However, all of this is moot, since even if they had a flywheel setup as you're speculating, it still doesn't explain why 25% of the floor went down. If the equipment was installed, maintained and loaded properly, they should've been able to get to the generators with no problem.
are you really telling me that you believe you and ElectricTurtle are smarter than the combined brainpower set loose by Google for building and maintaining this facility?
No, I'm telling you that I manage a data center, and I know first hand how they work (or in this case, should work). I fail to see an adequate explanation of how this was unavoidable.
I've heard a few rumors that they're re-thinking this strategy. I'm betting this event might keep those conversations going.
I have to say, I think ElectricTurtle is right. If the generators came online as they're claiming, how could it be that 25% of the load dropped during transfer?? There's more to this story than is being told, and instead, they're focusing on how they came back online rather than why they went offline in the first place. I'd be willing to bet you that heads are rolling behind closed doors. If there were properly functioning UPSs in the building (either the large ones or the server-mounted batteries Google sometimes likes), then there shouldn't have been any outage on the transfer to generator.
The otherwise top rated 365 Main [365main.com] facility in San Francisco went down a few years ago. They had all the shizz, multipoint redundant power, multiple data feeds, earthquake-resistant building, the works. Yet, their equipment wasn't well equipped to handle what actually took them down - a recurring brown-out. It confused their equipment, which failed to "see" the situation as one requiring emergency power, causing the whole building to go dark.
I think you made the right decision in changing providers. I remember that story about the 365 outage, and while I am too lazy to look up the details again, I recall it being as you're telling it. To that end, I'd simply say that they most certainly did have the proper equipment to handle the brown out, but obviously not the proper management. If you're having regular (if intermittent) power problems (brown outs, phase imbalances, voltage harmonic anomolies, spikes, etc), just roll to generator, that's what they're there for.
I'm sick of people making the assumption that the operators of the facility were just at the mercy of a power quality issue because they have redundant power feeds and automatic transfer switches. Yes, in a perfect world, all the PLCs will function as designed, and the critical load will stay online by itself. However, it takes some foresight and some common sense sometimes to make a decision to mitigate where necessary. I direct all my guys to pre-emptively transfer to our generators if there are frequent irregularities on both of our power feeds (i.e. during a violent thunderstorm, simultaneous utility problems, etc).
In other words, I'm agreeing with you that the service you received was unacceptable. Along with that (and in rebuttal to the parent post), I'm saying that it's not enough to talk about how they came back from the dead, but why they got there in the first place.
It's really just better to work your life around it than force yourself into unnatural (for you) sleep patterns. I find smoking weed helps if I need to get to sleep & wake early, otherwise staying awake all night is better than trying to sleep early if I absolutely must be alert and active before noon.
If I may, give this a try. Much healthier, legal, and you're likely to find yourself either sleeping more easily or needing less sleep. Just a personal suggestion (your results may vary).
Are you implying that the Chief Operating Officer for the entire organization was unaware that there were people on the phone? Are you honestly suggesting that because a bridge line was open on the phone, that Mr. Childs had the responsibility to question the COO's authority and judgment? This wasn't some random person asking for the passwords, rather it was a very senior management official asking for passwords to a company owned and operated network resource. Why is this so hard to understand?
Right in the middle of the "don't" list in the City's policy (which is freely available on the web) was "DO NOT DISCLOSE PASSWORDS TO YOUR BOSS". So, right there, he cannot disclose it and uphold the policy that he was told to uphold.
With all due respect (I'm not trying to be a troll here), the section of the policy that precedes the one you're referring to trumps your argument. That is, you're referring to the section of the policy relating to "user level" passwords (i.e. the ones to get onto the Exchange Email server, the one to log on to a PC, etc). The prior section (section 4.1) states very plainly that all "production system level" passwords were supposed to be stored in a central database, which this person obviously did not do.
To make the argument that the policy relating to user level passwords should apply in lieu of the specific section of the policy that relates to system level passwords is disingenuous. Also, to be fair, by your rationale, Mr. Childs would never have been able to disclose the passwords to anyone at any time. I still am having a hard time believing that you actually think that the Chief Operating Officer for the entire organization would not be someone who would be authorized to have the passwords.
My boss and I get along really well. However, if my boss called me in to his office and told me to tell him my password, my answer would be "no".
Your boss, maybe (big maybe), but the COO for your company?? So if an employee for Microsoft is asked by Kevin Turner for passwords to a company owned and operated network resource, you honestly believe that the answer should be 'no'??
Do you have a problem with reading comprehension? It is plain from the policy that he should not have disclosed the passwords, despite your claims.
From the actual policy (reiterated for you):
It seems it is you who could brush up on his reading skills. Why is so difficult to understand that this man was required by the policy to keep the passwords in a central database, and he clearly did not? Perhaps you (as an admin) are personalizing this situation, as opposed to my being "invested in his guilt". It seems perfectly plain the guy did not follow protocol.
This is addressed in a preceeding post. He was not following the information security policy, plain and simple.
According to the policy, he was supposed to have the passwords in a global, heavily-protected database. Whether he did or did not is open for debate. I don't know. And that could be a source of trouble to him.
For clarity's sake, here's the section of the policy relating to this type of "system level" password from your linked PDF:
As you alluded, it would seem pretty clear that Mr. Childs was indeed in violation of this policy. If he had the passwords in the database, as required by the policy, no one would have had to ask him for them.
According to the charges, he did not surrender the passwords to a group of people, many of whom were unknown to him, and many others of whom were clearly not authorized to possess that information
As I've noted in a few other replies (and detailed in this article):
That afternoon Childs "unwittingly" found himself in a surprise meeting in the city's Hall of Justice, where he maintained network facilities. At the meeting were his boss, DTIC Chief Operations Officer Richard Robinson, San Francisco Police Department CIO Greg Yee and human resources representative Vitus Leung
.
I'm having a hard time buying the whole "he didn't think the people were authorized" routine. When the COO, your direct boss, and a rep from Human Resources are there, I think it's time we drop the whole "oh, I don't think you're cleared for that" story.
From what I understand, they guy asking for them was not in the "need to know" list of people.
Then you would be mistaken. From this article: