Slashdot Mirror
Big Red Button Disasters?
FredDC asks: "The Daily WTF has a story about a Big Red Button disaster. What Big Red Button disasters have you experienced? Which ones have you caused? Are there any that you've heard about, or do you know of any that can happen any day now?"
When I was a young child, I found a fire alarm, and, with my father screaming ``No!'' in the background, proceeded to pull it. This is right after we moved to America from Russia, and dealing with the fire department, while barely understanding what they are saying, must have sucked.
...thinks that datacenters should be open to ANYONE besides critical staff? At work, we don't even let the janitor in...
Back in the day, the guy next to me on the trade desk bought the wrong $10mm of muni bonds at the correct price with the push of a button. It cost him 23/32nds or about $70,000 to swap to the correct issue.
The Big Red Button
Monday, January 22, 2007
More then 3 months old, eh?
I was doing I.T. support for a 400 person call center. In the server room there was a Big Red Button that was very clearly labeled "EMERGENCY POWER SHUT-OFF" near one of the sets of double-doors.
A technician from U.S. Worst had finished his work in the server room and on his way out he hit the Big Red Button thinking that would open the doors, like at a hospital.
Hilarity ensued.
Later that day I printed out several mock "Big Red Buttons" on sheets of paper to use as decoys next time the tech had to visit.
Well my Big red button get's mildly upset.
Alex tends to use hyperbole A LOT when editing a story for posting on WTF. :)
Stimpy, don't press the red button!
~> ftp www.workplace.domain /mis-typed/path /mis-typed/path: The system cannot find the file specified.
/index.html with something that was supposed to be a couple levels down is bad enough.
/index.html is owned by someone else entirely. Someone who now must be woken up in the middle of the night, in a different country...
Connected to www.workplace.domain.
220 Microsoft FTP Service
Name: shag
331 Password required for shag.
Password:
230 User shag logged in.
Remote system type is Windows_NT.
ftp> cd
550
ftp> put index.html
local: index.html remote: index.html
227 Entering Passive Mode.
125 Data connection already open; Transfer starting.
226 Transfer complete.
ftp>
The realization that one has just overwritten a public-facing, high-traffic
It's worse when
After I did this two or three times, I decided to stop being such a hardcore geek and got an FTP application with a GUI.
Village idiot in some extremely smart villages.
I was a QA intern at Fujitsu working on the WorldsAway chat world when I discovered a rare crash bug with a new artist tool that I could reproduce successfully but my boss couldn't. Since the tool was supposed to be used on the test server only, my boss approved release of the update to the production server. Everything was fine for a day before the production server started crashing. Turns out that the artists were creating new content on the production server instead of the test server and using the new tool that caused the crashes. The production server was shut down for three days a complete code rewrite was required and Fujitsu lost $250,000 USD in revenue. My boss kept his job as he led the programming team to rewrite the code. I, on the other hand, was given two weeks notice that my six month contract wasn't going to be renewed. Two weeks after I left the company, one-third of the division was laid off to pay for the lost revenue.
We had a literal Big Red Button near the door to our (small) data center at my last job. My boss and I didn't know what it was for, although we guessed it was a shut-off for the power to the entire room. Our APCs at that time could only keep things running for about 30 minutes, tops.
:)
We never did push the button, but after a couple years my boss had maintenance physically remove the button, just to get rid of the temptation.
Wow, I haven't posted in forever.
Anyway, we did a big datacenter migration at my last company. I'm not going to name names, but it's a Fortune 100 company based in Austin, TX. The move was happening because we built our own building with our own datacenter.
As part of the technical staff (network engineering/security), I was given a tour of the new datacenter before it opened. My boss and assorted other folks were on the tour. My boss, by the way, was a huge...jerk.
The electrician showed us the Big Red Buttons by each of the exit doors. He also said that each of the Power Distribution Units (of which there were three) had a Big Red Button that would cut power to just the areas powered by that unit.
My boss said, not jokingly, "If you need to cut power in an emergency, see if you can figure out which PDU is involved and just cut that one, so we don't lose the whole datacenter."
I piped up: "If I'm getting 220 across my nipples, cut the whole damn room. I really don't care enough about the company to die. I can see my epitaph now: 'Here lies Dimwit. He died saving two-thirds of the datacenter.'"
Man, if looks could kill.
...but it's being eaten...by some...Linux or something...
One of our HVAC workers thought the red button, with the plastic cover that says "EPO", was a light switch ;)
All new keyboards have a single key Shutdown/sleep thing.
Arghhhhhhhhhhhhhhhhhhh @ little fingers.
I either rip the bastard thing right off the board or dig out the regkey thingy to disable it.
liqbase
to tell people that "Halon" is French for "Exit," so if they ever get locked in the data center, they know how to get out.
"National Security is the chief cause of national insecurity." - Celine's First Law
1) I was working with a friend of mine, and we were setting up the graceful shutdown of the servers after getting all the UPS on the network.
He manually tripped the battery low condition with the intention that the UPS would abort the shutdown when the power came good again. Nope, all the servers were triggered for shutdown (Couldn't abort on the UPS or the servers) and had to be rebooted. The best part was that the UPS sent commands to another site for servers to shutdown there. We had to phone another data centre and get them to go power on the servers after we quickly faxed through forms telling them what they had to do (Cabinet number, server name etc)
2) Another job, I had just wired up a big red button next to the door in the new data centre (Someone had forgotten to install one, so I had to do it on a weekend). Well, one of the guys who I worked with phoned me up asking me if the switch was connected. I told him it was, and that I hadn't installed the Molly Guard yet, but was going to do it after I finished all the testing when I got back from lunch. He said OK, and hung up. He got it into himself to finish the testing to save me the time. Under normal circumstances, this wouldn't have been a problem, pulling apart APC UPS units wasn't a major concern to us at this point. What he assumed had happened however was that I had left the toggle switch for test on. No, I hadn't. The switch worked and was live in case it needed to be used (It was there for a reason, just because I am a block down the street getting lunch doesn't mean that it might not have a purpose as far as I am concerned). About 5 mins later, he phones me back up and asks if I can come back to the office, I say "Sure, not a problem, what's going on? The SQL server not patching?" (Something else we were doing that day) "I am in the data centre". At this point in time, I realise that normally I am asking him to walk out of the data centre cause it's too noisy. Glad it was the weekend and there wasn't much going on.
I have also had a UPS engineer blow dust into a VESDA and we had a few fire trucks turn up, but that wasn't big red button issue.
Curiosity was framed; ignorance killed the cat. -- Author unknown
In Soviet Russia the alarm sounds YOU!
Web2.0: I love when people Flickr my cuil and digg my boingboing until my google is reddit and I start to yahoo
"Wow, I haven't posted in forever."
[Dimwit groupies]
YEEAAHH! We missed you! You're in my heart, Dimwit! I want to have your baby, Dimwit!
On top of the support calls, he was of course getting his daily "yelling at escalation managment party conference call" because not everything was smooth, needless to say. For instance, that brand new customer was brand new to deploying a cell phone infrastructure: bad planning, downtime, crazy schedule were the least of their problems. To add some icing to this merry cake, our switching software was quite new as well, and prone to er, quirks?
Talk about a recipe for wide-scale disaster ;-)
But this day, instead of storming to the next support specialist to wearily beg for some random node reboot, X was running to each and everyone desk, doing his grand floor tour, smiling like a madmam, yelling for everyone to hear:
"They shut off the switch, they shut off the ***** switch, I tell you".
Turns out some cleaning lady tried to shut the **light** off after a good floor-cleaning session, but when for the BIG SWITCH, the one with the big conspicuous red handle, labelled "MAIN POWER - DO NOT..."
BLACKOUT
X did not get his yellint at party this day, and possibly a few days afterward.
(No real names was used, because all the abovementionned companies are still operating as of today:)
[Pruneau
You know the submission queue is slow when by the time the story is posted the site has changed its name.
The kind man who gave us Explorer Scouts the tour of the IBM Federal Data Center in Gaithersburg, MD (literally acres of machine room floor) said:
"Don't touch that red switch. Really don't. It takes us days to recover."
and
"Well, you actually have to *pull* the knob." (Why he gave half a dozen computer-starved teenagers that knowledge, I have no idea).
On IBM systems, apparently there's a knife of some kind that physically severs the power cables. It's a mess.
Any sufficiently advanced technology is insufficiently documented.
That was fun.
Web2.0: I love when people Flickr my cuil and digg my boingboing until my google is reddit and I start to yahoo
Around ten years ago I was looking to rent a house in Park Royal, London with a couple of friends. We went into a decent house on Twyford Abbey Road for those that know the area (just off Hanger Lane Gyratory).
The landlord was abroad in Tokyo, so it was just ourselves and the agent. Nice house, but whilst looking around we saw a big red button in the main bedroom. For those to whom it's obvious what the purpose was, at that time it was my first encounter with such a device - first encounter for all of us in fact. And so, with the agent waiting downstairs, the conversation went...
Friend 1: "What's that for?"
Me: "I say we press it. That's what big red buttons are -for-."*
Friend 2: "ok" (presses button)
The next scene - pandemonium as the alarms all round the house go off. It's a panic button of course - we'd never come across one at that point, so we pressed it anyway. Up runs the estate agent to find out what we'd done. We tell him - yep, love the house. We'll take it. Oh, the alarm thing? That's fine, it's because we pressed this big red button. Ah - the owner's in Tokyo and you don't know the code? And it's -what- time in Tokyo? Hmm. Err...
And out the house we went, as fast as possible. And away we drove, again as fast as possible. We'd left the agent in charge of a screaming house, which every neighbour for a mile must have heard, and with absolutely no way to shut the alarm off for several hours. It was, as the saying goes, time to be somewhere else.
Still took the house though - lived there for a few years, enjoyed it actually.
Cheers,
Ian
*I was actually quoting a friend of mine, who in turn says he was quoting some film or comic. If you happen to know the source of the quote, I'd be interested to hear it.
I spent $5 to buy the big red Easy button from Staples. What a waste, what a disaster.
I wanted to delete temp files that my editor created it did so by putting a ~ at the end of it. FileName~
/bin/rm -rf * ~
I also had some directories I wanted deleted so I renamed them with a ~ at the end as well So I can delete them in one swoop.
So I was in a rush and didn't want to be warned because I had about 50 or so Temp Files so I did a
That one acedental space whiped out all the files and folder in that directory then preceded to begin deleteing all the files in my home directory as well. I suck most because I had a 1000 line HTML code I just finished (This was in the days before dependable Javascript and CSS). I spent the rest of the day shifting thew the Cache files on my windows box for IE and Netscape for Windows testing. I was able to get most of it back, but man that was a bad day.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
I used to work help-desk, and late at night there would only be two people in the quite large building - me and one of the operators. Anyone who as worked with "ops" knows they generally turn a bit strange due to them working nights with nobody around and only DAT tapes for company.
So anyway, there is this big fire alarm panel with tons of buttons that we never really thought about, until one night when it started beeping constantly. The ops guy found a key to it, and then we both stood there looking at the probably 60 buttons and flashing lights, etc. Personally, I would have chosen one of the black buttons marked "mute", but the ops guy went straight for the biggest red one on the board.
The result was more beeping, lots of red lights and about 5 fire-engines.
In a previous life I was in the Navy, and onboard a Frigate. This Frigate just happended to have Gas Turbine Enclosures, the Turbines (which typically are the engines of Jet Airliners) turned the shaft. Anyway during a drill, an Officer (thank goodness), who just happened to be the Damage Control Officer (even better), accidentally flipped the switch to dump the halon in one of the enclosures for real. Needless to say the Engineering team that was responsible for the Turbines cursed him for days while they recovered from the mess he created.
I'm just thankfull I was only a bystander and in no way involved in this!
My "Big Red Button Day" occurred due to a new employee on a government program at Peterson AFB. He was escorting the cleaning people around our office area and datacenter. Part of the task is to turn on flashing red lights to let people know that there are uncleared people in the room. When he escorted them into the datacenter, he saw "The Big Red Button" (covered, but unlabeled at the time) and thought, "Red button turns on red light".
Unfortunately, our office area was built on the computer room floor with the rest of the data center and shared the same power grid. When the newbie pressed the button, 25 RS/6000s, 1 IBM 3033 mainframe, 15 3394 disk packs, several VAX, PDP, dozens of PCs, X-Stations, etc and all the lights went out at the same time. It's never easy to find the breakers in the dark. It took us 3 days to get all the hardware back up and running properly.
If "disco" means "I learn" in Latin, does "discothèque" mean "I learn technology"?
And as long as we're talking halon, who can forget the classic Vaxen, My Children, Just Don't Belong In Some Places.
There are big red buttons at the power station that kill the plant and are designed to stop all rotating machinery extremely fast. Turbines with a 10 minute cooldown cycle stop in something like 30s with the big red button. The power grid is designed to cope with such an immediate loss of supply- the grid controllers maintain a "spinning reserve" that is greater than the capacity of the the single largest plant. If one plant should happen to have a sudden mishap then nothing happens, an already running plant takes up the slack. But if two large power plants were to simultaneously kick off grid at the same moment on a very hot day bad things could happen.
There were a couple days last summer where there was no spare capacity in the Northeast. It was simply so hot that all the AC's were cranked and the grid was saturated in many places. This year should be interesting.
Even those who arrange and design shrubberies are under considerable economic stress at this period in history.
I've been looking to buy a few evil-looking functional big red buttons, but I haven't been able to find any at electric supply shops where I live (Toronto).
Any online retailers?
Not really a disaster... At my work computer, I installed a second hard drive to backup all the work that had been done throughout the day. Not the most efficient way of performing a backup, but certainly cost-effective: I created a batchscript which uses xcopy and a whole host of flags to copy the subdirectories and not to ask to overwrite the data, etc. etc.
Once it's done backing up, the batchscript has a 120 second delay where it prompts the user to press any key to abort the shutdown -- needless to say, anyone and everyone who encounters the prompt just has to press any key.
While not an official "Big Red Button" story I think it is worth telling.
In 1999 while I was working as a private consultant for the capitol city of a small New England state, a colleague of mine was attempting to make a change to the city's core switches. Per usual with this guy, he over-sold his skill set and was way out of his league - while never willing to admit it.
Meanwhile, I was working in the server room on the squid web caching server while he was attempting the change...
I kept hearing him say things like "I wonder what this command does", and "I wonder what the reset command means. Should I enter it?"
Suddenly I was no longer ssh'ed into the proxy server... I looked up and asked "What the hell did you do?"
His answer: "I entered the reset command"
Me: "Well, fix it. Restore the configuration. It looks like you just reset EVERYTHING..."
Well, needless to say, there was NO saved configuration to restore, and no documentation for the city's network nor the equipment installed, and on this equipment the reset command was the command to reset it to its default settings. (BTW, he entered the reset command on the core switch) There were several local switches (connected via copper), and many fiber connections to all the remote departments across the city - several fire departments, the main police department, city hall, you name it... All off-line.
In the end, the city's network was DOWN for 3-4 full days while he contacted qualified people to attempt to rebuild the network...
We would have been better off if he had hit the big red button near the sliding glass door at the server room's exit.
sigh...
P.S. I am pretty sure he blamed it all on me.
Windows is not the answer.
Windows is the question.
The answer is "NO."
Act One
Big test floor, where several large (multi-million dollar) computer systems are being configured and tested before shipment to the customer.
Tall skinny hyperactive developer (no, not me, I was just an observor) leaning against the wall of the test floor, actually *fiddling with* the Big Red Button.
Someone suggests that he ought not do that. He promises to be careful.
Act Two
Five minutes later. All the power has just gone out. It's amazing how quiet it is all of a sudden. Everyone is looking over at the tall skinny developer with his hand on the Big Red Button.
No words are spoken.
Act Three
Half an hour later. Electrician is leading the tall skinny developer around as he turns on each part of the power system in the right order. CEO and various unmollified developers watching. Back by the door, guy from facilities is bolting a flap over the Big Red Button.
Almost.
The guy I work for helpfully logged me into a root mysql prompt on the main db server, then told me not to change anything and left for the rest of the day. All I was doing was fixing some AJAX code.
First thing I did was panic and stab the xterm to death with ctrl+D.
But a little red switch. During class, I was sitting in the first row and stretching out a bit. I turns out the power strip was down there. Unfortunately, the smartboard and projector just happen to be plugged into the power strip. I cut off the instructor mid lecture. Even though I had worked with the professor outside of school and got along with him well, I've never seen him look so annoyed, except when a cell phone went off in class. Thought I was going to be kicked out.
When I was in middle school they had a single kill switch for all the machines for the end of the day. It was placed right above the professors desk which was right next to the door. I don't know how many times we had some prankster come by and open the door and tap that button before sprinting.
I once was working at a large company with an illuminated light switch near the exit door for an EPO. The red bulb in the switch eventually burnt out.
Eventually, an electrician was called in to replace the bulb. He did exactly that.
Then, he decided to test his work, by, um, completing the circuit.
For a while, they really weren't sure if the Unisys systems were coming back. They hadn't been power cycled in quite a while.
I believe another employee actually got the story (in much greater detail) posted to the "Shark Tank" on ComputerWorld at the time.
This sig was generated randomly by one million monkeys with Speak 'n Spells. . .
This story has been around for years and years. In case you haven't heard it, here it is again.
***
Magic Switch Story
Some years ago, I was snooping around in the cabinets that housed the MIT AI Lab's PDP-10, and noticed a little switch glued to the frame of one cabinet. It was obviously a homebrew job, added by one of the lab's hardware hackers (no-one knows who).
You don't touch an unknown switch on a computer without knowing what it does, because you might crash the computer. The switch was labelled in a most unhelpful way. It had two positions, and scrawled in pencil on the metal switch body were the words "magic" and "more magic". The switch was in the "more magic" position.
I called another hacker over to look at it. He had never seen the switch before either. Closer examination revealed that the switch had only one wire running to it! The other end of the wire did disappear into the maze of wires inside the computer, but it's a basic fact of electricity that a switch can't do anything unless there are two wires connected to it. This switch had a wire connected on one side and no wire on its other side.
It was clear that this switch was someone's idea of a silly joke. Convinced by our reasoning that the switch was inoperative, we flipped it. The computer instantly crashed.
Imagine our utter astonishment. We wrote it off as coincidence, but nevertheless restored the switch to the "more magic" position before reviving the computer.
A year later, I told this story to yet another hacker, David Moon as I recall. He clearly doubted my sanity, or suspected me of a supernatural belief in the power of this switch, or perhaps thought I was fooling him with a bogus saga. To prove it to him, I showed him the very switch, still glued to the cabinet frame with only one wire connected to it, still in the "more magic" position. We scrutinized the switch and its lone connection, and found that the other end of the wire, though connected to the computer wiring, was connected to a ground pin. That clearly made the switch doubly useless: not only was it electrically nonoperative, but it was connected to a place that couldn't affect anything anyway. So we flipped the switch.
The computer promptly crashed.
This time we ran for Richard Greenblatt, a long-time MIT hacker, who was close at hand. He had never noticed the switch before, either. He inspected it, concluded it was useless, got some diagonal cutters and diked it out. We then revived the computer and it has run fine ever since.
We still don't know how the switch crashed the machine. There is a theory that some circuit near the ground pin was marginal, and flipping the switch changed the electrical capacitance enough to upset the circuit as millionth-of-a-second pulses went through it. But we'll never know for sure; all we can really say is that the switch was magic.
I still have that switch in my basement. Maybe I'm silly, but I usually keep it set on "more magic".
GLS
(1995-02-22)
"Tell me doctor, with all of your defenses, are there any provisions for an attack by killer bees?"
Yeah someday they'll get Unicode support in Slashdot, but I wouldn't hold your breath. It'll be right after they come up with an "Edit" button on comments.
Could be worse; could be straight ASCII. At least we get accents this way. And the symbol. (We'll see if it disappears when I hit submit..)
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
So much for Preview ... it eats the non-ASCII characters on Submit, even though they're correctly displayed in Preview.
... nice.
Nice, Slashdot
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
...I tried pushing the button below the radio in my parent's car. That same day I got a long talk about auto cigarette lighters...
Ford: "What happened?"
Arthur: "A sign lit up saying 'Please do not press this button again.'"
(Douglas Adams)
Then again, this would make really want to push the button...
I had custody of the kids (2, 5 and 7 at the time) every other weekend. Saturday night, I get a call from work, and it's something that can't be fixed from home. I can't find my ex, I can't find a sitter, so I bring the kids in. One of the operators volunteers to show them around the computer room while I work on things. Five minutes later, the IBM mainframe mysteriously halts. Yes, one of the kids had wondered why there was a big red button on the console. My problem was suddenly minor, so I took the kids out for ice cream.
Nothing for 6-digit uids?
When I was at Purdue, an engineering club was given an office with a big yellow button on the wall. Late one night... figuring it couldn't be connected to anything... and slap-happy from studying late... someone hit the button and took down the whole engineering computing network :)
There are 10 types of people in this world, those who can count in binary and those who can't.
Ever have to escort a bunch of suits through a new data center and have the Chief Operations Officer open the goddamn EPO and punch it?
I have.
Last year we were undertaking an UPS upgrade in one of our server rooms at work. As we were working on the power system running all of our servers, we had arranged to shut them all down before the work on the UPS could proceed. Included with our many Dell servers were a few SUN servers.... ;)
;)
As I was the only remaining SUN Solaris amoung my coworkers, it was my task to power down these servers before the work on the UPS could proceed. Unfortunately for me, the root password had expired, and could only be reset through the local console. I didn't have a null modem cable, as the previous admin had taken with them on their way out the door (we were apparently too cheap to purchase a terminal concentrator...)
In my despiration, a plan was hatched. We went to a local department store, and purchased four 100ft extension cords, connected them to power strips, and pluged them into power outlets in our office suite outside the server room. One by one, we swapped the power connections onto the extension cords, maintaining redundant power the entire time.
Once we had gotten the servers off the server room power connections, the UPS work proceeded without any problem. Little did we know, the real fun was just about to begin.
As a last step, our facilities staff were to test the EPO switch in the server room, to ensure it was functioning as it should. However, the had apparently overlooked the fact that the EPO was wired to kill the entire suite electrical circuit.
About a week later, we had finally recovered all the drives on the SUN Solaris servers, and everything was back up and running again. Needless to say, that was a week I'd rather not have to ever relive...
I adminned for a LAN party once. We did it in the school cafeteria from 10AM to 11PM. The guys who set up the boxes had half of the machines plugged into a single outlet. Apparently this half of the place was pwnz0ring the other half because I remember things getting louder from that side. Then they got louder still in the form of some words I had never heard before when I walked past and tripped over it. I don't remember much after that. Good times.
Cmon now Imre.. time to confess :)
:)
this is an inside joke.. anyone working at a particular ISP in Ohio will get this
[I also had an experience in high school that taught me how much the administrators knew, after I made a key to open every P.E. combo lock in the school. They told me that their locksmith said that what I had done was impossible.]
The determined Real Programmer can write Fortran programs in any language.
Developer 1: There is a security hole in your code. If you send a carefully formed packet, it'll execute any arbitrary command. And its REALLY bad, since the server is running as root!
/'. On a shared development machine. That NFS mounts home dirs for 2000 developers.]
Developer 2: No there isn't. See, watch... [sends a packet with the command 'rm -rf
It only took IT 3 days to restore home dirs from backup.
Do you think everytime someone pushes one of those EASY buttons from staples a datacenter somewhere loses power? Just curious....
Not that I ever expect George Dubya Bush to ever post on Slashdot, but this is one story where I seriously wouldn't want to suddenly find him posting.
I hit THE Big Red Button. My job is a job to make decisions. I'm a decision -- if the job description were, what do you do -- it's decision maker. If mistakes were made, I accept responsibility. Meheheh. Now we have to stay the course to embiggen the Nucular. Meheheh. I think -- tide turning -- see, as I remember -- I was raised in the desert, but tides kind of -- it's easy to see a tide turn -- did I say those words?
-
- - You can't take something off the Internet! That's like trying to take pee out of a swimming pool.
As my coworker and I were leaving the server room (he was taking me to the airport) I noticed that the big red "Emergency Shut-off Button" could be accidentally pushed. So, I mentioned it to him and replied "I'm pretty sure it doesn't work" *click*. Before I was even able to yell stop he had clicked the button. We, along with the local mechanic, and his boss spent the next 30min trying to figure out how to undo it. We eventually had to leave for the airport and a few hours later they found out what was tripped. The button ended up turning off not only the server room (which was a really old mainframe room) and half of the office area. Needless to say he ended up with a bunch of staples Easy buttons over the next few days.
Working at a computer center, I think the best design I've seen was the "Big Red Button" was actually 2 buttons, spaced far enough apart that you couldn't hit them both at once with on hand, but close enough together that they were obviously related. They were also much higher off the raised floor than any other switches, and clearly marked.
Just as trivia, that type of circuit is common on industrial equipment (think of the big press from the end scene in Terminator 1) and is called a Two-Hand No-Tie-Down. Basically there are two switches, and they have to both be depressed within a certain interval in order to close the circuit (generally 0.5s or so). If you "tie down" one of the switches, or have something leaning against it, or whatever, pressing the second switch won't trigger (otherwise it would be just a simple AND gate).
The circuits to do it are pretty standard and easily available. What's cooler, is that you can actually get a basically-identical circuit that uses compressed air or other gas instead of electricity (for use in chemical plants and other explosive atmospheres). One of the cooler things I've gotten to see made was a pneumatic "circuit board" cut out of Lucite for this purpose. I've always thought they would make a nice demonstration device for teaching kids about electronic circuits.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
Two solaris oopsies.
One: Somehow, I don't know how, I accidentally deleted
Two: Not wanting to accidentally halt the machine without really meaning it, I moved the halt command to halt.ireallymeanit. I then replaced halt with a small shell script that echoed "You don't want to halt this machine" (sleep a few seconds) "If you do, type halt.ireallymeanit" (sleep a few seconds) exit.
Then, to test it, I type halt. Without (duh) first typing which halt to make sure there wasn't a halt command before the
Needless to say, it's not Solaris' fault, but somehow I always managed to screw up that OS without meaning to, so I have developed a healthy fear and loathing for it. I'd like to think I've grown up a bit since then - this has been like 3 or 4 years now, and I've learned a helluvalot since then.
~Wx
sig?
Yello?
Did it go "bum bum, Oh Yeah!" while it was going down??
Shawn's Tech Articles
I can't believe you're pulling this out of the queue now. The story is 3.5 months old.
Credit cards in cleartext? Eh. Sounds like a bonehead mistake, blah, blah, blah. Anyway, where did you say you worked? Why? Oh, no reason. :-)
Coderz 4 Life
You know that old thing that people (that have to deal with Windows) print out and put on their cubicle walls that says, "Bang Head Here"? Well, OK, blood does tend to stain brown after a few days.
I haven't had any "Big Red Button" disasters yet, but have had plenty of the "Little Red Wire" variety.
Knowing Google's lust for data collection, the Soviet Union is still alive and well inside the psyche of Sergey Brin....
I, along with a group of kids from my high school, took a guided tour of not one but two data centers when we visited Intel Santa Clara. At least for people our age, the "don't touch anything or we kill you" factor seemed to work well enough, and we had a great time learning about how data centers are set up and run.
A co-worker of mine (I work for a major financial corporation) had an old NeXT box running underneath his desk. Well, one day, while he was out sick, the box started whirring like an anemic helicopter. Drove me nuts, and half of the floor. Finally, someone said, "Just turn it off." So I did. The next day, he came back, confused as to why it was off, and turned it back on, but while he was at lunch, the noise started up again. So I turned it off again. This continued at least twice more. All the while, on the other side of the building, people were going absolutely bananas. Red lights were going off, everything was dying. What I hadn't realized was that this underpowered little machine was running an antiquated operating system was actually running a critical documentation generating program. My co-worker was convinced that the power supply was just going out on it, I was convinced it was trying to slowly drive me mad, and the financial honchos were quite convinced that the Apocalypse had come. :)
Case 1, first time I met a Mac, stuffed stiffy (small format floppy to you yanks) into drive, copied data off, search for button to eject stiffy, press nearest, Mac resets.
Mac had only software eject, the small beige button next to the stiffy drive was the reset.
Case 2, Mind killing migraine, pain like knifes been driven into my eyeballs, looking for place to plug USB device into, missed usb port and plugged it into the reset button....
it's not a button, it's a switch. and they now have a lil plastic sliding door that you have to flip open before you can hit the kill switch
~~~ They call me Little John, but don't let the name fool you...in real life I'm very big.
Flipping this baby means now the interlocks are disabled and will run until it melts through the floor.
http://en.wikipedia.org/wiki/Bizarro
In GOD we trust, all others we monitor.
First thing I noticed about this post was 3 prime numbers.
Brought to you by the numbers π, e, and 0x1B.
The CDC Cyber computers had a wide operator's display with a desk top in front. Just above desk level on the left was the Big Red Button. Just behind where many operators put the log book in which various activities were recorded. Eventually a cover was put over the Button after too many log books got pushed into the Button.
...on the way to the toilets.
...and some pipes.... ...this used to be a factory... ...compressed air? Sprinkler valve? What?
It is on a chain that goes way up to the roof...
I don't know.
I wonder, I wonder.
Other people wonder.
Maybe it has been pulled many times? Maybe someone will pull it and sprinkle all the PCs? Maybe someone pulls it and we all get flushed down the intertubes. (Funny, my kids have never seen a toilet with a chain)
Life is full of little puzzlements.
(It all goes wrong tomorrow, IT WASN'T ME! I HAVE RESISTED TEMPTATION FOR YEARS NOW!)
I used to work at a startup ISP as the one and only sys admin back in the early 90's. As per my contract I was supposed to get a $3500 bonus after a year. Well, the boss was a sob and decided not to award me this bonus. As you might imagine I was slightly pissed off, and decided to take a week off in protest. He fired my ass, pulled my workstation off the network and locked my hard drive in his safe in case I had setup a nasty cron job. (nope, I'm not that evil) Thing was, guess what system the dat backup was hooked up to for all the servers? You guessed it! It ended up costing them way more than my bonus when a crash happened, since nobody else in the place even had a clue regarding linux, let alone the custom kernel to allow the raid cards to function correctly, etc. Also, not to be bitter or anything, but I hope the $20k chandelier that was in his house fell.
Life was hell, then I discovered Linux...
I was doing support, bench work for a small company. A small .com startup had just finished a six month site re-design. Their top developer was relocating to New Jersey. They wanted me to ghost their drive. They handed me another drive which wasn't empty.
.com went bust shortly after :(
I didn't bother to wipe the new drive before the ghost. The classic newbie mistake.
I started a ghost on the drive, the wrong drive.
When I realized what had happened, I said, "it's ok, where are your backups?"
The developer went outside and threw up!
they had to pay for data recovery, luckily because they signed the liability release.
data recovery was able to recover his buju banton mp3s just fine!
That
They're using their grammar skills there.
I think that this image is appropriate... I do not own this image... It belongs to Sacha Goedegebure of BlenderArtists.org
Mommy. What's a karma whore?
1970, freshman computer science student, large PDP-10 system which supported the entire university computing. Working on a program which could find deleted files again by picking up the traces still on the disk. If the newly freed blocks hadn't been overwritten yet, that is... Testing the program, accidently wiped out the MFD, Master File Directory. (Man, it's been a lot of years since the though "MFD" went thru my head! Ah, the fond memories!) All files, many hundreds of students, several commercial operations, all gone in an instant! Entire system gone! RAN for the big red button. And yes, it actually was a big red button! Rebooted with no users, spent about 24 hours running the program. It took that long, it read each disk block sequentially with no buffering, wrote found files to tape, then restored from tape when all was done (so as not to wipe out files as it found files)! And I got all the files back!! Administration not too happy, but at least I proved the worth of my program!
I didn't even know we had a panic button.
sometimes, nothing.
on the phone with my web host's tech support (M6.net) I have come to the conclusion that they have literally lost my database. Their "daily" backups have no record of my database (which they've hosted for 3 years). My db is nowhere to be found on any of their machines. Worst of all I was asking them to give me a backup of it. That's what they screwed up.
Any suggestions on what I could do to press the issue with them? I live in the USA and they are based in Australia. Any sympathy/advice would be appreciated.
Back in the 70's when I worked in the IT department at TI Austin we had a fairly large computer room with a Big Red Button, and no clear box. We also had no UPS and power outages were a regular part of life (cheap bastards wouldn't listen), so as sysadmin I would deal with these outages. I was getting sick of it, so one day when I had just gotten done rebooting 55 systems from the front panel (7 words of 16 bit binary switches for each) and then read cards in to do the boot on the servers my boss was watching me go through the motions. When I was done and everything was up we talked for a few minutes, I walked out of the room and as I passed the door, right above the Big Red Button there was a light switch. I turned it off and you could hear him gasp. I grinned and turned it back on. They still wouldn't buy a UPS.
I used to live on a 120 ft ship that was modified a few times since being a fishing vessel. There was a small room that was rather out of the way of anything and seldom used except for the deckhands to get out of nasty clothing and slip into the bathroom/showers through the next door. In this room though there was a really rickety "closet" door and if you jiggled it the wrong way or opened it completely alarms would start sounding in the engine room on the next deck down. If you pressed the "big red button" inside of the closet, it then flooded the engine room with CO2. It's designed to put out fires, and I think it may have a twist and turn system or other mechanism to prevent accidental pushes. I guess it's also good that a good rule of thumb is to give have the chief engineer with you while you wait 30 - 60 seconds after sounding the alarm before pushing that particular button.
While it's hard to prevent that kind of deliberate sabotage, a recent session at Data Center World focused on strategies to mitigate the risk of EPO disasters. The bottom line: put a cover on the button, make it well-marked, and have separate buttons or switches for power distribution units, UPS power and HVAC (the code often allows this). And as a deterrent, have video surveillance of the exit area where the button is placed.
Great story on the Big Red Button. A well known company built a new datacenter, and started populating it with servers. Everything was going great. But the datacenter had the Big Red Button. It also had a somewhat smaller Yellow Button. You see, the datacenter's sprinkler system (yes, no halon... water sprinklers) worked in two stages. The first was to fill the tubes with water, and the second was to open up the heads in the sprinklers.
But, in their wisdom, they offered a Big Yellow Button to hold off on opening up the sprinklers. As long as the Big Yellow Button is held down, the sprinklers won't actually spray water. So you can see that this could be very useful in a false alarm situation.
Well, one day, an employee decided to play around with the Big Yellow Button. Yes. Do you know what is coming? No, you probably don't. And neither did the employee. Well, earlier in the story, I mentioned that this was a new datacenter. And it looks like they didn't do a great deal of testing of its emergency features before they put it into use. You see, the people who wired up the Big Yellow Button had swapped the wiring at the other end with the wires to the Big Red Button. But nobody tested the buttons out to make sure they worked. Since you are a Slashdot reader, I'm sure you understand the result of this unfortunate wiring mistake, and lack of testing.
The employee, however, was not fired or significantly disciplined for the significant outage (and disk damage from a sudden power loss) that resulted.
The shiny new data centre was so close to being finished it was already operational. The redundant aircon worked and had been tested. The UPS and generator had been tested. The servers were installed, running and live. And then someone needed to do a little bit of work under the floor. They lifted up the tile and accidentally dropped it.
Now, imagine the impact of that thud travelling through the raised floor, up the wall and into *the* relay in the BRS, tripping it. And then you can probably imagine the sound of all that hardware shutting down in a hurry, and the "Oh shit" coming from the unfortunate tech.
I work at a general motors distribution center in Colorado. When tehy buiolt this new building, the contractor didn't do anything right. The fire control system shutoff and water dump are inside the datacenter, the datacenter is on an outside wall that leaks very badly when it rains, and the Big Red Button was discovered to cut power to the entire 400,000 sq. ft building but didn't have a cover (lack of cover due to GM saying they could not justify the $7.50 each price).
Yep, the IBM tech was setting up our server which was placed directly in front of one of the 3 Big Red Buttons (all of which kill the entire plant), about 27 inches away from it. After connecting some cabling, he stood up and his back pressed the button. Well the building was out for 27 hours as a certified electrician (the union electricians are not allowed to touch live wires or fuses/breakers as they are dangerous) had to be called out, the main power into the building restored and EVERY fuse and breaker in the building reset or replaced.
However, the next day we did receive 3 plastic covers with key locks for the buttons.
Here's the Molly guard story - although the term is pretty much self explanatory.
(And while we're at it, BRS.) "It is alleged that the emergency pull switch on an IBM 360/91 actually fired a non-conducting bolt into the main power feed."
Quattuor res in hoc mundo sanctae sunt: libri, liberi, libertas et liberalitas.
On a warship of a US ally. We are moored in a US port and doing the PR thing, allowing the public to tour our ship. Lots of people take the opportunity, since USN tends not to allow public tours after 9/11.
Suddenly the collision alarm sounds. Crew push through the crowd to their positions, cursing weekend boaters; the tour guides are moving people away from compartment hatches in case they slam shut. I pop my head up from hosting VIPs in my cabin to the bridge above, thinking through the logistics of moving hundreds of tourists off or around the ship to give the crew room to work. A woman is standing at the rear of the bridge, notices me in my whites and says "I didn't think it would work". I, patiently as I can, point out that this is a real warship and we have a large crew which works hard so that it all works, all the time. I reach over, cancel the alarm, request a report from the lookouts to CYA, and stand down the responding crew.
I invite her on a "officer's tour" of the ship, to get her off my bridge before she does anything else stupid. One of the WOs, knowing his job, appears at my side and takes her off my hands. He's built like a god and she's just simpering at the attention -- too dumb to realise what his actual opinion of her might be.
I serve as IT support for a company which manages several properties and facilities in a national park. Just two days ago, we experienced an entire property (200+ machines requiring access to a variety of servers on another property and in two different cities elsewhere in the country) suddenly disappear from the grid. No connectivity of any sort, nothing. A team was dispatched who proceeded to troubleshoot everything from NIC cards to cabling to virus scans on the affected machines.
It turned out that a main switch had the power cable removed by someone who needed the outlet to charge her cell phone. Since she is a retail manager with a different company who happened to share one of our retail spaces, I traded out the knowledge of her mistake for goods from her store, including a nifty pair of Timberland boots and some sweaters.
1. Wait for BRB disaster.
2. Determine it's not your fault nor anyone on your team.
3. Blackmail the culprit.
4. Profit!!
The integrated tape / hard drive unit from Hewlett Packard had two red switches: tape -> disk and disk -> tape.
Yep.
Don't piss off The Angry Economist
Arguably the biggest shutdown-button screw-up in history ...
... didn't register any parameter changes that could justify the SCRAM. Commission ... gathered and analyzed large amount of materials and, as stated in its report, failed to determine the reason why the SCRAM was ordered. There was no need to look for the reason. The reactor was simply being shut down upon the completion of the experiment."
... "
From http://en.wikipedia.org/wiki/Chernobyl_disaster :
"At 1:23:04 the experiment began. The unstable state of the reactor was not reflected in any way on the control panel, and it did not appear that anyone in the reactor crew was fully aware of any danger. Steam to the turbines was shut off and, as the momentum of the turbine generator drove the water pumps, the water flow rate decreased, decreasing the absorption of neutrons by the coolant. The turbine was disconnected from the reactor, increasing the level of steam in the reactor core. As the coolant heated, pockets of steam formed voids in the coolant lines. Due to the RBMK reactor-type's large positive void coefficient, the steam bubbles increased the power of the reactor rapidly, and the reactor operation became progressively less stable and more dangerous. As the reaction continued, the excess xenon-135 was burnt up, increasing the number of neutrons available for fission. The prior removal of manual and automatic control rods had no substitute, leading to a runaway reaction.
At 1:23:40 the operators pressed the AZ-5 ("Rapid Emergency Defense 5") button that ordered a "SCRAM" - a shutdown of the reactor, fully inserting all control rods, including the manual control rods that had been incautiously withdrawn earlier. It is unclear whether it was done as an emergency measure, or simply as a routine method of shutting down the reactor upon the completion of an experiment (the reactor was scheduled to be shut down for routine maintenance). It is usually suggested that the SCRAM was ordered as a response to the unexpected rapid power increase. On the other hand, Anatoly Dyatlov, chief engineer at the nuclear station at the time of the accident, writes in his book:
"Prior to 01:23:40, systems of centralized control
The slow speed of the control rod insertion mechanism (18-20 seconds to complete), and the flawed rod design which initially reduces the amount of coolant present, meant that the SCRAM actually increased the reaction rate. At this point an energy spike occurred and some of the fuel rods began to fracture, placing fragments of the fuel rods in line with the control rod columns. The rods became stuck after being inserted only one-third of the way, and were therefore unable to stop the reaction. At this point nothing could be done to stop the disaster. By 1:23:47 the reactor jumped to around 30 GW, ten times the normal operational output. The fuel rods began to melt and the steam pressure rapidly increased, causing a large steam explosion. Generated steam traveled vertically along the rod channels in the reactor, displacing and destroying the reactor lid, rupturing the coolant tubes and then blowing a hole in the roof.[7] After part of the roof blew off, the inrush of oxygen, combined with the extremely high temperature of the reactor fuel and graphite moderator, sparked a graphite fire. This fire greatly contributed to the spread of radioactive material and the contamination of outlying areas
About a decade ago, I worked in the IT Department of a medium-sized hospital of a city in East Texas. As the new guy, I worked the overnight shift, which was fine by me.
Sometime after midnight, I was walking through the data center, when my left cheek had an itch. I scratched the itch, then dropped my hand to my side. As my hand fell, it struck the power switch that was located on the top of one of the hospital's servers. You would be amazed how many people are using hospital terminals at 1 in the morning, and how fast they can all call the IT Department phone number.
My boss laughed about it, saying that everyone in the department had done the same thing at some time. Another co-worker asked me how it happened. When I re-enacted what had happened, I narrowly missed hitting the power button a second time.
A few months later, the hospital bought all new servers. I was glad to see that the power buttons where all located in recessed cavities, behind plastic protective plates, about 6 inches above the floor.
Taking stuff apart since 1969 (TM)
A friend of mine was present at a CO when they were upgrading the power system. They forgot to disable one of the automatic power-transfer devices, so it (duh) automatically transferred power. The power was being rewired, though, so it placed a dead short across the battery banks. These are designed to supply 48V at around 4000A (yes, four thousand amperes). The first things to melt were the battery cases, pouring boiling battery acid out onto the ground floor. It was not cool, not pretty, and not pleasant.
☠
anyone remember ol' professional write? (http://en.wikipedia.org/wiki/Pfs:Write)
Some people seem surprised that i remember this vividly. I was 3 years old, my mother was just starting to work on her law degree, and was writing up something important via professional write. The "microcomputer" we had i believe was built by my uncle, and the power button on that thing was big and visible. From my vague memory it was a red circular button, with at least an inch in radius. Just like a frustrated horny bull running towards a red cape, my nubby little index finger launched towards that big red button. Result: (insert the dirtiest swear words here), my father comes running in, my mother starts crying (my guess is it was important, and back then autosave was only a dream), and the rest of the memory is too fuzzy to remember. Perhaps for the reason that it was something that shouldn't be remembered. Now that i look at my watch, it's almost time for my psychotherapy appointment! Cheers!
I had the problem of people pressing the red button on my rack's UPS to stop it from beeping every time we loose power.
In an earlier job I was doing sys support for unix.
:-)
:-)
I was trying to shutdown a terminal machine so I could move it but inadvertanty typed the shutdown command into a window I had open on the main production server. I was assured that everyone does that at least once
One of the managers was particulally dark at me coz they lost 3 hours of editing on a document. My response was "Don't you save it every ten minutes?" (as people who use Word do)
Their fault really for employing a windows programmer to to unix sys support.
This is a similar story.
A few years ago, I got a brand new Dell laptop. As you may know, it has a nice green light underneath the power button. Of course, there is nothing more tempting for a kid than a nicely lit button on a computer keyboard.
I guess it took about 10 shutdowns before she understood that pressing that button is not the preferred way to start a game.
Back in days with DigitalUnix (former OSF, todays HP whatever) I needed to kill several httpd processes on Digital Unix. ;)
Typed: killall httpd
and voila! The whole server went down. killall on Linux != killall on OSF
Cuba++ let's make ++ better
Don't know if it qualifies as a Big Red Button story but the effect was the same. I work in a satellite TV broadcaster. We were standing around in Master Control one afternoon discussing stuff and the cleaning lady snuck in when no one noticed. She proceeded to use a wet rag to wipe down the main switcher and switched every channel to black. It was pretty amazing to see a wall full of monitors (about 100 of them) suddenly go black. For a moment we all thought the SDI router must have melted until we noticed the cleaning woman polishing the desk.
The other thing she did was she worked out that she could get into the machine room with her pass if she went via the emergency exit. We kept finding puddles of water under the raised floor that we couldn't explain until one weekend I noticed her carting a bucket and mop into the machine room to give it a good scrub with a liberal amount of water.
She doesn't work here anymore.
1. When I was working with big iron (IBM 3090), I went to a customer to upgrade some software. We started friday 4 pm when the production line stopped for the weekend. The customer had a big party the same night, so we wanted to finish the upgrade in a snap.
Needless to say, some issues came up and the time schedule slipped a bit. The customers team lead became increasingly tired and bored. At some stage, he needed some support for his head, so he leaned against the wall. Do I have to write, what was on the wall? Correct. The Big Red Button. So after IML and the whole fuss, we completed the upgrade around midnight.
2. Another customer, still IBM 3090. This customer operated around the clock, so there was always somebody in the data centre. One evening, the staff played cards. One of the guys lost a round and got very upset. He threw his cards on the table, yelled "WHAT!", simultaneously throwingh his arms to the sides. Unfortunately, he was sitting close to the wall on which the BRB was placed and of course he hit it.
I contract to a major tier 1 provider and was handed a work order to go remove a certain fiber router for routine upgrades.
now this is normally not a problem because there is identical fail over equipment that switches over at the first sign of a loss.
However.. there was a glitch in the work order, that nobody caught. they had actually already removed the router and not yet reinstalled it. but accidentally issued a work order to remove the fail over so it too could be upgraded..
so off i went work order in hand, unlocked the nondescript site housing the backbone for the entire western network. Read off the serial number on the work order comparing it to the serial on the router.. hmm ok i got the right one.. seems to be abnormal amounts of traffic (usually NOC forces the unit to fail over before removal to prevent any glitches) so i phone the NOC.. they tell me ya go ahead.. it'll be fine..
so i pull the power, and start removing the main STM-64x connection and a dozen OC-48 peer trunks when i receive a phone call on my cell..
On the other end was a frantic tech at the NOC desperately trying to understand what went wrong.. then a few seconds later i swear had to be the entire swat team showed up, and things really got out of hand at that point. apparently someone at monitoring, didn't check with noc first before pushing the panic "the terrorists are attacking the internet" button..
so after the initial kerufule and the police had determined that i was not an immediate threat.. i took my work order and thrust it in the direction of the lead officer and with my best hogans hero's accent stated " I vast only followink orders". (i managed to get a chuckle out of him)
they had of course relieved me of my cell phone and were talking to the NOC on the other end whom im sure was informing them that things are not going to get re-connected whilst i was being "detained".
so after all that, and about 1/2 an hour to reconnect it all, the western portion of the continent had its network connection back..
I made sure i held on to that work order though.. it was the only thing to prove the screw up was not mine and that i was truly "followink orders"..
You are not entitled to your opinion. You are entitled to your informed opinion. -- Harlan Ellison
One. A colleague (at an IT company) managed 2 redundant servers. Server A crashed, and server B took over as expected. He had to turn off server A to change a harddisk, and as usual with unresponsive hardware, the easiest way to do so is to keep the power button pressed for 4 seconds, which he did. Unfortunately, the labels "Server A" and "Server B" were swapped during installation years before... Two. Another colleague walked through our office, past a customer's server that was temporarily in our office. This chap is known and respected for his Novell knowledge ("no, you HAVE to check that checkbox during installation. Otherwise, he won't support it."), but has no Windows knowledge. Some joker installed the sysinternals BSOD-screensaver, displaying and repeating a blue screen of death, and a Windows server splashscreen, as if it was continuously rebooting. Being the helpful chap that he is, he thought the best thing to do was to shut down the server between these reboots, and notifying the administrator responsible for this server. After said administrator calmed down, the Novell guy explained sarcastically: "Well, i thought we had a serious problem, and fortunately now we have. But next time, let's just stay professional with installing screensavers, shall we?"
Is all about not pushing the big red button no matter how much you want to.
Harking back to the good old days of high school...
We had several antiquated BBC micros in one of the classroom blocks public areas - in theory for getting work done during break but since no other classes used those machines they usually ended up having games on them for those that knew how to find (or write) them.
Bored one lunch time I typed in the same 20 or so lines of basic on each machine and with the help of a friend hit enter at the same time on each.
The screen now flashed from black to red and black, beeped every second and read "This computer will self destruct in: 5:00" counting down every second. After a bit of a giggle (ok we were 12) at how this looked we walked away and wondered who would find it.
It turned out to be one of the dinner ladies (for those that didn't have this concept in school - they are non teaching staff that wander the open building and grounds during breaks keeping an eye on things). Being about 60 or so she obviously beleived what the computers said and hit the fire bell!
One evacuation later and a short investigation of the computer screens (which if I had got the code right should have one letter on each screen - B O O M - I never did see the final result so I don't know) everyone returned to classes.
Didn't really hear much until my next IT class, at the end the teacher took me to one side - his comments were basically:
"Very nice trick, but please, don't do it again, ok?"
Was fairly obvious it was me, only the lowest years of the school were allowed in that area, I was the only one in that categry that could have done it. I do remember the teacher was trying to stop himself from grinning.
$_="Slashdotter";$syn="OTT";s;..;;;sub _{print shift||$_};s!ash!Perl !;s=$syn=ack=i;tr+LLEd+BLAH+;_"Just Another ";_
It wasn't a red button, thank god. I was in a tech school as part of becoming a missile technician on an ssbn...we were in the missile control lab...i got bored with the particular procedure read as we had done it a billion times already. So i wander over to a panel, and maybe I flip some switches here and there. Who knew they were actually connected to a huge lab on the other side of the building. The lab was um...in need of repair and I was almost eaten alive by the command. Until I pointed out that it wasn't 'red tagged' as per the SOP. I was still was on the shit list for a while, but nothing too bad. Yikes. Needless to say i let someone else press the buttons from there on out.
-those people who tell you not to take chances, they are all missing what lifes' all about-
I wouldn't call it a disaster, but equally stupid to IBM's BRS was our BRS.
The technical school I attended wasn't really big. Only 200 students doing electrical and computer enigneering and some 500 students in various forms of education like economics, logistics and even a teachers education.
Because it was so small, only the computer engineering had computers to speak of. HP Apollo systems, old Sun 3's and a spanking new network of IBM PS2/50's. Yeah baby, networked! (nobody used the Sun 3's for other things than playing LPMud, so to the general public they were not present).
So we had this big room filled with 16 PS2/50's on a network. Of course we techies didn't need the network anyway (Internet? What's that? That's how backward we were. Usenet wasn't even available to us).
So the logistics department had this big networked simulation program they were running every month. That's what the network was good for.
The room had it's own circuit breaker conveniently located in a plastic box with a BRS on it. Mounted on the feeding line which came in through the ceiling next to the door.
BRS, door, see where this is going?
Man that BRS did protrude far out of that circuitbreaker box. It had already happened once or twice that some joker "accidentally" hit the BRS after class, but what really made the technical staff concider moving this box was when one day, thoroughly pissed off at not being able to use the computer again all day, one of my classmates opened the door when the logistics guys were doing there simulation, said Hi and hit the BRS.
Needless to say my classmate was in big trouble afterwards.
Not me but the MD of a company I support had an electrician in to wire up some new lights. They couldnt figure out which circuit was for the upstairs part of the building so decided to start flipping breakers until they found the one that killed the lights. The electrician voiced his concerns about the servers to which the MD said "dont worry its on a UPS, it'll be fine". Well yes it would have been fine if it wasnt just the monitor that was plugged in to the UPS and someone had actually tested it in the last 3 years......5 days later, 1 motherboard and a lot of late nights, the server rebooted.
While not a big red button, there's a door lever for many transit-style busses that controls the pneumatic doors. In most (if not all) places, they have a brake interlock when engaged. They're designed so they can't be engaged above a few miles an hour (at least on the newer busses, unknown on the older GMC busses of the 50's-60's). If you've ever been on a bus going even a mile an hour when the brakes are stomped on, there's an awful lot of momentum involved. To give you an idea, I was going probably 5 mph when a biker went in front of me- I stomped the brakes and my clipboard shot up and hit the steering column (goes vertical from the floor) making a gigantic clunk and everyone on the bus thought I hit the guy even though he was still more than 6 feet away. Anyway, the point of that was, I had an empty bus one day and always had wondered if the door lever would still engage when moving while in a parking lot, decided to 'push the big red button' and open the door- bringing the bus to a screeching halt and damaging the interlock mechanism. I guess I wasn't going fast enough to be over the limit where it won't brake, but it was fast enough I hurt the bus and myself. Still, the temptation to try it at 60 exists ;)
I've seen that on some computers that were originally designed for the Navy. It makes sense in a shipboard environment, where you don't want the main fire control computer going off-line for some minor problem in the middle of an enemy attack.
Mea navis aericumbens anguillis abundat
Wow, got bitten by HTML tags myself.
Back in the late 90's when I used to work in local Government here in the UK, we had a power blip which set off the alarm on the Mainframe's UPS. I was in the machine room at the time and the noise was horrendous.
Anyhow, I went over to the UPS to press the mute button, but because I was distracted by the noise - and the fact that the power off button was next to it, pressed the wrong one. It was nice and quiet for a few seconds until the effects of the noise dissipated, followed by a lot of swearing - the mainframe and most of the other servers were dead.
It ended up taking most of the day to bring that old beast back online, but one good thing did come out of it - we discovered that the office next door were powered by the same UPS and we were wondering why the UPS was struggling, all their PC's were plugged in to numerous extension leads into a couple of non-standard plugs they found to plug into.
Java gaming nut - http://www.retep.org/ or for the rail http://uktra.in/
Of the whole operations group I drew the highest card out of the deck, and got to decommission our IBM 370/125 by pulling the EPO (on the console) - it didn't work, but the big red button finally killed it! I did, however, manage to power-down an IBM 4331 (the chest freezer shaped one) by leaning against the end of it - the power switch is exactly at hip height!
We had an older SGI supercomputer that, as most do, run pretty much flat out with clients crunching numbers, some clients having jobs running for the last 2 years.
/etc
One, pretty normal, day I'd requested a root window and was cleaning up some temporary program directories and had typed:
rm -rf
when the telephone rang. It was a client that was unable to connect to the machine from their office. So, to sort things out, I then typed:
cd
and hit the enter key.
It was a while before I was allowed to have another root terminal.
.
Perhaps this one is too nerdy for /. - no forget that I said that.
/etc/passwd in vi, then immediatelty realized that this was not where he wanted to be. Now, normally one qould use ':q' to exit a file without saving, but he was in the habit of using ':x', which is a convenient way of saving and exiting at the same time. Unfortunately he forgot the ':', which makes it a command to delete whichever character you are standing on. When nothing seemed to happen, he automatically did it again, this time getting it right. Then he logged out.
/etc/passwd? I'll give you a hint: it begins with root:x:0:0 - so this guy had deleted the 'r' in root, saved the file and exited. And since nobody else was logged in as root, we were stuffed - one couldn't log on as root, since he was not in /etc/passwd, and logging on as oot didn't work either because he was still called root in /etc/security/passwd (this was on AIX - it corresponds to /etc/shadow). And using 'su -' from an ordinary user didn't work, since this command actually looks for the username 'root'. Unfortunately it turned out that booting in single user mode meant that you had only very minimal access to the disks, and getting the others online is not easy when you know too little about AIX and have a very complicated arrangement of disks and volumegroups. In the end we had to reinstall. This of course had to have the traditional, serious consequences: the guy was .... promoted.
You can do really interesting things as root; in a place I worked one of my colleagues wouldn't admit that he had done the following on one of our biggest and most important UNIXes: He had logged on as root and opened up
Now, what is normally the very first line in
I thought our arm-mangling defense robots were meant to stop ANYONE getting in?
try this disaster -> http://yourhell.com/bigredbutton.swf
;) )
(no idea who made it.. i just copied it from some site and can't remember from where.. but it's great
Find me at http://herbert.poul.at
Now, this story is told by my father, who has been programming since around 1970... It involves a "minicomputer" - I know I have the proper term, I'm just not sure what make or model... Punch cards were involved...
... Oh. Sh*t."
He had a "friend" (reason for quotes about to be apparent) who loved to come in quietly, sneak over to the minicomputer, and press the LAMP TEST button. For the slower among you, this would light up all the panel lights on the front of the minicomputer (back in the days when lights on computers meant something... heh). So he'd press that button and say "OH, SH*T!". This would always put the victim into an immediate panic, of course... Funny stuff.
Right.
So one day he comes into my dad's office, sneaks over to the minicomputer, hits what he believes to be the LAMP TEST button, but is in fact, the BRS, and proceeds to say "OH SH*T!
My father turned a shade of red (so he was told by witnesses), and uh... told the gentleman in no uncertain terms what would happen if he EVER walked into his office EVER AGAIN.
And... that was the last time that prank was done.
(This was in the days where the BRS did physically disconnect the power, or otherwise disable the minicomputer until a company tech came out to open it up and reset it)
A cheerful little bird is sitting here singing.
Our lab not only had a collection of test machines, racks and racks of them running automatic tests - but all our build servers and the CMVC server and all other assortment of interesting machines.
One day one of the higher ups was in the lab. It was one of those with a card lock entry to get in. He just assumed the big red button by the door was the one that released it, and what do you know, despite the button having a large sign in 72 point font right above it saying "EMERGENCY POWER OFF SWITCH", he pressed it, plunging the lab into silent darkness.
Yep, that button had a little perspex lid put on it after that incident.
Oolite: Elite-like game. For Mac, Linux and Windows
I have a Big Red Button in my server room (right next to the light switch - nice idea, folks!) which is rigged up to cut mains power in the event of an emergency.
Unfortunately it is not connected to any of the UPSes, and being a server room, more or less everything is on a UPS.
Tjernobil RedButton, used to shutdown the powersupply to the plant, is def the most famous RedButton booboo ever made.
Hey, thanks to progress, no more janitors are required for this situation. Just use Windows, which will under certain circumstances perform a reboot at 3 a.m.
I took a great deal of effort to toddler-proof my study. PC and laptop with exposed buttons at desk height or above. Synth moved from wobbly stand to sturdy wall-mounted shelf. Linux server, under my desk, rehomed into a blacker-than-black case, fancy lighting rig unplugged, all buttons, optical drives and recesses safely hidden behind a plain black door. O'Reilly Wall moved from bookcase to high shelves.
I even got a "decoy" keyboard for my 11-month-old daughter to play with.
Of course, she found the UPS switch in seconds. It had a nice glowy LED above it, and was sitting on top of the Linux server just at her shoulder height.
All three PCs, the whole study, powered down, and not in a nice graceful apcupsd way, just a sudden BOINK, follwed by darkness and silence, penetrated only by a happy gurgle.
Thank heavens for Linux software RAID mirroring.
(A couple of months earlier, she managed to cause Windows to prompt "Add new hardware - Searching for drivers" [blur-o-matic cameraphone photo] by sucking the end of my iPod USB cable. Unfortunately I didn't have any Win2K drivers for a 9-month old baby. I bet Ubuntu installs them by default, even though the GNU crowd complain they're not truly free.)
Annabel is one on Sunday. Wish her happy birthday.
Andrew Oakley - www.aoakley.com
Back in 198x, I was working for an aircraft electronics manufacturer on their prototype flight management computers.
Unfortunately resetting them involved touching a wire to a ground pin....which was near to a 48V avionics supply pin.... as I found out....twice.
Burning out the only two systems in existence did not make me popular.
Donte Alistair Anderson Roberts - hi son!
Karma: Chameleon
Our company finally got a new medium-size UPS, large enough to power two racks of servers and a router rack. It was placed on the floor in the midst of them, giving us a sublime sense of security and well-being. Trouble was, the manufacturer put the UPS's reset switch in the front, exactly at knee level. We had several instances of all of the servers rebooting in the evenings after everyone had gone home. Turns out that the cleaning crew was bumping into that switch while making their rounds. Took us a while to debug that problem. The fix: open up the UPS box and snip the reset switch wire.
I started out as a lowly tech on an IBM System/3, with a card reader. Every now and then we had to process the records for a client's pension fund. Since this was not a regular job, the cards were stored in another building. That night, it was snowing, and as I staggered back through the dark, I slipped and dropped a box full of cards...
After spending ages grovelling around in the dark trying to find small white cards buried in the snow, I got inside to find that...they weren't numbered.
It was a loooooong night...
Not me, but a colleague; Not quite as significant as BRB, but significant enough at the time.
;)
:)
We're on-site peforming a minor upgrade to one component of a national card-transaction switching network, on the main switch which in those days was a lowley NCR 3000 system...
The upgrade has gone well, so colleague pushes the button to eject the floppy (remember them!) containing the patch... Of course he the realises which button he's *really* got his finger on, but before thinking that he could single-handedly type 'shutdown' with his free hand, is interrupted by the operator wanting to do his 2AM backup; his finger then slips from the button.
With the system down nobody in the [popular holiday destination] country can perform Visa/Mastercard/JCB or cross-bank ATM and POS transactions (from the main 4 banks). Luckily it was 2AM, a low volume time...
Ops guy is too stupid to even realise the consequences, insisting it's time for him to do the backup, whilst my colleague is frantically re-booting, fscking the volumes, and running consistency checks on our databases, and telling him to fsck right off
Luckily, all is OK and everything's back on-line about 20 minutes later.
Stupid place to put a power button, right next to the floppy eject... We 'fixed' it by taping a match-box cover over the power switch
In what was once a dilapidated part of Manchester (UK), there is a red button in the side of a building that has been there at least 15 years, surviving all the regeneration in the area, with a warning sign not to press the big red button; there's a hidden hose, like, 15 feet up that squirts water all over you if you do. Every time I go past, there's always a fresh bunch of nosey kids hanging around it, looking shifty and crying out for a soaking, even after all this time.
CheShA: Manchester Breakcore / Drill and Bass Yes I'm a s
-- Trinity in high heels carrying a whip: The donimatrix - there is no spoonerism
...but certainly qualifies..
6 .jpg
The US Navy's P-3 Orion (in many variants) is a 4-engine maritime patrol airplane. The engines are Pratt and Whitney T-56 turboprops, a powerplant shared with the C-130, the E-2 and the C-2.
In the flight station, in the top center of the instrument panel are four big yellow handles that you pull when you need to shut the engines off in an emergency. Because they are used for emergencies, the are cleverly called "e-handles". Underneath each e-handle is a red button. This is the the button that releases the contents of the high-rate-of discharge (HRD) fire extinguisher in the corresponding engine compartment. You can see a picture of this configuration here:
http://www.namsa.nato.int/gallery/systems/p3orion
I was in the navy flying with a P-3 crew in the mid 1980s. We were at Kadena Air Base on Okinawa, trying to take off and get to the same place in the ocean where some foreign naval unit was exercising its right to free navigation in international waters during the Cold War. Even though there are other P-3s on the ramp that day, *our* P-3 was special, since it had some sensors that that other kids didn't have yet..which is why we got to hang around the airplane during this maintenance delay...
During our engine starts, there was a problem with the number two engine (inboard on the port side). It was fixable in an our two, but the mechs would have to pull the plane into the hangar to do the work.
It's late spring, a mild sunny day, and I curl up by the port overwing exit in the tube; this part of the crew cabin it has enough space to stretch out and get a decent. The overwing hatch is open, cool breezes are flowing off the East China Sea. Others are lounging in their seats, on the bunks in the back, in the flight station, listening to the radio on the ADF receiver. We're just chillin', waiting for the mechanic on the ladder under the number 2 engine compartment to work his magic so we can go flying.
I can hear the sound of his tools banging around in the engine compartment, and just as I'm about to go asleep, I hear him call to the flight station (whose side window was also open): "Hey, somebody pull the number 2 e-handle"...
The e-handle does a number of things, including severing some mechanical connections between the propeller and engine turbine, cutting fuel flow, and generally making sure that the the motor you shut down during an inflight emergency won't be further damaged.
That's the 'splaining. Here's what happened next....
The guy in the flight station who responded to this request was neither an aviation mechanic nor an aviation electrician, nor an aviation hydraulics technician. Regrettably, he was an aviation electronics technician, and this particular one was not, shall we say, the sharpeset tool in our shed that day.
Here's what he didn't know:
He didn't know that the red button under the e-handle was *not* the push-to-release-button for the e-handle. So, before he pulled the e-handle, he pressed the red button underneath it, believing it *was* the push-to-release button.
When he pushed the button, the contents of the HRD fire extinguisher emptied --immediately-- into the number 2 engine compartment...where our helpless mechanic was still working.
The good new was that nobody got hurt (including the poor bastard who pushed the button, who was spared physical harm by the mechanic). The mechanic was none too pleased, because now cleaning the engine compartment just got added to his list of things to do...we didn't get to go fly that day because it takes many more hours to clean up the engine compartment after the fire extinguisher is emptied out in there.
Big red buttons ang big yellow handles...equal sources of entertainment.
The building engineers where I work tied our computer room air conditioners to the building's environmental control system: they can remotely monitor their operation and shut them off in a failure scenario. The monitoring system was a Windows PC.
So, the controls folks came in and did a software upgrade. It was a nice day out so they figured they could shut off all the A/C systems for a couple hours and it wouldn't be a problem.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
Never mind kids, don't let the #2 PHB in the same room as the server. Despite there being up to one hundred power sockets in the room, he will pick the one providing power to the server to plug his laptop in. E.G. There are ninety free power sockets in the room, there are ten with somethting plugged in, which one shall I choose, DOH ...
davecb5620@gmail.com
Well, Poland actually. A few years ago I was in Poland with my wife on way to visit her family in Belarus for the first time. Her mother and cousin had come to meet us in Warswaw and we were all taking the train to Belarus... one of the old fashioned trains with individual compartments for 6 or so people.
:-)
The light in the compartment was rather bright, so I decide to shut it off, but no visible switch had any effect, so I try the only one left, the pull-down chain by the ceiling (I guess I was tired), and immediately the train engines cut and it starts coasting to a halt. Being tired but not stupid I immediately sit down and try to look innocent (and not snicker too much), which is helped by the fact that I'm travelling with three females vs a bunch of hooligans. This being eastern Europe, we are very soon visited by three armed military guys who are going compartment to compartment looking for the culprit, but luckily we looked inoccent enough and they moved on.
It was pretty cool the way the train engines cut immediately when I pulled the emergency cord! No doubt made a great first impression on my wife's mother too.
When i was about 8 years old, during the mid 80's, my dad, a commander in the US navy and head of air operations at a naval air station, took me to work.
It was an awesome day, i got to see the pitch black control room with blinking lights and green radars, got a tour of various planes and helicopters at the base, then i got to hang out in my dad's office in the control tower.
There was this cool looking phone, not just any phone, but a bright red phone with no numbers on it. I had never seen a phone without a dialpad before. So, I pick up the phone. It rings maybe once and then someone answers it franticly, and I immediately hang up. Well about that time I notice fire trucks are rolling near the airfield and the phone starts ringing again and my dad rushes in the office with a startled look on his face. He answers the phone and apologizes to whomever was on the other line. My dad wasn't that upset, but let me know that I'm never to do that again.
time is a perception of a being's consciousness
time is your 6th sense, the wierd ones are 7+
Stimpy couldn't resist, so why should we expect a normal human being to?
This is the NFL, which stands for "Not For Long" if you keep making those bulls*** calls.
Don't plug a vacuum cleaner into the UPS, it tends to make the computer not work ..
davecb5620@gmail.com
You can kill people that way. Woops, my bad.
Banks are imposing places to do work. They take security seriously. Not only are there the usual locks and security doors, the data centre itself is a couple of stories below ground level (to protect against things like terrorist attack). But this guy had some work to do for the bank, and found himself in their subterranean data centre.
This is a place where you do not want to f*ck up - millions worth of transactions being carried by the cables every day, and other people's money residing on hard disk platters. But this was a modern data centre, with backups, the obligatory kill switch, raised floor, and the air conditioning and disk drives giving a constant eeeeeeeeeeeeee sound.
So there he was at the computer, when he had to phone the office to check up on something. As he was on the phone, he leaned back against a rack, and heard the two sounds you do not ever want to hear in such a place -
*click*. EEeeeeeewwwwwwwwwwmmmmmmmmhh.
The phone call ended with "Oh sh*t, I think I've done something". And our hero soon found himself mobbed by technicians who burst into the room when their screens went the proverbial blank.
Thankfully the bank suffered no significant loss as a result of this incident.
I friend of mine got roped into plugging in a UPS to a server for a local radio station (to protect it's identity, we'll call it Retired Gardeners FM). The mains cables were already connected, it was just the serial cable. He plugged it in when the server was switched on ("It'll be alright....").
He consequently displeased lots of little old ladies by accidentally taking the station off-air as the server powered off the second the serial cable was plugged in.
He later had a regular on-air slot guesting as the IT Expert.
When I was 3 or 4 years old I punched my Dad in the nuts to see what would happen.
2 seconds interesting
Half hour unpleasant and educational.
It's a lesson I've held on to since.
What do you mean "not truly free". She's open source, and created by relatively unskilled labor, right?
Best Slashdot Co
When I asked one of the elders, he told me the story:
The CL It is mainly a scanning electron microscope where the electron beam is used to excite light from a semiconductor.
If you know a bit about SEMs, you know that they require a vacuum to operate. And if you know about luminescence, you know that you can increase the amount of light coming from the sample if you cool it down, preferably to 4K using liquid Helium.
This of course means that you need an even better vacuum, for insulation and because otherwise everything (!) in there would freeze on the sample and cover it instantly with a layer of frozen air, water and whatnot.
This vacuum was ensured by a huge conglomerate of pumps, cryotraps, shutters, airlocks etc. and carefully monitored over months and years.
Until one day said professor entered the room, saying "Hey, what's going?" and casually leaning his back on --- the BIG RED BUTTON!
Disaster ensued:
- darkness fell upon the room (only the exit sign stayed on),
- hydraulic locks slammed shut,
- air gargled backwards through the oily backing pumps,
- turbomolecular pumps loudly whined from 10k rpm to 0,
- emergency valves tried to vent parts of the machine with specially cleaned nitrogene,
- the helium boiled and the exhaust meter rotated like mad,
- and --- because the insulating vacuum was gone and air humidity condensed on the hull --- the beloved CL quickly turned into a huge ice block, sitting in the center of the room.
This lead to violent cursing, a month of trying to re-establish acceptable conditions in and around the CL and a life-long exile for Prof. Smart Guy. He never dared enter the room. Even when he showed the room to visitors (Nobel prize bearers, kings and prime ministers), he carefully avoided crossing the threshold.The BIG RED BUTTON, of course, was secured with a cuff, crafted by our technician to be nearly invisible (to pass the yearly security audit).
Say out loud: I'm an Aspie and I'm somewhat proud, I guess. Uh. Can I write an email in all caps instead? Hm...
There was a big red button on the inside of the server room opposite the card reader on the outside. I got curious if pushing the button would make the card reader light up. It didn't. The suddenly silent fans gave away the button's real purpose. And then we discovered that there was a lot more interconnectedness than anyone throught. Shutting down our server room crashed most of the servers on the three building campus. Lucky for me that this was an R&D type system with some inherent flakiness. Folks were used to losing access for an afternoon now and again. I got quite a razzing from my manager, but nothing worse.
There's a big outsourcer located right outside of Dallas. Anyway, they had just installed this fancy GUI to manage their LPARs. These LPARs, BTW, were the running ridiculously important financial things (that's a technical term).
So the senior tech needed to take down an LPAR for maintenance. He goes to the GUI, clicks the shutdown button, and clicks a bunch of buttons that say 'OK'.
Of course, each one of those 'OK' buttons is asking him whether to shut down an LPAR...which means that he just shuts down all the LPARs on their 360.
Needless to say, he was scheduled for more training ASAP.
Chemical burn emergency shower?
In the 1980's I worked for a bank. We had a 24x7 realtime credit card authorization system that connected to card companies over IBM Series-1 boxes. A Series-1 hiccup cost big money, yet company policy required high level manager approval to even reboot a frozen machine. On call on my own, I waited hours for approval. Then an old hand taught me to tell the ops staff to start the approval request process, then when their backs were turned to strike like a cobra at the blue button that reset the machine. Oh look, it fixed itself! Goodnight everyone!
My company built a new building with a custom data center in it. We installed all the servers, routers, switches, etc and moved everyone in. On day my boss was showing the data center to a newbie and he noticed a big red button by the door, so he pressed it thinking that it would open the door. Apparently someone forgot to label it "Emergency Power Cut-off" and the whole room died. It took us 5 minutes to locate the switch to turn it back on, as no-one had shown it to us, and then another 10 to get all the servers back up. No one pressed that button again...
there's only one thing any self respecting geek can do.
Hang a note on it that says "Pull me."
Looking for Book Reviews? Check out Literary Escapism.
Does deleting the login exes of a college computer system count as a Big Red Buton Disaster?
What if I said I did it twice?
To do something right, you often have to roll up your sleeves and get busy.
I'm not the guilty party in this tale. Many years ago, I worked at an IBM 370 mainframe shop. Somehow, a young fellow leaned over in such a way that his pants pocket caught on the emergency pull switch. Everything went down, and an IBM service rep had to come out to restart the machine.
http://twitter.com/1389 http://pownce.com/1389 http://www.technorati.com/people/technorati/1389 http://www.gleamd.com
So, some new equipment was installed in our server farm (I think it was air handlers). As part of the contract, we were to receive a training session, and it would be taped so it could be used in the future. Now, a quick note about the power to the server farm is in order. There are two emergency cutoff switches to the room. The external one, if triggered, will shunt power from the emergency backup generators to the farm. The switch inside the farm, however, is a "Dead Man's Switch". It kills all power to the servers. Period. We return to the story in progress... I was not present for the training. After all, it wasn't for me. If there's a problem with the air handlers, I call the necessary maintenance staff. So I can only describe what happened as it was told to me. But what apparently happened was that the in-duh-vidual giving the presentation reached over to point out the dead man's switch, and learned the hard way that his depth perception was not all it should be. Next thing we know, the server status monitor is screaming bright blue murder at us. I was coming back from running a ticket to see the monitor glowing red, the lead network engineer standing on his desk (so he can see over the cube farm walls) with a phone in one hand and a stress ball in the other, and the hell desk phones ringing madly off the hook. 2 hours later, the trainers had beaten a hasty retreat (lest they be tarred & feathered), and we were still working on getting everything back up. I had multiple tickets waiting for me to work on them, and finally got word that I'd be able to work on them. To this day, our opinion is that the trainers needed to get their arses back here with a company credit card so we can go set up a very large bar tab somewhere... Oh, did I mention that this was taped? Did I also mention that the downtime was being tracked? I think that there was a trainer that was looking for a new job shortly thereafter.
"My brand of comfort isn't so much 'There-there' as it is 'There's a boot, pardon me while I connect it with your ass!'"
My friend was busily finishing some work off on a University PC, I was bored.. I was a red switch on the back of the PSU that said 230V... I wonder what the other setting says?? [FLICK] oh 110v...!!!!!1110one [BANG!].. My friend was not ammused
In the not too distant future, next Sunday A.D.
Quite a few years back I had some work to do in the data basement of a very large bank. An IT contractor had just finished and was about to leave through the maglock controlled doors.
:-)
Like many others have reported the Big Red Button was NOT a door release, and I will never forget the guys' face after the mass spindown started and he realised just what he'd pressed. He got exonerated because it had been on the risk reviews for 2 years running, I just don't think anyone expected that someone would prove the point - could have been a bit heavy on the liability cover..
It's quite awesome to be in a room full of noisy gear and hear it collectively spin down (the airco was ducted in). The post event silence is impressive.
And I've not seen anyone quite as white since
Insert
Does it make sense to anyone that the default behavior of a database is "destroy with no undo?" Let alone, the default behavior of MOST computer applications?
If it is a person's job to serve the computer, then yes, this behavior is fine - we need all the performance we can get, right?
Yet if it is the computer's job to serve people, then perhaps it is the computer that should change...
Spoon not. Fork, or fork not. There is no spoon.
This button is just hooked up to a counter, to see how many times it gets pressed.
dominionrd.blogspot.com - Restaurants on
I worked in a team which used a local server to store most of their working files. Generally 5 or 6 people in the team were using the disk on the machine at any one time (sometimes many more).
It sat on a table next to my desk.
While waiting for a system build process to complete my fingers, still wanting to be busy tended to fiddle (seemingly of their own accord) with things on my desk, perhaps cleaning the dust from my machine if the desk was too tidy.
One particular lunch time my desk happened to be tidy and my own machine free of dust (lots of building going on that day) so my hands started clearing dust from the front of the server (I really was just an innocent observer). While digging a particularly large dust bunny out of a crevice in the front of the machine my mind suddenly caught up with what my fingers were doing when it noted that I had just pressed the power switch to the server in.
There was a moment of blind panic during which I almost let go of the button but realised that the server still had power while I held the button in. The panic soon turned to acute embarrassment when I started to inform the people sitting next to me that they needed to close down any work they had open on the server as I had to perform a quick shutdown. They didn't buy it of course and I had to suffer hoots of laughter while they watched me try to shutdown the server one handed (b***ards refused to help me).
I don't clean my desk to this day.
I have no idea if this actually happened, but I heard about it from someone who worked in the datacenter at the time. The big red button was set on a wall and the new manager, inadvertently leaned on it without looking. Completely accidental. So, his solution was to purchase a plastic cover for the big red button, to prevent it from happening again, which when he attempts to install it, shuts down the data center again. Seems it was wrapped in cellophane, which he hadn't removed before placing it over the button.
as I quite like the suspend-to-ram from the power button on my keyboard with Win2k3 server.
NB - instead of removing keys you could try Power Options -> Advanced and use the option "When I press the power button on my computer > Ask me what to do".
Not certain if this worked under 2k but was fine for XP. AFAIK very few Linux distributions tend to support these additional buttons.
F_T
Well, I had just been promoted to sysadmin and was sitting at the console terminal of a VAX running Ultrix that many of our students used for their C programming coursework (remember the time they taught decent languages in IT? ;-), maybe 20-30 people were working on the machine during that time of day.
I needed to switch to another machine for checking something, and being totally oblivious to the fact that I sat at the server console, hit the break key, only to find out this would drop you to the boot prompt of the VAXen's firmware. Needless to say, my boss and I instantly smelled the burning of torches and heard the trampling of angry feet from the hall next door where the classroom was, and it took less than 15 seconds before our admin room door was being pulled open.
My boss had the wits to shout "damn students, they've brought down math5 again!!!!" which probably saved us from an ugly, untimely death as the mob retreated instantly to escape any blaming from an angry sysadmin, but boy, that moment had me scared quite a bit!
Ever wondered whats wrong with the world? http://www.ishmael.org/
Heh, funny. I've got a similar story in Austria. We spent a summer travelling around Europe on an interrail pass back in the mid 90s.
We'd hit Austria rather late in the evening and were taking a tram to the youth hostel. We saw the stop coming up and told our Australian friend to push the 'stop' button to notify the driver to stop. He pushes it, notices the tram doesn't stop immediately, thinks something is wrong and pulls the handle just above the stop button. Naturally that handle is the emergency break, the tram comes to a screeching halt and the driver is staring at us, furious.
We're thus stuck in a tram full of people that look like they're about to murder us on the spot for making them late for dinner or whatever. This is a somewhat uncomfortable situation, and we start walking up the tram to apologize and grovel. The grovelling plan changes as soon as the driver gets on the radio and asks for the police. We take one look at each other and quickly decide to shove open the door and leg it. I guess we didn't look innocent enough, so it's a good thing we were in better shape than the driver.
I once saw someone attempt, unsuccessfully, to move a production AS400 box that was responsible for running our entire operation...while it was turned on. Apparently, an engineer from an unnamed blue company has left a screw laying loose inside of the box. When moved, the screw somehow got into the power supply piercing the lining and letting all the magic smoke out. It wouldn't have been so bad, however, the battery on the RAID controller was broken and was scheduled to be replaced in the next week.... Well, apparently those old blue boxen don't like it when you turn the power off. All the DASDI got corrupted and we needed to restore from tape.... At the time, it took approx 9 hours. Well, the 7th tape in the 9 tape series didn't work. Good thing we ordered the last 2 backup sets from our off-site data storage vendor. Moral of the story: Don't put bigass servers on wheels "or" never let the IT director into your datacenter.
-\|/-\|/- If its not 1200 baud, its crap....
My company had just recently bought a smaller ISP in another town. It took us a while to integrate everything since we were also integrating another smaller ISP we'd just bought. In the meantime, we made a few changes at a time, but we had some server machines unmanned in their old NOC some 40 miles or so away from ours.
I added a network card to a mail server and added another box to the same rack one day at the remote office. The plan was to make the software changes from back at my desk when I got some time.
Well, I got some time around two weeks later. Everything was going smoothly as planned at first. Unfortunately, I'm not 100% accurate at typing like some slashdotters claim to be. I mistyped the interface name in one of the virtual adapters I was creating for the new card. In fact, it exactly matched the name of an interface on the primary network adapter.
Yes, I checked to make sure everything looked alright, but for some reason I didn't catch my typo. I guess my eyes glossed over from working such long hours back in the day when small ISPs were still the rage. I saved the network config files, and restarted the network. All I said to my boss on my way out the door to my car was, "Made a typo. Going to Jacksonville."
Once I drove all the way there, fixed the typo, got the network restarted (successfully this time since there weren't conflicting interface names), and drove all the way back I explained what I had done. Thank goodness we had MX and outgoing SMTP separate from the mailbox storage and POP toasters. People were unable to check their mail for about 45 minutes, but at least none got dropped.
Blue _ink_ ? Silver nitrate is the only way to go for this purpose.
I have a lifetime of BRB stories, but my first encounter with noisy machines and a rabid Drill Instructor left me more cautious than most.
/.ers were born.
When I was 12 or 13 years old I had already earned a reputation for being able to take things apart and on occasion put them back together again. I was into amateur radio, a relative had introduced me to computers using a teletype, and other geeky things from before most
I was with a group, scouts maybe, in a weather station on top of a mountain. One large room, with two 19 inch racks of equipment in one corner, some teletypes next to that, and the rest of the room had much meteorologic kit scattered around the edges. I was told not to touch anything, stay at the very back of the crowd, not ask any questions, and above all, not to touch anything. So there I was, with a few people between me and the racks of reel to reel tape and punch tape and teletypes, when the automated data collection process started it all up. Massive amounts of noise, as the teletypes spit out the 15 minute reports, punch tape boxes started up, and all the tape recorders did their thing.
All eyes turned on me, and the leader of our group started yelling at me. The leader had been a Drill Instructor at some point, but was probably discharged because he was too psychotic. Role model for the DI in Full Metal Jacket. He ripped me up one side and down the other for touching something, breaking the system, whatever. He yelled for so long that the next 15 minute automated collection process started, which shocked him so much he stopped to draw a breath. With that slight pause, the rangers who ran the weather station managed to tell him that I hadn't touched anything, it was an automated process that ran every 15 minutes, and they had been between me and the machines.
Several people later congratulated me on standing my ground against this asshole for so long.
For my entire career I've always been cautious about touching equipment I know nothing about. Not that it has saved me from recognizing many of the stories in this thread, and I'm certain some of my former colleagues are behind some of these stories.
Obligatory story for the thread, not mine, but from watching my colleague sitting at the desk across from me.
A phone company in Africa decides to buy a matching pair of voice compressors to put on a new satellite link they want to light up. The machines finally get delivered to our offices in Europe, where the main telecoms engineer is going to configure them before sending one down to Africa. For an experienced commissioning engineer it takes a full two days to get everything right on these boxes even when they are sitting next to each other. If there were to be 6,000 miles between them, commissioning and tuning is almost impossible. Since the phone company owns these boxes, we pack up everything with the one being shipped to them, manuals, diskettes full of configuring software, all of it. Some weeks later we hear that they have finally received the shipment, but since it is a Thursday afternoon they will be installing it for first tests on Monday. The upper management has been told repeatedly that the system is delivered pre-configured and nothing is to be touched. But what engineer ever listens to their bosses?
Monday comes, and we find that over the weekend the engineers in Africa were so impatient to get started they loaded up the configuration software on a PC and started reading at page one of the commissioning manual, which covered resetting the stored configuration. These systems were so badly engineered there was no concept of backup for the configuration, except to print out the configs screen by screen from the PC. Which my colleague had done, because he knew better. We had to courier a copy to them, but when I left a few months later they still hadn't managed to get much of it working. They were still paying for a mostly unused satellite channel.
the AC
Hemos is like...sci-fi fans;he thinks technology is cool, but he hasn't bothered to understand the science it's based on
My current job involves, amoung other things, safety wiring and control systems.
One thing I found interesting is that the emergency stop button for safety
systems always has electrical current going through it. In the case of a saws
and robots, that current might hold a relay closed, which in turn delivers power
to the saw or robot.
The reason they wire it that way is so that if the wire ever breaks or becomes
disconnected from the emergency stop button, the machine stops.
For those systems, stopping the machine when you don't mean to is preferable
to not stopping the machine the one time in ten years that you really need to.
I had worked with computers for years and would never have though of doing it that way.
The most common "big red button" I see turns off the power to subway third rail
power. Now if they could do something about workers getting hit by trains.
"We can't solve problems by using the same kind of thinking we used when we created them." -- Albert Einstein
Used to work on an IBM mainframe. I cannot remember the name fo the program but it basically printed reports from a command line.
Type in filter criteria followed by the name of the reoprt. The report generator would use the filter information and create the report. The report was printed on a mainframe printer (big laser printer, about 10 feet long, printed paper at about 1 box every few minutes).
One day my co-worker forgot the flter criteria.
He waited about 1/2 hour, did not get his report from the printer techs, so he did it again.
We got busy doing other stuff.
About 2 hours later, the printer tech wheeled over a PALLET of boxed paper containing his report. Hundreds of thousands of records had been printed, not the few he thought he was going to get.
I still wonder why some programmer had put in a warning that a report was being run without a filter.
- - - - - - - - - - -
I am a programmer. I am paid to produce syntax not grammar. Deal with it.
That story is several months old so I do believe it is time to pull this gem out of my magic hat...
s cellaneous/timeline.gif
http://i7.photobucket.com/albums/y262/Direpath/Mi
"It's amazing what velocity can do when human beings are in season" -Matthew Good
Suppose that, in some implementations of rm -r, the code checked to make sure the directory recursion function didn't walk back up to the directory it was called from. Seems good, right? But in this case, / is not the directory the "recurse into /home" part was called from - it was called from "/home/foo". So it could walk up a directory tree to the root directory, and then back down to the rest of the system. I don't know if this explanation is true, but it sounds plausible to me.
For those wondering about what we're talking about: The shell expands
to something like or whatever. Notice the "-r" in there. What that does is tell rm that, if it encounters a directory, to recursively chdir to that directory and process the contents of said directory. Notice the ".." in there. That's the parent directory. So go into the parent directory. Which also contains a ".." entry. Lather, rinse, repeat.dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Two from my military days, early 80's...
... right about the time a whole mess of guys with rifles came roaring up. Seems I'd hit the silent alarm. OOPS!
Ran into a buddy who was pulling gate duty. It's late afternoon, there's no light on in the shack. I reach over & flip the switch, but the lights don't come on. I mention this to my buddy
I was stationed where the Air Force makes up all their software. We were testing what was to be the next generation of equipment the AF would purchase. One side Burroughs, the other Sperry (side note: One side won, the other didn't like losing, within a couple of years they'd morphed into UniSys. While that was going on, the Burroughs repair techs one day said they were going to have a company-wide naming contest for the new entity. I mashed the letters together & came up with Yurr's Burger Shop. The tech's loved the name, but apparently Corporate had a different idea...)
I'm on the Sperry side, a fancy new system that can actually boot off an 8" floppy instead of tape or paper like most of the stuff on our production floor. Way advanced. It's the graveyard shift, when a lot of testing gets done. One night we had to boot the system over a dozen times, just trying out different things.
Never did figure out *why* it was programmed this way, but there was this part of the boot process wherein if you happened to down the disk drives, the entire system would be hosed. You could not simply reboot, you'd have to do an entire system restore, something like 60 tapes. I'm running the console, all these people standing around me, it's 3AM, we're booting for the seventeenth time. To this day, I swear someone told me to do it. I shouldn't have, but I was busy paying attention to my superiors as they issued orders, my fingers went to work before my brain did.
I downed the disk drives.
Now I'm feeling very small. We're debating whether or not to wake the Base Commander. Loading 60 tapes times 20 minutes per, we're talking about being back online sometime in the wee hours. Tomorrow. No way we're going to make 7AM. I am so screwed.
Then, the last cup of coffee must have hit, because I proceeded to save my own ass. Wait a minute, I say, we have 10 tape drives, they can load concurrently. Why don't we just spin them up all at once?
They let me stay at the console to orchestrate, so I had guys way above my ranking furiously mounting & demounting volumes as we loaded the system back in as fast as possible. Around 6:45 it was done, and we got to mention the disaster to the day crew in the past tense, which is always the best way to mention disasters.
This also became the new way to do a system restore.
We were opening the new building for our plant expansion at the factory company. Construction wasn't finished yet, but it was mostly done. They were just doing things like painting and wiring and fixtures and equipment and stuff like that. So we had this big ceremony in the new loading dock, sort of an open house. Whole company is there, along with some suits from our biggest customer. We've got a PA and a projector going as they run a video of fighter planes (which used some of our product). The Big Boss thinks, hey, it would look a lot better without the overhead lights being on and washing out the projected image. So he sends one of the maintenance guys over to the breaker panel to shut off the lights.
Everything goes dead. No PA, no projector. Oops.
Since it was a construction site, and they hadn't finished the permanent wiring yet, everything - including the one outlet that had power - was running off one breaker. The best part was waiting for the digital projector to cool itself off enough that it would turn the bulb back on after power was restored.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
That's because computers and heavy machinery differ in one important point: If you cut the power to a computer, it's safe. Yeah, maybe it'll lose data, take days to get back up again, or even break the hardware, but it'll stop sparking and/or electrocuting the guy who stuck the screwdriver in the wrong place.
If you just cut the power to heavy machinery, bad things happen. People can get maimed, killed, or it can become impossible to recover severed body parts from the inside of the machine. The thing needs power long enough to enter a safe state where power can be cut safely without causing bad things to happen.
Heh. The man page for killall(1) on Linux has the following remark in the "KNOWN BUGS" section. Quite possiblly the biggest understatement of all time:
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Systems like that are found in nearly every multistory building with a fire sprinkler system.
The pump that pressurizes the sprinkler piping is installed contrary to most standard provisions of the electrical code, in that it has very little overcurrent protection, no overtemperature protection, and power for it is tapped off ahead of the main building disconnect.
The reasoning here is that you want that pump motor to run itself to complete failure if need be in event of a fire. No shutting off because the motor starts to overheat, or because the FD pulls the main power switch.
Remember "News for Nerds, Stuff that Matters"? Help make it a reality again! http://soylentnews.org
I used to work for a large telecommunications company. They had guards posted at the entrances for the data center, and elaborate procedures in place to handle vendor, employee and data center worker access. Mostly it boiled down to the guard having to disarm the door to let you in after you properly identified yourself. This worked great, unless you got the fateful combination of a door alarm, and a rookie guard who didn't know how to turn it off. On at least two occasions, the big red button was selected as the solution to the door alarm. It was later determined that while the guards were required to speak English, they were not required to read it... After the second event, they got a new set of guards.
Well, at least one office.
This was nearly 3 decades ago. I was a new clerk and had never even had a computer. I had done a good job with my paper-shuffling, so much so that my boss lent me out to another function (big perk, goes on your record as a wider set of experience) where I was to work in the "NEC room." This was a tiny room with a big NEC computer hooked up to an even bigger line printer inside an acoustic shell that held pin-feed, 8-part forms. These forms were Revenue Agent Reports, the final results of all audits, the paperwork you sign to agree to a change in your taxes.
I *really* impressed those folks. Seriously. First off, they couldn't get the forms aligned in the printer; they were always printing everything a fraction of an inch too high or too low on the form. It seemed a simple job to me. I noted the position of the pin-feed holes relative to some random part of the printer, printed one report, changed the position of the forms relative to that printer part to a degree roughly designed to compensate for the error on the first form, then printed another sample form. It was, as expected, properly aligned. I had aligned a new box of report forms with just one test print. The long-time workers in that little room thought I was a god. Literally, mouths dropped opened. They were accustomed to spending hours and half a box of forms getting set up. They loved me.
Next, they had a bunch of garbage records in the database that kept printing out. It was pretty simple to figure out that if I deleted each record, they wouldn't start each day with 20 garbage prints. By this time, they loved me so much the manager stopped by to meet me, sent an official memo of praise to my boss (something normally never done until a detail is complete) and started making noises about creating a position for me in his group. I was flyin' high.
A couple of days later, I asked the question I'd been curious about since I got there but there was never anyone around who could answer. "What's the button for?" "What button?" "This big red one next to the door" I said as I pointed at it. I SWEAR that I didn't intend to touch it; the tip of my index finger just barely kissed the dome of the button.
KLUNK!
Every light in THE ENTIRE BUILDING went out. This was the emergency shut off for EVERYTHING, pre-dating the installation of the computer equipment and intended to be tripped only in case of fire. It took building maintenance about 6 hours to go floor by floor and get every circuit up and running again.
My temporary boss called my permanent boss who called me at home that night and informed me that not only was I no longer on the detail, I was not to set foot in that building until further notice. There were apparently about a hundred Revenue Agents who lost their cases (Remember, this was back in the days of dual-floppy computers without hard drives and saving your work meant deliberately pulling out a disk and inserting another) that morning and had to rebuild their files. Each and every one had apparently vowed to strangle me on sight.
This one is actually pretty recent. I am deeply embarrassed.
I was working on a shell script designed to stop Sybase replication on an AIX server. The AIX server in question was at a client site, and I was working remotely across a VPN. Fortunately, it wasn't their production machine. Unfortunately, I forgot two cardinal rules of shell scripting:
1. Unlike most "grown-up" programming languages, you can't include a function definition after the function call: the definition has to come first.
2. On AIX, the shutdown command doesn't require any arguments.
Synthesize these two facts and you get an important corollary:
3. It is a very, very bad idea to name a shell script function "shutdown".
Once I realized what was going on, I spent most of the countdown trying to recall how to abort a shutdown in AIX. (Answer: you can't). With about ten seconds to go, I realized that I didn't have any other options, so I called shutdown again with the "fast" and "reboot" options. So at least I was spared having to call someone at the client site and asking them to start the machine up again.
How about a German electrician at a US military facility trying to find which power box he needs to shutdown to work on a specific circuit killing the 400 cycle power to an IBM mainframe, and then turning it back on before everything everything finished shutting down. Several days and and emergency dispatch of several IBM engineers later, the critical system finally came back online.
In our computer room the big red button is at about 50" from the floor. It's under a plexiglass cover that has to be lifted, and then the button pushed.
That big red button cuts power to the entire computer room. Everything from phones to networking gear, and servers. The only thing that stays on is the overhead lights and the air conditioners, because they're wired to different circuits.
The big red button is the cutoff for the APC Symmetra that everything hangs off of. In case of sprinkler activation we're supposed to hit that button. I argued that all they really needed to do is put a flow sensor on the sprinkler lines to the computer room. That got shot down when they explained that sometimes there might be flow when the sprinklers don't activate. Doh!
And we don't allow kids in the computer room.
I work for a small precision machine shop.
This year we spent $750,000 to purchase our first horizontal CNC mill. The manufacturers rep spent 4 days setting it up and making sure it was operational. Our lead machinist set up the first production run and made sure it was working properly. Only 4 days and we were making money! It was wonderful.
Night shift comes on. Operator gets basic instruction on what the differences were between the new machine and the vertical mills he was familiar with. Everything seemed okay. Machine was making nice parts, everyone went home with smiles on their faces.
Two hours later, phone rings. New mill is "broken". After an hour of just loading and unloading parts, the operator happened to get curious and had looked in the window to watch the machine run. First time it changed tools, it was moving so fast, things were rotating, he got afraid and hit the "BIG RED BUTTON". Shut it down in the middle of a tool change cycle. After an hour, the night guys realized they couldn't make it run again, called for day shift. Day shift couldn't make it work again, called the manufacturer. Phone support tried to walk them through it for 8 hours, still couldn't get it to run.
Next day, technician arrives from the factory. He can't make it work. Spends two more days, finds a sensor that was connected at the factory. Tells us, the only reason that big red button is there is because the government makes us include it. Please use the normal stop button in the future.
Three months later, machine is still running well. Making lots of nice parts. Night shift avoids watching it run. They are content to just load and unload parts.
I was working IT at an investment firm back in college. End user support, software install, and managing a single Novell 3.11 server.
The owner was an incredible know-it-all jerk.
He wanted more space on his server, and had me add a second hard drive to it. I had to span disks. I warned him that this would make the server twice as likely to have a catastrophic failure, since now the main volume had two hardware points of failure. I recommended he buy a tape backup.
"No, I don't really see how that would make us money." His standard answer to any good idea I'd come up with. I warned him in the strongest possible terms that Bad Things Can Happen.
Then, the inevitable Big Red Button event happened. They got a postcard in the mail saying the power company would be cutting the lines to their block and replacing them in a week. His crack office staff promptly ignored the card. See where this is going?
D-Day for the lines comes, and *snap*. Down goes the whole building. And of course the server won't boot afterwards. A spike had destroyed half their spanned disk.
Still my fault. My fault because I didn't somehow know the lines were being cut. My fault for not warning him strongly enough. My fault because I couldn't resurrect the server.
He even had a tape backup on his personal PC which I had hijacked to run a backup of the server every midnight, as a last-ditch attempt in the event of a failure. And of course - he found that and disabled it, too. And wrote over my backup tapes.
STILL my fault.
I'm so glad I don't work there anymore.
Weaselmancer
rediculous.
I was at my dad's take-your-kids to work day, and the CIO kept doing these god-awful presentations.. but my dad is an 1337 super hax0r like me, and we both knew that there was an IBM P695 with a big red button on it that uber-switches the server off in milliseconds. The high-availability systems were currently undergoing maintenance, so I knew it'd get me out of this crap presentation if I were to press it....... bam! it took them 72 hours to recover all of the virtual systems from backup. Unfortunately, the CIO didn't really learn his lesson - he still does these damn presentations one day a year.
Vehicle Stars used car search is my current project
I work for the MIS department for a hospital. One day in the data center the generator was showing an error. The alarms were blazing. After about 30 minutes of working the facility ops "Now dubbed facility oops" noticed the big red button. Evidently he though it would shut off he alarms. Well it did shut off the alarms, killed power to the whole computer room, and trigger the halon gas. As the room filled with halon all the people with good sense ran out of the building. The facility oops guy just stood there with his mouth slacked jawed. It took 24 hours for a full recovery.
After the halon alarm was sounding for 20 or so seconds he decided that it was probably going to go off soon, and hit the big red button on his way out the door. Down goes our Internet connection just as I was getting to the good stuff!
Keep passing the open windows...
Where I used to work was a big red emergency cutoff button on the wall of the machine room that went to the power distribution units. While moving a machine and cleaning behind it, someone's elbow went right into it. Fortunately, it just shut down the power to the machine room and didn't kick on Halon or anything... but it disrupted about 150 people as the fileservers and many other number-crunching machines crashed.
So, we learned "hey! let's make it so you can't accidently press it" as we rebooted and fscked all the computers.
A couple days later, we built a little wooden box to go around it to help avoid accidental pushes, and painted it red.
Guess what happened as we were mounting the little wooden box around the big red button.... Yep, we hit it again.
Our entire company uses an X login server for login sessions from X terminals. One day I wanted to log off of my X terminal, which I do very rarely. I clicked the logoff button in my window manager, and it popped up a window asking if I wanted to "Shut down". Since I wanted to shut down my X terminal, I happily clicked yes, and happily re-entered my login password. Immediately after hitting enter, I realized that I had just told the login server to shut down, in the middle of the day, while many people were actively using it. Doh!
I work for a government office, that receives documents that must have a date&time stamp of receipt for legal reasons.
...
Back in the 90's the department set up a networked computer system all controlled by a server in a secured site in our capital city.
The particular brand of server had a hardware clock reset button, common for a 'nix box in those days.
A couple of years after all this started, one day I got a label printed with the time shown as 00:12.
I didn't think anything of it at first, until I realised the date underneath was something like 01/01/1980
I paniced, talked to my manager then phoned HQ to advise them what I'd seen
Needless to say it took 2 days of a few hundred general staff sitting on their hands waiting, before the whole mess could be sorted by the IT people, and management could figure out a work around for the hundreds of documents received statewide, in the 20 minutes or so it took to tell everyone "Stop taking documents!".
I heard later that someone, who should have known better, had wondered "What does that button do?" and tried it out.
Don't blame me, it's usually 2 in the morning when I post
I didn't realize shopping at Staples was so hazardous.
o n/index.html
http://www.staples.com/sbd/cre/marketing/easybutt
So, at a previous employer , our test lab had two identical systems, which were going through a formal test. There were representatives from two branches of the service, and some civilian observers as well, from DC, Virginia, LA, and Colorado Springs. There was a limited amount of time set aside for this test.
:) Like it says in that Covey book they made everyone read. I decided to work on the "backup" system's server, to see if it was exhibiting the same symptoms. I logged in, and took a look at the event logs, and noticed that one of the services seemed to be hung. Well, the workaround for that problem was to reboot the box. And you can guess what happened next.
Well, I wasn't really directly involved in the test - but I'm one of those "go to guys" (not the kind that uses goto statements in programming, the kind everyone goes to for answers when things are broken). Well, in an early-morning dry run, before the test, they saw some weird stuff going on, on the system they were using for the formal test. They sent me an email asking if I could help troubleshoot. Only, I wasn't in yet. When I came in, I read the email, and went into the lab to check things out. It was empty, except for the govt witnesses and our tester, clustered around one of the computer screens, in the corner. My POC was the chief engineer, not the test manager. Chief engineer wasn't in, so I decided to proceed on my own. You know; to be proactive.
Everyone's head popped up from the screen and looked over at me.
What I didn't know, is that they had decided to proceed with the formal test, by swapping the main server with the backup server (they did not move the "don't touch this box" sign). So I rebooted their box during the formal test, which halted the workflow, and caused them to have to re-start the procedure at the beginning (two hrs of work).
Needless to say - with the hung service, they would have hit a brick wall in about another hour anyway. But guess who had to take credit for rebooting the server?
So something like 8 people had to fly back home, with the test uncompleted. And they had to fly back again after 60 days (when they could all schedule the time to do it). Total cost of the fiasco was on the order of $150,000. In the post mortem review, official blame was laid at the foot of the test manager, who decided to proceed on the backup system, and who didn't send an email update, and who didn't move the "don't touch this box" sign. But if you ask anyone who to razz about it, they'll say it was my fault.
Now you know (one reason) why these big government projects take so much time and money, and deliver crappy results anyway. I love this job!
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
When I first started working at my company many years ago, I had a crappy bargain PC as my office computer. The horrible budget case had a power switch that protruded, also at knee level. Add to the mix a tiny, cramped room that had three people working in it.
It got knocked a few times, once by my boss' daughter while I was working on an important machine schematic. She was only 7, so she got spared the cursing that usually followed.
Shortly after that, our electronics nerd pulled-out the switch and replaced it with something less volatile.
Does it make you happy you're so strange?
My story is not directly about a janitor but a maintenance crew that didn't even have direct access to the data room. They didn't need it to bring the system down.
As I remember it the UPS for one of our main servers was aging and needed replacement. It was not 100% reliable. One day before it could be replaced the maintenance crew were performing scheduled testing/maintenance of our generator.
There must have been a split second delay with the UPS when building power was switched to or back from generator power and one of our servers went down hard. It just so happened that we were in the middle of processing payroll. Hey, it doesn't get any more mission critical than processing payroll, right?
From then on the maintenance department had to get an okay from I.T. before testing the generator. We pretty much let them as long as payroll wasn't processing or it was between payroll saves and it was a "good time".
"A government is a body of people, usually notably ungoverned." - Shepard Book Quoting Malcolm Reynolds
Friend of mine recently got a job at a bank. His training one day was the "vault hard count" (I'm sure that means something to banker type people). His boss took him into the vault, explained what they were about to do (count money), and began the training. Toward the end his boss looks around, scratches her head, and admits that there's about $2000.00 she can't find.
Being the helpful kind, my friend looks around, sees a stack of $20's and as he picks it up he says "Is this it?". The boss's face turned bright red, eyes wide, mouth open, and I'm picturing the super-slow-motion "NOOoOooooooo..." coming from her mouth.
Turns out that is the "bait bundle" of cash attached to wires or the pressure sensitive silent alarm or whatever they use to make it call the cops.
Ruined his morning and he missed his lunch while filling out incident reports with the police.
I was working on a Sparc 5, billions of shells, editors open ... alone in this little lab. Got uncomfortable after a while, and took off my shoes. Boy that felt better, flexing my toes under the table. Guess where the power strip was.
But I remember a story by a colleague of mine that goes like that:
Big company gets hired to set-up the traffic flow information and management system in a big french city (which must come as a Big Budget(tm) project as well...)
Come the day of starting the monster system. Everything has gone through QA and is perfect.
Mr Big responsibility (read: sign his name and takes glory) prepares for the grand moment and puts his finger on the "enter" key... and lets it here as the press takes photos.
Every technician worth his salt in the room starts sweating and silently praying for the guy to RELEASE THE DAMN KEY.
At the same moment, every display in the city nicely starts displaying "Kernel Panic !"
I was working for a state agency in the process of moving buildings. The HP mini and associated stuff in the typical raised-floor room, along with us machine servants, were the last to leave the old building. The new tenants couldn't wait to start banging around the rest of the building, creating new office spaces, etc. One day I was talking to my boss in the computer room, backing towards the door... no, she shouted before my shoulder could quite reach Big Red. Every morning for a week we came in to find the mini working fine but having been powered off for several hours during the wee hours. WTF? Turns out the construction people had been shutting off ALL power (which did not include the outlets in the computer room but did include the computer room's A/C). Soon after, the CPU would overheat and turn itself off until after the room had cooled back down. I should mention that we did have an environmental chart recorder but that it had just run out of paper and since we were about to move... Who had wired the A/C into the ordinary circuits? Dunno. Told them to stop turning off that circuit. We thought we'd be OK for a few more weeks. But we came in one morning to find everything off. We had a big line conditioner but no UPS. Cabling between the conditioner and the equipment was secure, but the conditioner was plugged into a non-twist-lock outlet on the wall. Some overzealous worker had banged on the opposite side of that wall hard enought to dislodge the clock, which, you guessed it, was positioned just above the outlet to which the conditioner was attached. Oops. Tune in next week to read about playing network Doom shortly after its release with an exec from another department, former CIA, who apparently didn't see enough action while there.
I've seen two of them: At one mainframe installation they installed a flashing light to signal paper out on a row of high speed printers. This light was right next to the console of a Fortune 100 primary IBM mainframe. A security guard walking through the computer room saw the flashing red light, panicked and pressed the big red IBM panic button which caused an immediate shutdown of everything in the computer room. I heard it took them almost 24 hours to bring the beast back up.
Another time I was in a computer room when workmen building an enormous UPS system out of 1000 car batteries shot a nail from a nail gun through the 440 Volt Data Center Power bus. I nearly crapped myself from the exploding sound of a thousand disk heads retracting at the same time. I left the next morning when the IBM guys were coming in the door to restart the systems.
I had to browbeat our people for several years before they recently relented and made the change to a dial-out number of 8 (vs 9). The irony is we owned the VOIP PBX and we helped write the software that ran it. There had to be a pile of folks who knew enough about it to change the dial plan. It was just inertia I figure (maybe there's something I don't know).
The problem was exacerbated by phones whose physical buttons didn't always press easily and sometimes when you did jab them hard enough, you got a double digit collection. So you'd hit 9, then 1... the 1 wouldn't register so you'd really jab it, then you had 9-1-1 because it collected the digit twice...
Use 8 to dial out. At least until they really do switch to using '9-1-2' as the real emergency number. (Nods to Homer)
-- Mal: "Well they tell you: never hit a man with a closed fist. But it is, on occasion, hilarious."
When I was in the Army our missile system ran on 416V, 3 phase, 400Hz power like most avionics systems do to save weight in the transformers; because of this our shop had a 160KW frequency convertor, to change the 50Hz European power into 400Hz and a constant whine. Inside the shop, was the production control area, who's enterence was enclosed in chain-link cage for security and inside its electrical conduits was an intermitant short.When somebody slammed the door shut too hard the circuit breaker killed the lights in the area. One day clowning arround with my friends I slammed the gate, killing the lights, after hearing some cursing from the dark bowels of producting control we noticed that all the lights were off and I said something like "Did I do that?" and then we heard the convertor winding down and I said something like "I didn't do that, did I". We went out side and didn't see any signs of power anywhere on post, and I said something like "We're not going to tell anybody we did this are we". The next day I found out that the Air Force dropped a 500 pounder on the main power line for the post, which must have happened within seconds of my kicking the gate, a coincidence that really caused my ass some serious pucker factor.
Apocalypse Cancelled, Sorry, No Ticket Refunds
The point of a "Big Red Button" is to shut down EVERYTHING right THEN. Not wonder around pushing several buttons.
If you don't need the function, then there should be no Big Red Button.
The Kruger Dunning explains most post on
Corporate networks down? Phone systems down? Pffft. Pikers and wankers.
I've 'dialed' numbers by tapping it out on the hook button. ...more often than not, I'd get somebody random.
--
"I have also mastered pomposity, even if I do say so myself." -Kryten
I think you're missing my point. Consider the following algorithm as an implementation of the -r switch:
That might seem like a reasonable implementation. It will recursively descend into subdirectories and remove their contents, taking care not to walk back up to the calling parent. But if you happen to feed it .. as an explicit argument in step 1, it will eat the whole filesystem. Is that a bug? You bet! Did it ever happen that way? I have no idea. Is it plausible? I think it might be.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
As a pilot, the one big red button I'm most familiar with is the autopilot quick disconnect. It's large, it's red and it sits right underneath your thumb on the yoke. Any aircraft with an autopilot has to have one - from a little Cessna on up to anything Boeing or Airbus make. Pilots are specifically trained to press the button if the autopilot is engaged and the airplane does something they don't expect. In many designs, it's actually an electrical interrupt - it removes current from an engage clutch or solenoid. This is, obviously, a safety feature. Use of this button is quite routine, though. Pilots often use it as the primary method to disconnect the autopilot because it is so convenient and easy to use.
Visitors: Pull lever to alert guard to unlock bathroom.
Fnord.
No one is going to read this but I had to share. I was a Co-op at a large TLA company mentioned here. EPO BRS had a fairly strong looking safety cover. One of the infrequent visitors to the machine room decided to tye his shoe standing up and lost his balance onto the BRS. Cover breaks, EPO triggers, switch is pressed into the wall, and then falls down inside the wall. After moving some equipment out of the way and a crow bar to rip open the wallboard the switch is turned back on.
He didn't visit as often anymore, and always kept away from it when he did.
The sound of the drives spinning down is pretty neat if you can focus on it instead of all the work you are about to have to do. . .