LiveJournal Blackout Analysis Online

Hakubi_Washu writes "LiveJournal has posted their official analysis of what happened last Friday. Apparently someone "accidentally" pushed the emergency power off (which should keep all power off, even UPS), reset it and ran off. They had problems to come back up fast, because of "9 machines with faulty motherboards with embedded NICs that don't do auto-negotiation properly", Machines not fully rebooting for analysis reasons and few others. "

Comments

Lesser OS...

[Anonymous Coward]

They should be using OpenBSD. It can run right through power failures

The less we've learned...

[geoffspear]

Don't let your clients near the Big Red Button without an escort. Preferably an armed one.

Re:The less we've learned...

[Chris Mattern]

"The beautiful shiny button! The jolly, candy-like button!"

Chris Mattern

Re:The less we've learned...

[geminidomino]

Evil overlord list item #9: I will not include a self-destruct mechanism unless absolutely necessary. If it is necessary, it will not be a large red button labelled "Danger: Do Not Push". The big red button marked "Do Not Push" will instead trigger a spray of bullets on anyone stupid enough to disregard it. Similarly, the ON/OFF switch will not clearly be labelled as such.

Oppsie

[darkstar949]

"I'll just set my coffee down here, and..."
...
"Oppsie, I hope that button wasn't anything important."

History Eraser Button

[bsd4me]

Ah, the famous History Eraser Button rears its ugly head. I think that everyone who has worked in a large datacenter or lab environment with one of these has a story to tell...

Re:History Eraser Button

[Local ID10T]

I was consulting for State Farm installing machines that were supposed to help with the Y2K problem.

Hey! I worked that project too... it was fun, but mindnumbing. They actualy sent me to New Orleans for an install on fat tuesday.

Mardi Gras on an expense account :)

Re:History Eraser Button

[Aaden42]

Nobody remembers!

Re:History Eraser Button

[cgenman]

If I ever catch anyone putting a cover over a critical piece of safety equipment, like an Emergency Power Cutoff switch, I'll put their head on a pole in front of the data centre as a warning to others.

You of all people should realize that putting someone's head on a pole in front of a data centre is dangerous. For one, it tends to become a disease vector, as for some mysterious reason everyone feels the need to touch it. Rats are usually attracted to the smell, and you know how rats wreak havock on ethernet cables, especially the rats of the dead. Furthermore, putting the dead on a spike on your front lawn tends to attract ghosts, which are no problem if you're running a secure OS but everyone knows what havok ghosts can wrack on a Windows Server 2000 installation.

On the other hand, how would putting a clear, hinged plastic cover over an emergency power kill switch be likely to kill someone? I know people panic in desperate situations, but if someone can't get a plastic hinged cover off of a button quickly during an emergency they shouldn't be trusted with electricity.

There are many ways you could safely "fuck with" the safety equipment while making it less likely to take down your entire network. You could make it a handle that had to be pulled down, like most fire alarms are. It could be "Break flimsy plastic and press button to kill power." Heck, it could just be recessed, like many good last-resort buttons are.

Missing opportunities

[Rosco P. Coltrane]

Apparently someone "accidentally" pushed the emergency power off

They had to power back on when they realized deadjournal.com was already taken...

LJDotting: LJ user base vs Slashdot user base.

[TrevorB]

If Mr. "I Pushed The Big Red Button"'s personal information ever gets published....

LJ's active user base is easily 10x that of Slashdot's. We'd have to come up with a new term for the internet event that pales any slashdotting that ever came before.

...and ran off?

[stratjakt]

What do you mean, ran off?

Ran off skipping and giggling, like a 13 year old who just put toothpaste on the toilet seat?

Or do you really mean, slunk off, like my dog does when I walk in and find her curled up on top of the remains of the remotes for the TV, TiVo, DVD player and stereo?

My dog likes remote controls more than snausages.

OT: Anyone know where (brick and mortar) to get a replacement (original) TiVo remote?

Ahhhh silence is GOOOOLDEN

[ShatteredDream]

*crickets chirping* That's the sound millions of teenage girls not using up bandwidth and disk space talking about boys, jcrew and high school/college drama.

Re:Ahhhh silence is GOOOOLDEN

[metalhed77]

So says the author of yet another political weblog whose startling impartialiality and sense will pave the way for a brave new world?

LOL! Kindof like when...

[GillBates0]

...when I was on AOL and I hit the X and I couldn't talk to my AOL Buddies anymore.

And I was like OMG I shut off the internets and stuff!!1!!

And i called the AOL helpdesk and they helped turn it back on.

Re:I want to name this file.....

[Cocoronixx]

uhhh 0? Well I guess 1 since I can count you now.

Re:How do you do that by *accident*????

[FudgePackinJesus]

Stimpy couldn't resist "The Red, Shiney, CANDY-LIKE Button!!"

13 yo? :P

[Spy der Mann]

Ran off skipping and giggling, like a 13 year old who just put toothpaste on the toilet seat?

By any chance, was his name "Zero Cool"?

2 accounts of the powerloss

[Spazholio]

The one they tell you about and the real one.

Re:Auto-negotiation

[Undertaker43017]

The part I like is they are claiming that everyone else is wrong, and they are right. ;)

I don't buy Cisco anymore for this very reason, it's not just their switches, it's on everything they make that has a NIC.

I deployed some CSS's, right after Cisco bought ArrowPoint, and they did auto correctly. Another client deployed some a couple of months ago, and auto was broken. Cisco is the Borg! ;)

Re:How do you do that by *accident*????

[AndroidCat]

Another customer in the facility accidentally pressed the EPO button, then depressed it

I'm trying to figure out how depressing a button reverses a press. (Since the button is depressed by pressing it.) Unpressed it?

Re:Its a Small World...

[radish]

LiveJournal got hit the hardest, they had some IDE drives on their servers, doh!

I was unaware that SCSI drives had the ability to run without power - thanks for the info!