Slashdot Mirror
LiveJournal Servers Go Down
Wind writes "According to any journal hosted off of LiveJournal.com, the LiveJournal data center Internap has suffered a critical power failure, leaving all of LiveJournal and its content temporarily offline and requiring the revival of 100+ servers. Perhaps Six Apart wasn't quite prepared for the responsibilities of a website of this size? Updated information is posted here."
On the Livejournal main page:
Update #1, 7:35 pm PST: we're up on 'dirty' power for now (it works, but it's unreliable), and we're working to assess the state of the databases. The worst thing we could do right now is rush the site up in an unreliable state. We're checking all the hardware and data, making sure everything's consistent. Where it's not, we'll be restoring from recent backups and replaying all the changes since that time, to get to the current point in time, but in good shape. We'll be providing more technical details later, for those curious, on the power failure (when we learn more), the database details, and the recovery process. For now, please be patient. We'll be working all weekend on this if we have to.
Lovely. I just bought another year's subscription for my wife, figuring the change to Six Apart wouldn't change anything for a few months at least. LJ could lose a lot of subscribers with an outage just after the takeover.
This is another thing that bothers me about this scenario. I can't say that I've ever admined 100 servers, the most I've ever had was about 30, but if we had a power loss of any kind, you'd just repower them and walk away. Most of them were DEC Alpha gear running Tru64. Why would you spec out a box that has to be handheld every reboot? The only time you should have to handhold a server is during an upgrade. A power cycle without proper SIGHUP or term signals should just run fdisk on it's way back up. (K, so it might take an hour for the server to go live again, but still.) I mean, am I missing something here? Maybe since nothing I've admined got the traffic these things do .... I'm just lost. Some one hit me with the clue by four.
The only thing I can even think of is they have explicit services that must be started manually ..... but why would you want that? If you have a power hiccup in the middle of the night, you want it to come back up, and be live and happy again *before* you even get the first page. I mean sure, if there was a surge, and that destroyed components, and those components have to be replaced ..... but ..... a reboot is a reboot, man. Here, smoke some source. It's the good stuff.
"Genius may shine aloof and alone, like a star, but goodness is social, and it takes two men and God to make a Brother."
Back in the great days of the .com boom, people were building their colo facilities to insane (in a beautiful way) standards. I remember touring Exodus and Above.net (I don't know who you're referring to, though I only ever heard of above.net adopting flywheels) and being just very amused at the cool stuff they were putting in place.
... how much are all these people paying LiveJournal again? Couldn't they request some sort of partial refund of their monthly fee?
I recently (~8 months back) did some contract work for a small company whose servers were based in some colo facility in San Francisco. One of the first things I noticed was a damn heavy UPS at the bottom of their rack. Weird, I thought -- why not rely on the colo's battery system?
Because they don't have one.
Mind you, this was also the colo that had a cardkey system that had long ago stopped being usable, so when you needed access you used a Radio Shack $29.99 wireless intercom system and someone would come to open the door, and when you checked in they carefully wrote your name on a little nametag.
I think standards have slipped, significantly. In some respects, this is likely a good thing -- it means you have more options now, because you can choose either the super duper "we hook up to two countries' power grids, have eight flywheels and a direct feed from microwaves in orbit" or the "err, here's your cabinet. We'll give you decent power until we don't" options.
So
Oh, wait...
Sorry, but one man can't control power to an entire co-location. I used to work for a local telecom and one day our fiber went down for about 1/3 of our customers. The reason? Some guy shootin' squirrels blew the fiber lines apart. It was on a Sunday too, when there was minimal phone tech support, I think one guy ended up fielding 350+ calls by himself.
For those who don't know what's so hot about it and for those who think Livejournal is just a bunch of teenage girls whining.... Livejournal has just about four years of my life documented. The ease of use and the ability to "vent" is comforting, but the real value comes in the interaction. My friends see my life at their convenience and I see theirs at mine. We can choose to ignore the whining of others or we can choose to relate and comment on our own experience. Think of it this way: Open-source philosophy, emotion, and life. I put my own out there and others add to it. I add mine to others. Granted ... those quiz/meme things HAVE TO GO. I do not want to read about "what frog best resembles me" or "which 80's hair band song is me." Grrr.
I recall a story from a CP&L (now Prgress Energy) lineman about how someone had shot the fiber line strung on the high voltage tower -- between the million volt lines. As he put it, you put a pencil through the hole and line up where the "hick" (his word) proped up the rifle to shoot what he must've thought was a power line. It no fun working up there. And even less fun having to make fusion plices surrounded by million volt lines. (linemen work up there, not fiber techs.)
I seem to remember that a few years back they had a similar problem (Internap lost all power) and it turned out that some idiot had hit the big red "shut down all power to the entire datacenter" emergency button. This isn't the first time this has happened, and last time it wasn't under Six Apart's management.
I'd say it's Internap's incompetence that caused this problem. If they can't keep their datacenter running even though they have multiple redundant power supplies then something is very wrong. I see from the outage page that LJ people are now planning to buy their own UPS so that they don't have to trust Internap anymore.
For power outages, my house has a better record than Internap right now, and I don't even own a UPS!