Slashdot Mirror
A Note On Thursday's Downtime
If you were browsing the site on Thursday, you may have noticed that we went static for a big chunk of the day. A few of you asked what the deal was, so here's quick follow-up. The short version is that a storage fault led to significant filesystem corruption, and we had to restore a massive amount of data from backups. There's a post at the SourceForge blog going into a bit more detail, and describing the steps our Siteops team took (and is still taking) to restore service. (Slashdot and SourceForge share a corporate overlord, as well as a fair bit of infrastructure.)
oh, I thought some of that shitware they sling got loose and bit them in the ass
Sourceforge is Badware risks .. http://i.imgur.com/Hhtgv0H.png
like unicode support and ipv6.
Could have been far worse...
Lately I have noticed quite a fair bit less of the typical trolls.
An increase in numbers has resulted in an increase in their diversity - plus there's so little time when they've so many places to go in their desperate battle for attention. The troll union proposed hot-bedding and time-sharing but, for obvious reason, were unable to get the propositions ratified.
Noted troll think tank UnderTheBridgeWatch, recently published a report predicting that the recent balkanization of troll unions due to a large number of them getting married (the others just want to sleep around) under the new gay marriage laws will further increase the diversity of their appearance. Because unlike humans, trolls of the same gender do produce offspring.
All right! Nobody moves, or the storage gets it! .... Help me! Help me! .... Shut down! ..... Won't somebody help that bad drive?!
The reboot is near.
much of left-wing thought is a kind of playing with fire by people who don't even know that fire is hot - George Orwell
I clicked on a "firehose" link and the most recent story was "YouTube's ready to select a winner" from March 2013.
But the "help us select the next story" link was ok, as was directly entering Slashdot.org/recent.
Good luck with the restore / clean up / troubleshooting. That's not a fun way to spend a weekend.
much of left-wing thought is a kind of playing with fire by people who don't even know that fire is hot - George Orwell
Hardware isn't free and employees aren't free. I seriously don't understand how Sourceforge has kept the lights on all these years.
And by the way, I'm a very satisfied user of their services. But I do worry about their future.
Thank you to the Slashdot team. Bringing systems back up like that is emergency-mode-fun, but a lot of work, and we appreciate it.
It was fast as hell!
here
http://saveie6.com/
Yes, but they have to kill you after they restore your data.
Try Chinese gov't instead.
Table-ized A.I.
And right before the Pluto flyby.
Seriously, though, imagine the thoughts going through NASA minds when the probe crapped out a week before the big encounter. Their toilets must have been full of bricks.
It's not like rover problems where you can continue where you left off after you fix it. New Horizons couldn't stop.
Table-ized A.I.
It's great to see how you responded to the failure and got services resumed pretty quickly. However, I'd rather like to see a follow-up sometime, describing a root cause analysis. With all the clustered, distributed servers and filesystems you use today, such an outage shouldn't be possible, right?
To Terminate, or not to Terminate, that's the question - SCSIROB
Serious question: how much of this could have either had been prevented, or restored much more quickly if they were using ZFS with proper parity, checksuming, snapshotting, and sending (backups)? This really is the one-size-fits-all storage solution at this point.
[...] This incident impacted all block devices on our Ceph cluster.
Power/communications/routing down event? Was monitor quorum lost? Inquiring minds that are not trolls are curious and grateful that the path to restoration was clear. Best wishes.
<blink>down the rabbit hole</blink>
Personally, Ive off-shored & out-sourced a majority of my trolling, passive-aggressive self-rightous diatribes and compensated product endorsements. This leaves more time for, well, pr0n. Gotta have a life sometime ya know.
The Russians have won. They have made the world a cesspool of distrust, greed, fear and hate.
I've negligible experience in this sort of failure and recovery, but...
Shouldn't slashdot and sourceforge be entirely separate, so that the failure of one can't bring down the other?
Shouldn't there be live redundant systems, so that when one fails, one of the redundant systems is switched online in minutes? I don't mean just redundant storage, but 3 or 4 systems running concurrently, taking the same input and monitoring to confirm that the output is the same.
Is this too expensive or not technically feasible?
Contribute to civilization: ari.aynrand.org/donate
Yup, they kill you BEFORE they restore your data.
...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
Your important files encryption produced on this computer: photos, videos, documents, etc. Here is a complete list of encrypted files, and you can personally verify this.
Encryption was produced using a unique public key RSA-2048 generated for this computer. To decrypt files you need to obtain a private key. The single copy of the private key, which will allow you to decrypt the files, located on a secret server on the Internet; the server will destroy the key after a time specified in this window. After that, nobody and never will be able to restore files.
To obtain the private key for this computer, which will automatically decrypt files, you need to pay 666 Bitcoin like similar amount in another country. Click >nezt> to select the method of payment and the currency. Any attempt to remove or damage this software will lead to the immediate destruction of the private key by server.
The Russians have won. They have made the world a cesspool of distrust, greed, fear and hate.
"Storage corruption" is fairly vague. I've been bit by it in the past - once due to a vendor software bug (Oracle block corruption), and once due to hardware (flaky storage controller chip writing garbage (Supermicro MB)) I would like to hear more about the root cause.
RM
}#q NO CARRIER
It looks like /. had a Plan B ready in the case of a catastrophic failure. For some sites one just gets a blank page with some strange message when that happens. /. did the right thing letting users know they had a problem and were working on it and then let us know a bit about what happened. Thanks, /. techs.
In a time of universal deceit, telling the truth is a revolutionary act. George Orwell
On the same Thursday that Slashdot experienced data storage corruption, the 1TB hard drive on my Windows gaming PC crashed, reporting 4GB of free space available and unresponsive to IO block commands. (I've seen that behavior on USB sticks, but never on a hard drive.) Except for several years of email, all my data was on the file server. Oh, well. I got a good excuse to rebuild my eight-year-old PC, especially with Windows 10 around the corner. Meanwhile, I'm using a $250 Dell laptop for everything except gaming.
Adjectives are words though.
This is sad news. I was starting to miss the GNAA and goatse.
Not once I am done with them!
You can't handle the truth.
Because unlike humans, trolls of the same gender do produce offspring.
How does that work biologically?
They tried Unicode before. It allowed spoofing comment scores. SoylentNews claims to support Unicode; I wonder how it prevents spoofing comment scores.
Because unlike humans, trolls of the same gender do produce offspring.
How does that work biologically?
Like flies on shit.
Note that there is only one gender of trolls, and yet they increase in number. Proof!
Witling faecus the rightfully endangered, potty-mouth troll.
Habitat: under bridges, in sewage systems, amongst the cruft of computer systems. Dark places close to humans.
Weaknesses: Humour, facts and sunlight. Humour causes them pain. Exposure to sustained Facts or strong Light are fatal.
Appearance: Various aberrations. Recognisable by the unique pin-like growth on their neck - the only visible feature that reliably distinguishes them from sentient bipods.
No pictures exist due to their sensitivity to light of any form Artists impression
History: A hydrogen-sulphide based life-form, possibly originating from the interaction of decomposing faeces and swamp gas. Whether they in fact qualify as a life form, or possess sentient capabilities is uncertain. It's theorised that they appeared when the first ancestors of humans developed intelligence, and that trolls have been devolving ever since. The theory is much debated and purely hypothetical as the only historical records are in the form of ancient legends due to the lack of fossil record. They have no backbone and upon death leave only a nasty stain and a foul odour.
Biology: Their "closed-loop" digestive system allows them to survive their entire life eating only their own excrement. As their brain is composed of only two cells (neither of which function) they are unable to support any distro other than Windows - and even then, only the sliding kind.
Additional references: Troll study, Suler, J.R. and Phillips, W. (1998). Deviant Behavior in Multimedia Chat Communities.
This is sad news. I was starting to miss the GNAA and goatse.
Take heart, there is always gonorrhoea and syphilis. And Foxx News.
I say this as someone that runs ZFS on his backup/file server; if you do have to restore or resilver it can take a long while! A single slow drive in a vdev will limit the entire pool's IO (the extent of which is entirely dependent on topology, but the weakest link always crushes you in ZFS). After a handful of TB of data, even with a pool of mirrored vdevs and a flash cache device, the resilver for a single drive can take a day unless you've got some serious spindle count at high RPMs. Even SAS drives don't provide that many IOPS.
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
Nice to see that blurb of text again. Can we get this to happen every time you post a Nerval's Lobster/Dice slashvertisement, too?