Slashdot Mirror


SSD Failure Temporarily Halts Linux 3.12 Kernel Work

jones_supa writes "The sudden death of a solid-state drive in Linus Torvalds' main workstation has led to the work on the 3.12 Linux kernel being temporarily suspended. Torvalds has not been able to recover anything from the drive. Subsystem maintainers who have outstanding pull requests may need to re-submit their requests in the coming days. If the SSD isn't recoverable he will finish out the Linux 3.12 merge window from a laptop."

40 of 552 comments (clear)

  1. Really? by koan · · Score: 5, Insightful

    No backup?

    --
    "If any question why we died, Tell them because our fathers lied."
    1. Re:Really? by gagol · · Score: 4, Insightful

      I found spinning rust to at least give some clues prior to a crash and burn. I would say, single ssd is not ready for anything critical, in my opinion. Worst case scenario, you can always get the platters transfered in a good drive and recover from there (pricey, bur cheap if data is valuable enough).

      --
      Tomorrow is another day...
    2. Re:Really? by Anonymous+CowWord · · Score: 5, Funny

      Haven't you heard?

      "Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it ;)" - Linus Torvalds[1]

      1: https://groups.google.com/forum/#!msg/linux.dev.kernel/2OEgUvDbNbo/bTk-VE1zrnYJ

      --


      Disclaimer: My opinions are my own and do not, in any way, reflect the opinions of my employer or university.
    3. Re:Really? by Anonymous Coward · · Score: 5, Informative

      No backup?

      http://lkml.indiana.edu/hypermail/linux/kernel/1309.1/01690.html

      I long ago gave up on doing backups. I have actively moved to a model
      where I use replacable machines instead. I've got the stuff I care
      about generally on a couple of different machines, and then keys etc
      backed up on a separate encrypted USB key.

      So it's inconvenient. Mainly from a timing standpoint. But nothing more.

      Linus

    4. Re:Really? by SJHillman · · Score: 5, Funny

      Maybe Linus doesn't consider Linux to be critical...

      Microsoft sure as hell doesn't seem to find Windows to be critical.

    5. Re:Really? by Anonymous Coward · · Score: 5, Insightful

      I used to think that too, until I had a mechanical hard drive experience controller failure without warning. Single drive is not ready for anything critical, regardless of the storage mechanism.

    6. Re:Really? by stewsters · · Score: 4, Funny

      Yeah, i wonder if anyone has ever told him about git. Too bad he didn't back it up. Now we will have to start a new Linux kernel.

      Sarcasm Intended.

    7. Re:Really? by pubwvj · · Score: 5, Funny

      Ah, even Jesus saves. ;-)

    8. Re:Really? by chuckinator · · Score: 4, Interesting

      Seconded. I've had a RAID1 mirror on my primary workstation at home for roughly... 4 years. I had one of those "oh, drat, my drive is starting to click, and we all know what that means..." moments and barely had time to backup the /home partition to an external machine while I went hardware shopping. Since that event window closed, that configuration has saved my butt twice. One time, the mirrored pair started to go after kinetic shock from moving to a new residence, and it didn't even stress me out to wait for a new pair from my online vendor of choice. I don't know what happened the second time, but I'm guessing that some bad components on the mobo were dirtying the 5V and 3.3V power rails into the drive connector because the whole rig decided to go kaput shortly after in a way that forced an upgrade to the latest CPU socket du jour mobo. Thankfully, I was already budgeting for new guts for that rig due to performance demands.

    9. Re:Really? by tlhIngan · · Score: 5, Informative

      I found spinning rust to at least give some clues prior to a crash and burn. I would say, single ssd is not ready for anything critical, in my opinion. Worst case scenario, you can always get the platters transfered in a good drive and recover from there (pricey, bur cheap if data is valuable enough).

      Sudden SSD failure is actually not really a failure that's detectable. Good SSDs have tons of metrics available through SMART including media wear indicators that tell you impending failure long before it happens.

      But when an SSD suddenly dies, it's generally because the controller's FTL tables got corrupted. For high performance drives, it's remarkably easy to do as performance is #1, not data safety. There's nothing wrong with the disk or the electronics.

      The FTL (flash translation layer) is what maps a sector the OS uses to the actual flash sector itself. If it gets corrupted, the controller has no way of accessing the right sectors anymore and things go tits up. It's even worse because a lot of metrics are tied to the FTL, including media wear, so losing that data means you can't simply erase and start over - you're completely hooped as the controller cannot access anything.

      If you want to think of it another way, treat it like the super block on a filesystem, and the filesystem tables. Now imagine they get corrupt - the data is useless and recovery is difficult, even though the underlying media is perfectly fine. It's possible to hose it so badly that recovery is impossible.

      For speed, FTL tables are cached - and modern SSDs can easily have 512MB-1GB of DDR memory just to hold the tables. Of course, you can't write-through changes since the tables themselves need to be wear-levelled on the flash media.

      One of the iffiest times for this comes when an SSD is power cycled - pulling the power on an SSD can cause corruption because the tables may be in the middle of an update. But things like firmware bugs and other things can easily corrupt the table as well (think a stray pointer scribbling over the table RAM). A good SSD often has extra capacitance onboard to ensure that on sudden power failure, there is enough backup power to do an emergency commit to flash. This protects against power cycling, but firmware bugs can still destroy the data.

      Of course, SSDs without such features mean the firmware has to be extra careful. And sometimes, such precautions can miss a point in time where you cannot pull the power at all.

      It's sort of reminiscent of that Seagate failure that resulted in a log file reaching a certain size disabling the drive - the data and media were perfectly fine, it's just that the firmware crapped out.

    10. Re:Really? by You're+All+Wrong · · Score: 5, Informative

      Are you attempting to claim the prize for the person with the least understanding of the Distributed Source Code Control System in use?

      There was absolutely no code on his system that wasn't on between dozens and thousands of other systems depending on its age.

      Just read TFA: "I had pushed out _most_ of my pulls today". His "pulls" are code that is *elsewhere*. He's just a conduit (and gatekeeper) between a few dozen elsewheres and a server with a fat pipe. And by the construction of the system, it really shouldn't matter how those pulls ordered. (If there'll be a merge conflict one way round, there'll be a merge conflict in other permutations too.)

      --
      Your head of state is a corrupt weasel, I hope you're happy.
    11. Re:Really? by Talderas · · Score: 3, Funny

      Apparently Linus.

      --
      "Lack of speed can be overcome. In the worst case by patience." --Znork
    12. Re:Really? by Guspaz · · Score: 4, Informative

      What makes you think you can't take FLASH devices and access them in a similar way to platters?

      Because on most SSDs, the data is encrypted, and on all SSDs, the pages are in an effectively random order. If you've lost the controller, you've lost both the encryption keys and the table that enables a logical platter-style presentation of the pages. No amount of soldering is going to fix those problem.

    13. Re:Really? by michrech · · Score: 5, Informative

      That's just as easy as popping off the back of the HD removing a couple a screws and pulling out the platter.

      You do that outside of a cleanroom and your data is gone forever.

      False -- I've done it on a number of occasions (to drives I didn't care about), and was able to run the drives for months without their covers. I'd still be using the drives if I had need for drives as small as they were (somewhere in the 80GB range)...

      Would I use a drive in this state for something critical? No, but saying you immediately lose the data if you pull a drive cover is just flat wrong.

      --
      bork bork bork!
    14. Re: Really? by Cyberax · · Score: 5, Funny

      You've misspelled 'NSA'...

    15. Re:Really? by gagol · · Score: 5, Informative

      This is more like a MS employee workstation crash. The linux infrastructure is not hosted on Linux home machines, and replicated around the world. I was simply pointing my favorable opinion for slow spinning disks... not blaming Linus or whatever, shit happens.

      --
      Tomorrow is another day...
  2. Eggs, Basket by Sneakernets · · Score: 5, Funny

    That's all that Ballmer needs to stop Linux? Just find Torvald's SSD?

    --
    "No freeman shall ever be debarred the use of arms." -- Thomas Jefferson
  3. Linus said something... by IMarvinTPA · · Score: 3, Interesting

    Linux said "So I don't want to necessarily blame the harddisk, since it's just ten
    days since I upgraded the rest of my machine, after it worked years in
    the previous one. That just makes me go "hmm". As far as I know, all
    the fans etc were working fine, but.."

    There's his problem: "after it worked years in the previous [machine]."

    His SSD died a natural death of old age.

    IMarv

  4. Re:Next project - backups! by geek · · Score: 5, Funny

    Allow me to channel Linus Torvalds a minute:

    "What do you mean there wasn't a backup disk? Fucking kill yourself with a pipe wrench. I hate you, your mother was a whore and your dad was the neighbors dog. People like you make me sick."

  5. why this news? by Laxori666 · · Score: 4, Insightful

    Why is this news... is this our version of People magazine, where instead of hearing about all the details of the Kardashians' lives, we hear about every email or event that happens to Linus?

  6. Welcome to how SSDs fail. by Mike_EE_U_of_I · · Score: 5, Interesting

    I've owned several hundred hard drives over the last 30 years. I've never had an active hard drive drive just blank out. I have had drives that had not been powered for a couple of years refuse to ever come back. But if I did not feel the need to even power the thing on for years, you can imagine how little I cared for what was on it.

        In the last four years, I've owned around 20 SSDs. I've had five failures. Every single one was the drive just instantly lost everything. Amazingly, in four of the five cases, the drive still worked fine! It had simply lost all the data on it and believed itself to be a blank drive.

        That said, the speed of SSDs makes them worth the risk to me. But I take backups far more seriously than I used to. I need them far more often.

    1. Re:Welcome to how SSDs fail. by RichMan · · Score: 3, Informative

      A hard shutdown of high-speed SSD is death. It takes really really good firmware to recover without reinitializing the drive.

      The basic SSD "format" is susceptable to damage on power fails in a way that hard drives are not. The mapping and setup stables of the SSD are critical and constantly in flux unlike a harddrive where the mapping is only updated when a failure occures.
      SSD drives need internal power fail control so they can gracefully shudown and firmware that supports it.

  7. Re:Someone flame him... by sjames · · Score: 5, Insightful

    He has backups all over the world. But like with any backup, you can't actually restore from it until you replace the failed disk.

  8. Re:None of that mattered, because by Zero__Kelvin · · Score: 4, Informative

    That is correct. In fact he wrote the code that is the industry standard and uses it every day. How else do you think he is going to continue completion of the project on his laptop.

    --
    Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
  9. Kernel Panic!!!! by Cmdrx · · Score: 4, Funny

    Now there a new meaning for Kernel Panic!

    --
    I could write something witty for my sig, but instead wrote this...
  10. Re:Intel? by stkris · · Score: 3, Informative

    More info here: http://goran.krampe.se/2013/01/02/ssd-nightmare/
    "So power cycling can apparently trigger this - and the disk for some odd reason (self protection?) decides to decapitate itself and set accessible cylinders down to 16 instead of 16384."

  11. Re:No RAID? No backup? by samjam · · Score: 4, Funny

    His SSD gave up out of shame for all the threats and abuse it had been forced to witness

  12. Re:Next project - backups! by PRMan · · Score: 4, Insightful

    It's comments like these that make me wish Slashdot mods could go to 10 instead of 5. Nicely done.

    --
    Peter predicted that you would "deliberately forget" creation 2000 years ago...
  13. RAID by Larry_Dillon · · Score: 5, Interesting

    I'm not nearly as much of a believer in RAID for the home environment. If you (accidentally) delete something on one drive it's gone from both. Better to buy two drives and do a daily rsync. That way you have a window of opportunity to recover data. Personally, I use rsync without --delete until the 2d drive starts getting full, then I use the --delete flag to clean up.

    --
    Competition Good, Monopoly Bad.
    1. Re:RAID by Trogre · · Score: 5, Informative

      You guys should really look at the --backup and --backup-dir options in rsync.

      I use them in conjunction with --delete to always have a "current" copy of the data, along with any old files (ie that have been updated or deleted) in a separate backup folder, named after the current day of the month.

      That way you get a directory structure as follows:
      01
      02
      03
      04 ...
      31
      Current

      You can restore the up-to-date set from Current at any time, and if you want to retrieve a file you deleted or over-wrote five days ago, go look in folder 06.

      --
      "Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
    2. Re:RAID by Miamicanes · · Score: 5, Interesting

      The thing that really sucks about SSDs (at least, Sandforce-based drives) is the fact that 99% of their failures are due to firmware bugs that can be simultaneously triggered on an entire array at once (especially the sleep-related bugs). It's a mode of failure the creators of RAID 1, 5, and 10 never anticipated.

      IMHO, the worst thing about SSDs (at least, those with Sandforce controllers) is the fact that they have mandatory full-drive encryption that can't be disabled, using a key you aren't allowed to set or recover, and gets blown away whenever you reflash the firmware. This means, among other things, if the drive's controller gets itself confused:

      * You can't reflash data-recovery firmware onto the drive. The act flashing it would blow away the encryption key and render the data gone forever.

      * If the drive decides you're trying "too hard" to systematically extract data from it while it's in a confused state, it'll go into "panic mode" by blowing away the encryption key. If this happens, your data is gone forever AND you have to send the drive back to OCZ or whomever you got it from in order to get it unlocked. For your protection, of course. And Hollywood's. Among other things, dd_rescue/ddrecover can trigger panic mode.

      * You can't even do the equivalent of removing the platters from a conventional drive in a clean room and mount them to another drive for reading, because the data on the flash chips is all encrypted, and the key is unrecoverable.

      This is BULLSHIT, and it's why I refuse to buy any more SSDs. I, as an end user, should be able to download a utility from somewhere, reflash the drive to firmware that includes an offline recovery mode that simply dumps the flash chip content from start to finish, and either disable the encryption or set it to a key *I* control, so the 99.99999% of the data on the drive that's good when the embedded firmware freaks out can be dumped and recovered offline.

      If there's a God, Linus will go NUCLEAR over this, get a few seconds on CNN & other networks to rant about the unreliability of SSDs, and scare enough consumers to hit the industry HARD where it'll hurt the most... their bank accounts.

      It might not be possible to make SSDs reliable, but DAMMIT, they should at least be RECOVERABLE. There were goddamn hard drives with recoverable data pulled out of laptops left in safes in the Vistamark hotel when a tower sheared it in half and buried it under flaming rubble, yet a SSD that dies if you so much as look at it the wrong way due to firmware bugs ends up being fundamentally unrecoverable for no hard technical reason.

      And yes, I'm bitter about having my hard drive commit suicide for no reason besides Sandforce Business Policy. As long as they keep making controllers that cause drives to self-destruct at the drop of a hat, I'll keep doing my best to talk people out of buying drives tainted by their controller chips. Sandforce sucks.

    3. Re:RAID by Solandri · · Score: 5, Informative

      I stopped using RAID in any of my systems after I started using WHSv1. WHS2011 has the same feature -- live system backups. If a drive fails, I pop in a new one (of any type/size), boot a CD that came with WHS (essentially a WinPE environment with a recovery software baked in), select my backup (I save 7-10 days -- I forget what it's set to), and in about an hour my system is back to the state of the last backup.

      There's the operative phrase. RAID is for systems where you can't have or don't want an hour of downtime while restoring from a backup. The R in RAID stands for redundant. As in you can have a failure and keep going.

      Note that this is the converse of "RAID is not a backup!" Just like RAID is not a replacement for a backup, a backup is not a replacement for RAID either. They do different things (and if you're smart, you will also backup your RAID). From your own description, you wanted a backup. RAID was never the correct solution for your needs.

    4. Re:RAID by drinkypoo · · Score: 3, Insightful

      You're right in that you should never rely blah blah blah, but he's right in that you should be able to attempt recovery. And he's more right, because he never said you shouldn't make backups.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    5. Re:RAID by fnj · · Score: 3, Informative

      Why not do it right?

    6. Re:RAID by bemymonkey · · Score: 4, Insightful

      So... stay the fuck away from Sandforce controllers? This has been common knowledge for years...

  14. Re:You trust Torvalds after this? by hawguy · · Score: 3, Insightful

    As someone who's taken over server administration from very talented developers a number of times, I've found that being a great developer doesn't mean that you're a great sysadmin. Developers may understand conceptually that RAID and backups are important (but sometimes think that RAID is a backup), but that doesn't mean that they actually set them up.

    And as a sysadmin, I'm tired of hearing that. RAID1,5,6,10,Z is a backup. It's not an archive. An archive is what you go to when you want the old version. A backup is generally one of two things:
    1) Something that lets you keep chugging through a failure (raid5, a backup generator with automatic cut-over, etc)
    2) A standby spare (tape, NAS/usb drive, secondary location with desks/computers/etc.

    RAID (other than 0) is absolutely a backup. It's not the perfect backup but it is a backup. What it is NOT is an archive - last night's/week's/month's/quarter's data.

    No, RAID is *not* a backup, RAID's only purpose is to improve reliability/uptime by letting you ride through hardware failures, but it does nothing to protect you from all of the rest of the things that can destroy your data, like file corruption, fat fingering a "rm -rf / home/someuser", a virus, a website hack attack, etc. That's what your backups are for, but you can call them archives if you like, but don't call RAID a "backup" because it's not. Depending on what the problem is and when you discover it, you may need to go back through several archives before you find the data you're looking for.

  15. Re:None of that mattered, because by Zero__Kelvin · · Score: 4, Insightful

    ". Now people have to redo a lot of effort, because he was too lazy or arrogant to install one of the many effortless backup systems available."

    That is a ridiculous statement. Work is lost every time a drive fails unless it happens to fail immediately after a backup. Full backups take lots of time. If you understood git better you would realize that a lot less work is lost the git way than with old school backups. I'm sure that every time Linus does a successful merge he pushes it to a git repo elsewhere. All history is in the git logs. I am certain the work he lost is minimal, and is much less than if he was relying on nightly backups and the failure happened near the end of the work day. Just the effort of trying to determine what was done and what has been lost would be far more time consuming without git.

    --
    Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
  16. Re:Really? Naa by Psyko · · Score: 5, Interesting

    trying to desolder 100 pins spaced 0.01" apart then resoldering them, unless you have a 0.1 mill precision soldering robot it is impossible, you can't even buy wire thin enough to do it by hand.

    SMT rework by hand isint rocket science, but takes more tools than the average garage has.

    Desoldering you use a custom tip for that socket/package type (one tip per package & they're not cheap). It's essentially a metal ring that heats the solder on all the pins at once. In the center of the assembly is a vacuum probe. You heat all the pins, melting all the solder & hit the button on the handpiece to suction the chip up off the board. Then clean up the pads on the board. Careful with the heat because you dont want to lift pads off the board, if you do then you have to either fix them, or make a new pads. And then if you manage to trash a via (conductivity path to a different board layer), then you've got to drill out a new one and you have to use a esd safe conductive drill with a resistance cutoff. You put a clip from the drill in contact with the layer you're trying to get to, drill down and when the drill tip makes contact with that layer the drill turns off because the circuit is complete. But it still sucks and if you don't know how all the board layers are put together you may end up trashing a trace a couple layers into the board and wrecking the whole thing.

    Soldering it down you do this. Align all the chip legs on the pads. Then you can either run a small bead of solder paste across all the pins or use a wave soldering tip (small cup, uses surface tension to hold the solder in place) and drag the tip over all the pins. Heat on the pin & pad draws the solder down into the joints. If you put too much solder you might have to vac it back up and redo it if you've made bridges etc. Alignment is key, and keeping the part in position is key. I used to try and avoid using glue underneath because that made it difficult to get it back off if you needed to down the road.

    Doing hand rework on that kind of stuff the hardest thing for me was dealing with smt chip caps, little bastards will crack if you heat em to fast, so you have to get a temp regulated hot plate, heat em up slow, then pick and place em quick with tweezers/needlenose & solder em down quick.

    --
    01:36AM up 426 days, 2:46, 1 user, load average: 0.14, 0.11, 0.05
  17. RAID != Operating System by dutchwhizzman · · Score: 5, Interesting

    You have a software feature in a server OS that supports certain client OSes to do backups to the server. RAID may be a software feature, but even if it's "software raid", you often have BIOS bootable raids that even work with one of the drives missing. This essentially means that you can work OS agnostic on a lower level than "I have a backup system that works". For Linux, you can have a backup system too that will restore from a LiveCD/USB stick and stores on a remote server. The same amount of time roughly will be needed to backup and restore, differential, incremental, full backups, the works. The solution you are providing is really nothing comparable to RAID. It's fundamentally different because it works on a totally different layer, doesn't prevent downtime and it's not OS agnostic. RAID should prevent downtime, making working backups should prevent data loss. Maybe WHS is the shizniz, you rock for making actual backups, but other than that, your post is totally offtopic in this context and doesn't even begin to solve a problem that Linus was facing with his desktop.

    I'm not modding you down, even though I have mod-points, but I'm telling you exactly why I think you shouldn't have posted this. I hope you learned something from it and in the future will implement both backups and RAID when unscheduled downtime is important. Maybe you would even implement a system that works for all relevant OSes in the environment you have to do it for, without relying on a single vendor that offers a closed source product. It's a risk that means you'll have to support their product and licencing and other requirements until the data isn't relevant anymore, even after you have migrated to a competing product.

    --
    I was promised a flying car. Where is my flying car?
  18. Will never work with modern drives by dutchwhizzman · · Score: 5, Informative

    Modern drives for the last five years at least, have calibration factors for platter/head packs on the EEPROM on the controller board. If you swap boards, the board most likely won't be able to read the data on the disk, since it's not calibrated to the head/platter kit.

    --
    I was promised a flying car. Where is my flying car?