Linux 4.0 Has a File-System Corruption Problem, RAID Users Warned

← Back to Stories (view on slashdot.org)

Linux 4.0 Has a File-System Corruption Problem, RAID Users Warned

Posted by timothy on Thursday May 21, 2015 @01:23AM from the don't-store-the-ark-there dept.

An anonymous reader writes: For the past few days kernel developers and Linux users have been investigating an EXT4 file-system corruption issue affecting the latest stable kernel series (Linux 4.0) and the current development code (Linux 4.1). It turns out that Linux users running the EXT4 file-system on a RAID0 configuration can easily destroy their file-system with this newest "stable" kernel. The cause and fix have materialized but it hasn't yet worked its way out into the mainline kernel, thus users should be warned before quickly upgrading to the new kernel on systems with EXT4 and RAID0.

135 of 226 comments (clear)

Min score:

Reason:

Sort:

Linux is clearly unstable! by Anonymous Coward · 2015-05-21 01:26 · Score: 5, Funny

I'll stick with Windows Vista, thanks.
1. Re:Linux is clearly unstable! by lsatenstein · 2015-05-22 04:42 · Score: 1
  
  I'll stick with Windows Vista, thanks.
  I have the 1.44meg and the large 5½inch meg soft floppies, and my msdos 3.1 system just flies. Everything is written in 8086 assembly. I have a small btree application. If it is not broken, why change it. I have accumulated a lifetime supply of spares.
  
  --
  Leslie Satenstein Montreal Quebec Canada
It's RAID 0 by Anonymous Coward · 2015-05-21 01:28 · Score: 1, Insightful

Losing data goes with the territory if you're going to use RAID 0.
1. Re:It's RAID 0 by Enry · 2015-05-21 01:58 · Score: 2
  
  I have 4 drives in a RAID 10, so two RAID 1 arrays of two drives each combined together in a RAID 0. I did it mostly because I can add new drive at any time and just chain them onto the RAID 0.
2. Re:It's RAID 0 by Qzukk · 2015-05-21 02:14 · Score: 1
  
  Linux's MD raid10 isn't the same as RAID 1+0, so I'm not entirely sure it would be affected by this.
  
  --
  If I have been able to see further than others, it is because I bought a pair of binoculars.
3. Re: It's RAID 0 by AvitarX · 2015-05-21 03:43 · Score: 1
  
  Linux raid 10 does not allow adding the drives. That is the advantage to the GPs setup.
  I always use the raid 10 driver, for the purpose of 3 disk arrays, and the performancw benefit of the far layout, but not being able to grow them is annoying.
  
  --
  Wow, sent an e-mail as suggested when clicking on "use classic" banner, and got a fast response that addressed my msg
4. Re:It's RAID 0 by kthreadd · 2015-05-21 05:12 · Score: 2, Insightful
  
  Or it could work just fine. RAID 0 is not dangerous, you may just as well loose your data even if you only use a single drive. Hard drives and SSDs don't go bad that often that it's a problem.
5. Re:It's RAID 0 by tehlinux · 2015-05-21 05:57 · Score: 1
  
  The 0 in RAID0 stands for how much data you are going to get back if you have to recover.
  
  --
  Most linux users don't know this, but the man pages were named after Chuck Norris. Chuck Norris fsck'ing hates noobs!
6. Re:It's RAID 0 by Ksevio · 2015-05-21 06:28 · Score: 1
  
  You should switch to a RAID5! Then you'd get a little extra capacity while still being protected against 1 drive failing.
7. Re:It's RAID 0 by Swave+An+deBwoner · 2015-05-21 07:18 · Score: 1
  
  The standard mantra you are chanting is correct. But given the human propensity for failing to do something that they planned to do (regular fine-grained backups), a redundant array (pretty much anything except RAID 0) can mean the difference between losing some valuable data or development work and not losing it if a disk fails.
8. Re:It's RAID 0 by KingMotley · 2015-05-21 07:50 · Score: 1
  
  Raid 10 can survive SOME 2-drive failures (in a 4-drive raid 10), and has significantly faster write speeds than Raid 5.
  Personally, I use a combination of RAID-0 and RAID-6 (not the same array), because Raid-5 for large arrays is almost useless. I've seen too many raid-5's die when the bad drive is replaced and the added stress of the rebuild then kills a second drive. Ouch.
9. Re:It's RAID 0 by sound+vision · 2015-05-21 08:20 · Score: 1
  
  It's not quite "as well", with RAID 0 if you lose either of two drives, the data is gone. That effectively doubles the chance of failure.
  If you only have the budget to play with 2 drives, you should be using one drive normally and the second drive as an external backup. Not RAID 1, that leaves you open to software/file system errors and the like. Having the backup as a separate drive that's not plugged in except when running backups negates a lot of those failure modes.
10. Re:It's RAID 0 by Forever+Wondering · 2015-05-21 11:40 · Score: 4, Informative
  
  Based on the commit fixes, it's in a function called raid0_make_request, which is only used in raid0.c
  raid 10 is in raid10.c, so it doesn't use this function.
  The bug is based on the fact that a macro "sector_div" modifies it's first argument [and returns the remainder]. I've removed the obligatory backslashes for clarity:
  
  # define sector_div(n, b)(
  {
  int _res;
  _res = (n) % (b);
  (n) /= (b);
  _res;
  }
  )
  This is used in some fifty files. Some just want the remainder [and they don't want the first arg changed so they do]:
  
  sector_t tmp = sector;
  rem = sector_div(tmp,blah);
  This is effectively what the code wanted, but the actual fix was to do a restore afterwards:
  
  sector_t sector = myptr->sector;
  ...
  rem = sector_div(sector,blah);
  ...
  sector = myptr->sector;
  ... // use sector [original value only please ;-)]
  The last line to restore sector with the original value was the fix.
  They should do a full code audit as their may be other places that could be a problem. I've reviewed half the files that use this macro and while they're not broken, some of the uses are fragile. I paraphrase: "sector_div considered harmful"
  What they really need are a few more variants which are pure functions that could be implemented as inlines:
  rem = sector_rem_pure(s,n)
  s2 = sector_div_pure(s1,n)
  Or, a cleaner sector_div macro:
  sector_div_both(s,n,sector_return,rem_return)
  
  --
  Like a good neighbor, fsck is there ...
11. Re: It's RAID 0 by cthulhu11 · 2015-05-22 07:37 · Score: 1
  
  Beat me to it. RAID5 is almost always the wrong answer.
Which RAID are they referring to? by fang0654 · 2015-05-21 01:29 · Score: 1

Article isn't very clear - are they referring to softraid, fakeraid, and/or hardware raid?
1. Re:Which RAID are they referring to? by bakaorg · 2015-05-21 01:46 · Score: 5, Informative
  
  md raid. The bug was in commit md/raid0: fix bug with chunksize not a power of 2 causing, you guessed it, a bug with a chunksize not a power of two. I guess "fix" was a bit diversionary.
  
  The real problem was a macro modifying arguments that were later expected to be the unmodified version.
2. Re:Which RAID are they referring to? by Anonymous Coward · 2015-05-21 02:14 · Score: 1, Informative
  
  That patch also says that the bug exists since Linux 3.14. So it is NOT 4.0, but every kernel since 3.14 that is affected here.
3. Re:Which RAID are they referring to? by msauve · 2015-05-21 02:39 · Score: 4, Informative
  
  No. There was a minor bug introduced at 3.14. The patch to fix that, completely different issue, went into 4.0 and caused this corruption issue.
  
  --
  "National Security is the chief cause of national insecurity." - Celine's First Law
4. Re:Which RAID are they referring to? by MSG · 2015-05-21 04:55 · Score: 3, Informative
  
  That fix is actually in the wrong place. The fix for that is tracked in kernel.org's bugzilla # 98501. I'm not linking directly as linking to bugzilla tends to place too high a load on those systems. It's impolite.
  Neil Brown said that he'd push the fix to Linus "shortly" at 2015-05-20 23:06:58 UTC. I still don't see the fix in Linus' tree.
  Watch for a fix titled "md/raid0: fix restore to sector variable in raid0_make_request"
5. Re:Which RAID are they referring to? by nightsky30 · 2015-05-21 05:36 · Score: 1
  
  Hmmmm, let's call it RAID PI
The disks usually don't ... by CaptainDork · 2015-05-21 01:32 · Score: 1, Offtopic

... need to be debugged, so using Raid® is probably the cause of this.

--
It little behooves the best of us to comment on the rest of us.
1. Re:The disks usually don't ... by Culture20 · 2015-05-21 01:43 · Score: 1
  
  You laugh, but the first computer bug was a moth.
  https://www.youtube.com/watch?...
2. Re:The disks usually don't ... by Anonymous Coward · 2015-05-21 02:14 · Score: 1
  
  No, the cited log entry shows that the word "bug" was in use, but that they weren't commonly "found". That's because at the time "bug" meant some kind of mysterious issue with electronics, aka gremlin. The joke was that a bug had been *found*, at last. Computer bugs are often found, implying that this was the first application of the word "bug" to computer issues (with actual solutions) i.e. the first "computer bug". It's also where "debugging" was coined.
  Fairly well explained here.
stable by rossdee · 2015-05-21 01:34 · Score: 4, Funny

this is obviously some strange usage of the word "stable" that I wasn't previously aware of.
1. Re:stable by Anonymous Coward · 2015-05-21 01:38 · Score: 5, Funny
  
  If you ever owned horses, you would understand what "stable" means in this context
2. Re:stable by Deep+Esophagus · 2015-05-21 02:13 · Score: 2
  
  This. My first thought upon reading TFS was, how did this ever pass peer review and testing to get into the "stable" kernel? They do still perform peer review and unit testing, don't they?
3. Re:stable by Trevelyan · 2015-05-21 02:20 · Score: 5, Informative
  
  It's stable as in terms of features and changes. i.e. No longer under development and will only receive fixes.
  
  However! Kernels from kernel.org are not for end users, if someone is using these kernels directly then they do so at their own risk.
  They are intended for integrators (distributions), whose integration will include their own patches/changes, testing, QA and end user support
  
  There is a reason that RHEL 7 is running Kernel 3.10 and Debian 8 is running 3.16. Those are the 'stable' kernels you were expecting.
  
  When kernel development moved from 2.5 to 2.6 (that later became 3.0), they stopped their odd/even number development/stable-release cycle. Now there is only development, and the integrators are expected to take the output of that to create stable-releases.
4. Re:stable by dave420 · 2015-05-21 03:01 · Score: 3, Insightful
  
  I understand if you are emotionally attached to Linux to the point where someone accidentally criticising it makes you feel uncomfortable, but you really should be able to figure out that "but... but... they're worse!" is no rational response :)
5. Re:stable by Anonymous Coward · 2015-05-21 03:19 · Score: 1
  
  Microsoft has made bigger blunders, and then refused to fix them.
  So what? Saying that does not lessen the severity of this Linux bug.
6. Re: stable by andydouble07 · 2015-05-21 05:18 · Score: 1, Insightful
  
  Meanwhile, my Win keeps BSOD.
  Really? Sounds like you're screwing something up pretty bad, haven't seen one of those in about 6 or 7 years.
7. Re: stable by jbengt · 2015-05-21 06:08 · Score: 2
  
  I can routinely cause a BSOD (about 1/3 of the time) on my HP laptop running Windows 7 Pro if I use the touchpad at the log-in screen on start-up. It's apparently a known bug in the touchpad driver that will not get fixed.
8. Re:stable by duke_cheetah2003 · 2015-05-21 06:40 · Score: 1
  
  This. My first thought upon reading TFS was, how did this ever pass peer review and testing to get into the "stable" kernel? They do still perform peer review and unit testing, don't they?
  Testing? Who does that anymore? That is the user's job.
  MMO's and Microsoft have made it so.
9. Re: stable by oobayly · 2015-05-21 07:30 · Score: 3, Insightful
  
  It's not. However it isn't beyond a reasonable expectation that a dodgy touchpad driver shouldn't be able to kill an OS.
10. Re: stable by jeremyp · 2015-05-21 21:57 · Score: 1
  
  I can write you a touchpad driver that will crash Linux even if you don't use the touchpad on start up. That Linus Torvalds really doesn't know what he's doing.
  
  --
  All I want is a secure system where it's easy to do anything I want. Is that too much to ask ~~ Randall Munroe
11. Re: stable by jeremyp · 2015-05-21 22:08 · Score: 1
  
  On every popular PC operating system in use today (and Linux), the kernel and the drivers share the same address space. It is thus beyond reasonable expectation for the OS to be able to protect itself from faulty drivers. Although Linux was designed this way from the start, OS X and Windows both started as microkernels (where the drivers have their own address space) but were "downgraded" to monolithic kernels because of performance concerns. Switching between address spaces has an enormous cost.
  
  --
  All I want is a secure system where it's easy to do anything I want. Is that too much to ask ~~ Randall Munroe
12. Re: stable by drinkypoo · 2015-05-21 22:52 · Score: 1
  
  On every popular PC operating system in use today (and Linux), the kernel and the drivers share the same address space.
  There are linux drivers which run in userland.
  
  --
  "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Warning: RAID 0 by Culture20 · 2015-05-21 01:37 · Score: 2, Interesting

RAID 0 is unstable to begin with. Medium case scenario here (for legitimate use) is some data gets corrupted on a compute node. Run the program on two nodes; if you get the same result on both, that result is probably fine. If you're running RAID0 on any filesystem that isn't temporary or at least easily replaceable, you're doing it wrong.
1. Re:Warning: RAID 0 by Enry · 2015-05-21 02:08 · Score: 2, Insightful
  
  RAID 0 is only as unstable as its least stable component. In this case it's most likely a drive failure, and most drives are fairly long MTBFs. The chances of a disk failure increase as a function of time and number of drives deployed. A two-drive RAID 0 will be more stable than a five-drive RAID 0 which will be more stable than a 10 drive RAID 0 that's three years old. In the case of higher RAID levels, you can remove a single (or multiple) drive failure as the point of failure. In this case, the point of failure is the kernel, so it's perfectly legitimate to consider this a really bad problem. Would you say the same thing if the bug affected RAID 1 or RAID 5?
2. Re:Warning: RAID 0 by danbob999 · 2015-05-21 02:10 · Score: 1
  
  There is no valid reason for corruption to occurs on RAID0 anymore than on any other setup. The problem of RAID0 is data loss (drive failure).
3. Re:Warning: RAID 0 by nine-times · 2015-05-21 02:39 · Score: 4, Insightful
  
  Would you say the same thing if the bug affected RAID 1 or RAID 5?
  I suspect not, since his point seemed to be that you shouldn't be using RAID 0 for data that you care about anyway.
  It doesn't really make it ok for a bug to exist that destroys RAID 0 volumes, but it does mitigate the seriousness of the damage caused. And it's true: Don't use RAID 0 to store data that you care about. I don't care if the MTBF is long, because I'm not worried about the mean time, but the shortest possible time between failures. If we take 1,000,000 drives and the average failure rate is 1% for the first year, it's that that comforting to the 1% of people whose drives fail in that first year.
4. Re:Warning: RAID 0 by TheCarp · 2015-05-21 03:18 · Score: 1
  
  I have been running a 4 disk RAID 5 array for a few years now at home, and did a replacement upgrade a couple of years back.
  Overall I find in a 4 disk scenario I lose just a bit less than one disk per year. Maybe one disk every year and a half.
  So when you say RAID 0 that is 3 years old, that sounds about right. I would call such an array in serious danger of loss.
  
  --
  "I opened my eyes, and everything went dark again"
5. Re:Warning: RAID 0 by AthanasiusKircher · 2015-05-21 03:34 · Score: 1
  
  Would you say the same thing if the bug affected RAID 1 or RAID 5?
  I suspect not, since his point seemed to be that you shouldn't be using RAID 0 for data that you care about anyway.
  
  Exactly. About the only reason I would ever use RAID 0 is for some sort of temp data drive where for some reason I wanted to string multiple drives together. You've basically taken a bunch of drives that each would be vulnerable without redundancy and have produced one big drive that will fail whenever any component does, thereby greatly increasing failure rate over individual drive failure rate. There are only a limited set of use cases where this is a helpful thing, and basically all of them are situations where you can assume that 100% data loss won't hurt you.
  
  t doesn't really make it ok for a bug to exist that destroys RAID 0 volumes, but it does mitigate the seriousness of the damage caused.
  Well, it mitigates the seriousness of the damage a bug should cause, assuming that people use RAID reasonably. But that's obviously a poor assumption, since many people seem to love playing Russian roulette with essential data.
  I was lucky enough to have a significant (but not catastrophic) data loss hit me when I was young and didn't have a lot of essential data to lose. It taught me the importance of redundancy and backups. Most people who haven't experienced such things are more cavalier with data -- and a RAID 0 bug could be catastrophic for them.
6. Re:Warning: RAID 0 by iONiUM · 2015-05-21 03:46 · Score: 1
  
  For the record, I have a 6 year old machine running Windows 7 with a RAID-0 setup (asus p5k-e motherboard, WD 250gb drives), and it has never had an issue. It it typically on 24/7, but it has gone through many power outages where the UPS ran out of battery and it hard-reset.
  I do, of course, keep all data on a separate regular drive, along with an external back-up of that. So if the RAID-0 did die, it wouldn't be a big deal (and I could finally move to SSD!).
  Anyways, the point I am trying to make is that RAID-0 is not as "crazy unreliable" as some people would have you believe.
7. Re:Warning: RAID 0 by nine-times · 2015-05-21 03:47 · Score: 2
  
  Well, it mitigates the seriousness of the damage a bug should cause, assuming that people use RAID reasonably.
  I'm going to go ahead and say that it mitigates the serious of the damage caused in actuality since most IT people entrusted with serious and important data aren't going to be that stupid. I mean, yes, I've seen some pretty stupid things, and I've seen professional IT techs set up production servers with RAID 0, but it's a bit of a rarity. There could still be some serious damage, but much less than if it were a bug affecting RAID 5 volumes.
8. Re:Warning: RAID 0 by AthanasiusKircher · 2015-05-21 06:37 · Score: 1
  
  I'm going to go ahead and say that it mitigates the serious of the damage caused in actuality since most IT people entrusted with serious and important data aren't going to be that stupid.
  And that's where your assumptions are different from mine. I was discussing people who are probably NOT "entrusted with serious and important data," but nevertheless have their own personal data (which they think is at least somewhat valuable) and choose to run a RAID 0 setup because of some stupid reason, like it makes their system run a bit faster.
  (Well, that's not a completely stupid reason, but it is a reason to have a good backup strategy for essential files and to segregate your data so only the minimum files are at risk on RAID 0. Many people don't worry about these things until it's too late.)
  If you doubt such people exist, do an internet search or read some gamer forums. And given such people are more likely to be running a bleeding-edge new version of software than a IT pro with a server who does thorough stability testing before upgrades, I'd imagine that a bug like this will disproportionately affect those hobbyists.
  I'm not talking about IT pros here. I'm talking about random idiots who run RAID without thinking of the consequences. For them, this bug could be really serious.
9. Re:Warning: RAID 0 by nine-times · 2015-05-21 06:56 · Score: 1
  
  If you doubt such people exist, do an internet search or read some gamer forums.
  I think you missed my point. I don't doubt such people exist. I doubt such people are generally safeguarding information that I think is important.
10. Re:Warning: RAID 0 by TheCarp · 2015-05-21 07:14 · Score: 1
  
  Hard to say but it will happen eventually. I have seen it go a few years, then lose 2 within a few months. Always make sure monitoring works and will alert you if its degraded. You can run degraded mode for a long time without monitoring.....till the next one fails.
  They are mechanical, so manufacturing quality and environment will factor in. My drives likely see a lot of shake and heat being on the third floor of a 100 year old house, between the wind, the washing machine and seasonal heat.... its no data center in here.
  
  --
  "I opened my eyes, and everything went dark again"
11. Re:Warning: RAID 0 by Enry · 2015-05-21 07:30 · Score: 1
  
  I suspect not, since his point seemed to be that you shouldn't be using RAID 0 for data that you care about anyway.
  I meant, what if there was a bug in the RAID 5 code that caused similar corruption? This is equivalent (almost) to blaming the victim. Yes, you did risky behavior, but the problem wasn't caused because of the risky behavior.
12. Re:Warning: RAID 0 by Enry · 2015-05-21 07:33 · Score: 1
  
  I was really just throwing out drives and times. I had name-brand systems that were in a RAID 0 to consolidate two drives (the drive contents were expendable since this was just scratch space) and they ran for many years with few failures.
13. Re:Warning: RAID 0 by nine-times · 2015-05-21 07:43 · Score: 1
  
  I meant, what if there was a bug in the RAID 5 code that caused similar corruption?
  Yes, I understood. And I way saying, yes, it seems clear that we would all care more if it were a problem with RAID 5.
  I understand that you think "we would respond differently if this were RAID 5" is a sign of hypocrisy or something. But it's not really that.
  It's a little like saying, "There was a design flaw in trash cans that cause items stored in the trash can to be damaged." And people respond by saying, "Yeah, well... that's not great, but it could be worse. Things stored in trash cans are usually things nobody cares about anyway."
  And then you say, "Would you respond differently if this trash can problem were discovered in long-term storage bins?"
  And so the response is, "Yes, we would care more about that. Of course we would all care more about that. Because people probably care about things in long-term storage bins, and usually put trash in trash cans. I understand that someone somewhere may be storing their valuable family heirlooms in trash cans, but they probably shouldn't be doing that."
14. Re:Warning: RAID 0 by Enry · 2015-05-21 08:29 · Score: 1
  
  I understand that you think "we would respond differently if this were RAID 5" is a sign of hypocrisy or something. But it's not really that.
  Yes it is, and that's a very short sighted approach. I hope you're not a developer.
15. Re:Warning: RAID 0 by nine-times · 2015-05-21 09:05 · Score: 1
  
  As I said:
  
  It doesn't really make it ok for a bug to exist that destroys RAID 0 volumes, but it does mitigate the seriousness of the damage caused.
16. Re:Warning: RAID 0 by styrotech · 2015-05-21 09:58 · Score: 1
  
  Linux md RAID10 is a 'non standard' single level layout that does not have a RAID0 component/layer.
  There are 3 layouts available for it, one of which can mimic the underlying block layout of the 'standard' layered/nested RAID10.
17. Re: Warning: RAID 0 by AvitarX · 2015-05-21 13:34 · Score: 1
  
  This is an important concept. Corruption of temporary / high performance scratch data can have terrible consequences, when the loss may not be that big of a deal.
  I'm shocked at how people have trouble grasping that.
  
  --
  Wow, sent an e-mail as suggested when clicking on "use classic" banner, and got a fast response that addressed my msg
18. Re:Warning: RAID 0 by strikethree · 2015-05-21 23:32 · Score: 1
  
  Erm, uh... what about the situation where you are running 4 drives in RAID 0 but the pairs are RAID1?
  I keep forgetting if that is called RAID10 or RAID0+1. Meh. Regardless, RAID0 is not just used for unimportant data.
  
  --
  "Someone needs to talk to the tree of liberty about its ghoulish drinking problem." by ohnocitizen
19. Re:Warning: RAID 0 by nine-times · 2015-05-22 01:16 · Score: 1
  
  Yeah, that's RAID 10. If this bug affects RAID 10, then it's a bigger deal than if it only affects RAID 0.
Why ext4 by silas_moeckel · 2015-05-21 01:38 · Score: 2

If your running a brand spanky new kernel, with data you do not care about why an old FS. Plenty of newer better FS's to choose from.

--
No sir I dont like it.
1. Re:Why ext4 by pla · 2015-05-21 02:13 · Score: 1
  
  Ext4 should in theory be the best choice. It's widely used and has a large enterprise support. Lots of business people get angry if it does not work properly.
  
  On a modern system with multiple disks you want to configure as some variety of soft-RAID, ZFS hands-down counts as the clear best choice (short of going for a "cluster" FS). It allows an arbitrary number of extra parity drives (think "RAID 8"), as well as arbitrarily many hot spares; it quickly and easily recovers from having someone pull out all your drives, shuffle them around, and put them back in (for a more real-world version of that - Ever updated your BIOS only to find all your drives remapped?); it detects and (usually) corrects corrupted files; it supports online snapshotting and snapshot exporting to another; it uses dynamically sized storage pools rather than fixed "partitions", and can even grow the underlying total available space.
2. Re:Why ext4 by houstonbofh · 2015-05-21 02:18 · Score: 1
  
  Name one that actually boots the Linux kernel, and doesn't just run in user space. (Yes, I am a fan of ZFS, but not the Linux implementation.)
3. Re:Why ext4 by silas_moeckel · 2015-05-21 02:32 · Score: 1
  
  XFS for starters it's the default nowadays on rhel/centos.
  
  --
  No sir I dont like it.
4. Re:Why ext4 by Enry · 2015-05-21 02:38 · Score: 1
  
  You can't remove drives from a ZFS pool - once they're in (even if you have free space on other drives), the number of drives can't go down. Which really bothers me. With LVM you can evacuate data off of drives and shrink the pv. LVM in itself isn't a filesystem, but if you think of a pool as an LVM volume the functionality is somewhat similar.
5. Re:Why ext4 by Rich0 · 2015-05-21 02:47 · Score: 2
  
  The problem is that the feature-list for ZFS is very enterprise-oriented.
  Why would you want to add just one drive to a server with 5x 6-drive RAID6 arrays? Just add another 6 drives at a time.
  On the other hand, if you have a PC with 3 drives in RAID5, you could easily want to turn that into a 4-drive RAID5 or a 5-drive RAID6 in-place.
  Btrfs has a lot of features that are useful for smaller deployments, like being able to modify the equivalent of a vdev in-place. ZFS on the other hand has a lot of features like ZIL that are very useful for larger deployments.
6. Re:Why ext4 by Pope+Hagbard · 2015-05-21 02:49 · Score: 1
  
  It's newer than ext2, which ext4 is based on.
7. Re:Why ext4 by goarilla · 2015-05-21 03:43 · Score: 1
  
  Why would you want to add just one drive to a server with 5x 6-drive RAID6 arrays? Just add another 6 drives at a time.
  
  ZFS isn't ideal for growing like that since it doesn't do rebalancing. Your younger raid arrays will always have more data on them.
  Also zfs destroy is very expensive.
8. Re:Why ext4 by goarilla · 2015-05-21 03:53 · Score: 1
  
  *Your older raid arrays*
9. Re:Why ext4 by pla · 2015-05-21 04:01 · Score: 1
  
  I will readily admit that as a "shortcoming" of ZFS, but honestly, I don't quite see any obvious use cases for it. On the short term (months), I've only ever needed to *add* storage, never remove it.
  
  On the longer term (years), I have found that I go back and forth on how many drives I need, but when I do eventually upgrade my home NAS to bigger and better hardware, I don't even try to salvage old drives 1/10th the size of modern ones - I bring up the new system, with however many brand new drives I consider appropriate, and clone the old one to the new one.
10. Re:Why ext4 by fnj · 2015-05-21 04:01 · Score: 2, Informative
  
  Name one that actually boots the Linux kernel, and doesn't just run in user space. (Yes, I am a fan of ZFS, but not the Linux implementation.)
  You really should get out more. ZFS on Linux is not to be confused with the ZFS Fuse project. You can boot from a ZoL filesystem. In general ZoL is about as stable, complete, and reliable as any ZFS.
11. Re: Why ext4 by wed128 · 2015-05-21 04:34 · Score: 2
  
  ReiserFS predates ext4, and it's hard to be an active software developer in prison.
12. Re: Why ext4 by kthreadd · 2015-05-21 05:14 · Score: 1
  
  If the file system actually was great I'm sure someone else would pick it up, but I don't know if it was that great.
13. Re:Why ext4 by kthreadd · 2015-05-21 05:17 · Score: 1
  
  XFS isn't newer than ext4.
  Depends on how you look at. XFS is continuously developed and improved, they just don't stick a version number after it like the ext developers.
14. Re:Why ext4 by Rich0 · 2015-05-21 05:24 · Score: 1
  
  Why would you want to add just one drive to a server with 5x 6-drive RAID6 arrays? Just add another 6 drives at a time.
  ZFS isn't ideal for growing like that since it doesn't do rebalancing. Your younger raid arrays will always have more data on them.
  Also zfs destroy is very expensive.
  Perhaps, but my point was more that if you want to grow ZFS this is the ONLY way to actually do it, as far as I'm aware. You can't add individual drives to individual "vdevs."
15. Re:Why ext4 by goarilla · 2015-05-21 06:16 · Score: 1
  
  Well you can grow the underlying component devices of the vdev. But yes even the "perfect" ZFS has its weaknesses.
16. Re:Why ext4 by pla · 2015-05-21 07:36 · Score: 1
  
  Perhaps, but my point was more that if you want to grow ZFS this is the ONLY way to actually do it, as far as I'm aware. You can't add individual drives to individual "vdevs."
  
  You can replace all the drives in the array with bigger ones, resilvering after each replacement, and when you get to the last one, poof, you magically have a bigger pool. I certainly won't claim that as terribly efficient, though. :)
  
  It has its shortcomings, no doubt. But compared to old-school RAID or even LVM, it takes a huge step forward.
17. Re:Why ext4 by Rich0 · 2015-05-21 14:29 · Score: 1
  
  Sure, but with btrfs you can just add one drive and sometimes get its entire capacity added to your array - it works fine with mixed-size disks.
  Of course, it might just decide not to boot the next day, and that is the downside to btrfs. It does tend to be a bit more friendly in scenarios where you have a small number of disks, though, which was my main point.
18. Re:Why ext4 by Rich0 · 2015-05-21 14:37 · Score: 1
  
  Agree, as the other reply pointed out as well. And you can do the same with mdadm raid too (though obviously with none of the benefits btrfs/zfs bring for data integrity like checksumming and copy-on-write). Mdadm will also let you reshape an array in place (that is change raid levels or number of disks), though with mdadm that will often result in messing up your stripe alignment and of course it is more likely to eat your data if something goes wrong since if it finds a parity mismatch it has no way to know which copy is bad.
  I was just commenting that btrfs tends to have a lot of features that appeal to small system users that you'll actually find missing on zfs, even if it is far less mature overall, and lacking in many enterprise-scale features. It just reflects the emphasis of the developers behind it.
  I really can't complain about zfs - it is a great filesystem. However, things like not being able to reshape an array or mix disk sizes in an array are some of the things that hold me back from adopting it. Heck, btrfs will let you switch from raid1 to raid5 without touching any of the data already written - newly-allocated chunks will use raid5 and existing chunks will continue to use raid1 - it doesn't manage arrays at the whole-device level. In practice though you're likely to tell it to rebalance your data of course.
19. Re:Why ext4 by sjames · 2015-05-21 15:09 · Score: 1
  
  You're thinking of the ZFS that goes through FUSE. There is also ZFS on Linux that runs as kernel modules like any other fs.
  There's also btrfs.
  Of course, neither of those needs the md driver at all, they have their own raid like systems.
20. Re:Why ext4 by sjames · 2015-05-21 16:16 · Score: 1
  
  I'm using ZFS in production for now but I'm actively testing btrfs for that reason among others.
21. Re:Why ext4 by houstonbofh · 2015-05-21 16:17 · Score: 2
  
  If you trace it back, all of that fear originates on one post from the freenas forums. A post from one of the key developers says that you should use ecc for any server with critical data, but zfs is neither more or less sensitive to it.
22. Re:Why ext4 by sjames · 2015-05-21 16:31 · Score: 1
  
  In a production server, I can see value in stepwise evacuating old drives and then swapping them for new larger drives only once the data is stable on the filesystem. Done right you could pull it off with zero downtime without opening a window where a single failure brings you down.
23. Re: Why ext4 by sjames · 2015-05-21 16:32 · Score: 1
  
  It had some REALLY ugly failure modes.
24. Re:Why ext4 by jabuzz · 2015-05-21 22:40 · Score: 1
  
  If you really care about the checksumming then switch to SAS/FC and use the Data Integrity Field. ZFS fixes something that has already been fixed years before it even came into existence. It offers better protection from silent corruption than ZFS does as well, because with ZFS you have no guarantee that what you send to the disk is what actually gets written, DIF mitigates against that.
25. Re:Why ext4 by Rich0 · 2015-05-22 01:28 · Score: 1
  
  Somehow I doubt that is going to work on my $70 Seagate 7200RPM hard drive plugged into my $50 motherboard or $20 SATA controller card. However, by all means let me know if it will.
  Plus, COW filesystems offer a lot more than just data checksumming.
26. Re:Why ext4 by goarilla · 2015-05-22 06:37 · Score: 1
  
  So DIF is both an on disk checksum and an inflight error check like UDMA CRC ?
27. Re:Why ext4 by goarilla · 2015-05-22 06:46 · Score: 1
  
  Also the wikipedia article https://en.wikipedia.org/wiki/... makes it look like it's a failed/superseded technology. Is it still being used ?
28. Re: Why ext4 by drinkypoo · 2015-05-22 08:57 · Score: 1
  
  It was great... until the other filesystems caught up while it was not under development.
  It was great... until it went on a rampage and murdered your data. I kid, but I'm also serious. When it was in current development, no other fs was as efficient with small files, and there's a lot of those on the average Unix system so that's of great interest. But it also was the least reliable filesystem in common use. So it was really never worth using.
  
  --
  "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
29. Re: Why ext4 by vilanye · 2015-05-22 09:02 · Score: 1
  
  That should be fixed.
  Sadly, there are a lot of skilled and educated people in prison just rotting away. They could still be very useful and productive.
30. Re: Why ext4 by wed128 · 2015-05-26 02:27 · Score: 1
  
  Part of what makes prison a punishment is that you're disallowed to be productive...
31. Re: Why ext4 by vilanye · 2015-05-26 11:41 · Score: 1
  
  That is stupid and wasteful.
  Prison is supposed to be rehabilitation, treating it as punishment is counter-productive for both society and the inmate.
Two issues in play? by jones_supa · 2015-05-21 01:39 · Score: 1

There seems to be a fix in RAID code and a fix in Ext4 code.
The latter was incorporated in Linux 4.0.3 (changelog), and according to the Phoronix article the RAID bug is still unfixed.
New version ... by JasterBobaMereel · 2015-05-21 01:40 · Score: 5, Insightful

This is the new 4.0 kernel, A Major version update , less than a month old, that most Linux systems will not have yet ...and the issue has already been patched
Bleeding edge builds get what they expect, stable builds don't even notice

--
Puteulanus fenestra mortis
1. Re:New version ... by Anonymous Coward · 2015-05-21 02:10 · Score: 1
  
  Since this was an undiscovered until now, you also have to consider how far back this bug goes. Apparently, it appeared in 3.14-rc1. For some people, that's significant.
2. Re:New version ... by Anonymous Coward · 2015-05-21 02:14 · Score: 2, Insightful
  
  The last major Linux version update that actually meant something was 1->2. The "major version" bumps in the kernel are now basically just Linus arbitrarily renumbering a release. The workflow no longer has a notion of the next major version.
3. Re:New version ... by houstonbofh · 2015-05-21 02:20 · Score: 1
  
  The down side is that since no one runs business critical loads on new stuff, business critical tools do not get tested as well as simple stuff.
4. Re:New version ... by torqer · 2015-05-21 02:46 · Score: 1
  
  Agreed... Tovalds having a google+ poll for changing major version numbers is as arbitrary as not having windows 9.
  https://plus.google.com/+Linus...
5. Re:New version ... by jedidiah · 2015-05-21 03:52 · Score: 1
  
  No. They just don't run PRODUCTION on the bleeding edge code. That doesn't mean that this stuff isn't being tested with non-trivial use cases. Any reputable IT shop is going to be putting version n+1 through it's paces before it does anything important because everyone wants to keep their jobs.
  The last time I used RAID0 for anything it was a high volume R&D project. The OS vendor probably got a couple of good bug fixes out of us.
  
  --
  A Pirate and a Puritan look the same on a balance sheet.
6. Re:New version ... by Anonymous Coward · 2015-05-21 04:20 · Score: 1
  
  Skipping Windows 9 is not arbitrary. They do that because some way old java stuff out there would think it's Windows 95 or 98 because it does some shitty version string parsing.
7. Re:New version ... by Yunzil · 2015-05-21 05:45 · Score: 1
  
  Uh, 4.0 is a stable build, chief.
8. Re:New version ... by F.Ultra · 2015-05-21 06:34 · Score: 1
  
  No it wasn't. The patch that caused this problem was a fix to another problem that where introduced in 3.14-rc1.
9. Re:New version ... by Yunzil · 2015-05-21 07:01 · Score: 1
  
  kernel.org:
  stable: 4.0.4 2015-05-17
10. Re:New version ... by KingMotley · 2015-05-21 08:08 · Score: 1
  
  I'll wait for 4.1, and then I'll wait for 4.1.2 just to be safe.
11. Re:New version ... by sound+vision · 2015-05-21 15:20 · Score: 1
  
  And which stable distros are using 4.0? My Debian stable box that gets updated at least weekly is on 3.2.
12. Re:New version ... by houstonbofh · 2015-05-21 16:18 · Score: 1
  
  Most places I know do not have identical hardware for testing. They have retired production hardware for testing, so it is older stuff, with older drivers.
13. Re:New version ... by vilanye · 2015-05-22 09:15 · Score: 1
  
  "Stable" distros like Debian and redhat patch in a lot of bug fixes and other things into their kernel, so there is no real sense of where they are feature and bugfix-wise.
  3.2 on Debian is not the same kernel as a distro that ran 3.2 when 3.2 was the current kernel.
  Those "stable" distros crack me up, the kernel and almost all of its packages are significantly behind and they are no more stable than my opensuse 13.2 servers that have been upgraded in place since 11.3, but this is a rant for another topic.
14. Re:New version ... by petermgreen · 2015-05-28 01:27 · Score: 1
  
  My Debian stable box that gets updated at least weekly is on 3.2.
  Your box is now a Debian oldstable box.
  Debian stable (jessie) has 3.16, debian testing (stretch) also has 3.16 right now though i'd expect it to get 4.0 in the not too distant future. Debian unstable has 4.0.
  
  --
  note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
Avoid even-number OS major releases by Anonymous Coward · 2015-05-21 01:43 · Score: 1

That's a good rule of thumb for Windows and Linux. Not sure about Apple :)
1. Re:Avoid even-number OS major releases by houstonbofh · 2015-05-21 02:22 · Score: 1
  
  And 2.6 was the worst since both numbers were even!
2. Re:Avoid even-number OS major releases by Penguinisto · 2015-05-21 02:39 · Score: 1
  
  That's a good rule of thumb for Windows and Linux. Not sure about Apple :)
  On the Apple side, the rule seems to be an iOS-only thing (and even then only recently... thanks iOS 8!)
  On the OSX side? 10.0 sucked pretty hard, and (IIRC) 10.2 had some problems, but it's been rather rock-stable since then (at least from my POV - I've used OSX from the ill-fated 10.0 all the way up to Yosemite, but YMMV).
  
  --
  Quo usque tandem abutere, Nimbus, patientia nostra?
From the Article... by Rashkae · 2015-05-21 01:56 · Score: 1

It also looks like if dropping the discard mount option you will also avoid being hit by this serious issue.
There's very little good reason to use 'discard' on Linux, and many reasons not to. (This isn't the first data corruption problem, and there are several performance issues as well.) Fstrim in a con job is the way to go.
1. Re:From the Article... by Rashkae · 2015-05-21 01:58 · Score: 1
  
  Having said that, considering the nature of this bug, I wouldn't be surprised is using fstrim would also trigger this particular bug.
2. Re:From the Article... by unrtst · 2015-05-21 02:18 · Score: 1
  
  Good thing all those 'eyes on the source code' caught this little nasty before it went out the door.
  Name one distro where this is out the door.
  For example, the latest ubuntu, released just a month ago, is using 3.19.
3. Re:From the Article... by FredK · 2015-05-21 02:23 · Score: 1
  
  Using 4.0 on Gentoo for about a week now.
4. Re:From the Article... by marsu_k · 2015-05-21 02:32 · Score: 1
  
  There's very little good reason to use 'discard' on Linux
  Care to elaborate on that? My bible says that discard is the first choice, fstrim when that isn't applicable for whatever reason. Bear in mind that I use Linux mostly as a desktop OS, so whatever caveats there may be in server use do not affect me.
5. Re:From the Article... by marsu_k · 2015-05-21 02:35 · Score: 1
  
  4.0.2-1-ARCH #1 SMP PREEMPT Thu May 7 06:47:54 CEST 2015 x86_64 GNU/Linux
  
  Arch, at least.
6. Re:From the Article... by present_arms · 2015-05-21 02:37 · Score: 1
  
  uname -r 4.0.4-pclos1
  
  --
  http://chimpbox.us
7. Re:From the Article... by Rashkae · 2015-05-21 03:03 · Score: 1
  
  This is the first time I've found someone suggesting discard as the first choice over fstrim. The reasons to use fstrim is stated right in that article. Performance bottlenecks when there are file delete opperations. (And no real benefit to trimming on the fly vs trimming in a batch process.) However, while I usually have nothing against debaing my betters and making a spectacular fool of myself, I'm not going to go out of my way to contractict the Arch Linux documentation.
8. Re:From the Article... by marsu_k · 2015-05-21 03:22 · Score: 1
  
  Oh, I'm not saying Arch Wiki is infallible (although, it is correct pretty much the whole time). I was just looking for rationalization to discard or not to discard. As a personal anecdote, this Zenbook has been running discard since day 1 (24GB SSD and 500GB HDD, discard on the first drive only of course) - the OS partition (the 24GB drive, ext4) is still spanking fast. Although, it has never been close to running out of space (/var is on the HDD).
9. Re:From the Article... by unrtst · 2015-05-21 06:06 · Score: 1
  
  Ok, I stand corrected.
  Still, not many (I don't count gentoo, as that's just whatever you compile; and unstable (ex. debian unstable) shouldn't count either).
  Running down the list in distrowatch:
  mint: 3.13
  ubuntu: 3.19.0
  debian (stable/testing): 3.16.7
  mageia: 3.19.8
  fedora: 3.17.4
  opensuse: 3.16.6
  arch: 4.0.4
  centos: 3.10
  pclinuxos 2014.12: 3.18.1
  slackware: 3.18.11
  freebsd: ... it's freebsd, not linux ...
  So, out of the top 11 (I don't know why freebsd is even on there), arch is the only one whose current release is on 4.x.
10. Re:From the Article... by marsu_k · 2015-05-21 06:17 · Score: 1
  
  If that is your criteria, then perhaps Arch shouldn't count either - sure, it is not a source-based distro (the only package I compile frequently is Firefox, and that is due to it being the KDE-friendly fork, kudos to OpenSUSE for that), but still very much bleeding edge. Remarkably stable at that, but comparable to to Debian Unstable.
11. Re:From the Article... by stooo · 2015-05-21 08:50 · Score: 1
  
  >> I hope there are lawsuits from people who lose their valuable data from this.
  ????? is there a law against bugs in the US ?
  can you accuse source code ?
  
  --
  aaaaaaa
Bob Wilson knows they're real! by Thud457 · 2015-05-21 02:51 · Score: 1

Way to go Barney Buzzkill!
Next you're gonna try to tell us that gremlins aren't real.

--
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
Re:Happy Corruption Day from The Golden Girls! by Anonymous Coward · 2015-05-21 03:56 · Score: 1

Thank you for feeding the troll
Took the bait and back again
Your brain is weak, you're a doof and a space cadet.

And if they threw a inquest
Invited everyone you blew
You would see the biggest sore would be from me
And the verdict would say, thank you for feeding the troll.
Ahh there it is by drinkypoo · 2015-05-21 04:13 · Score: 2

Tunneled down into the articles, http://git.neil.brown.name/?p=... has the patch. I'm building a system with 4.0.4 right now so this was material to me

--
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
1. Re:Ahh there it is by TeknoHog · 2015-05-21 05:15 · Score: 1
  
  I'm building a system with 4.0.4 right now
  Where's that? For some reason I can't find that version anywhere.
  
  --
  Escher was the first MC and Giger invented the HR department.
2. Re:Ahh there it is by TeknoHog · 2015-05-21 20:07 · Score: 1
  
  I love replies like this, especially the whooshing sound they make as they go by.
  
  --
  Escher was the first MC and Giger invented the HR department.
Or just use a power of 2 chunk size? by tlambert · 2015-05-21 04:16 · Score: 3, Insightful

Or just use a power of 2 chunk size?
What idiot configuration did someone have to have to trigger this bug?
1. Re:Or just use a power of 2 chunk size? by samwichse · 2015-05-21 23:58 · Score: 1
  
  But I always use 315 for my chunk size because that's my grandma's birthday!
Re:Happy Corruption Day from The Golden Girls! by lister+king+of+smeg · 2015-05-21 04:45 · Score: 1

best response to troll feeding ever

--
---Saying gnome 3 is better than windows 8 not so much a compliment as it is damning with light praise.
Re:Woot! I pissed off the fucktarded shitdot sheep by tshawkins · 2015-05-21 05:01 · Score: 1

You need to feed him MOAR BRAINZZ
Raid kills bugs dead! by TeknoHog · 2015-05-21 05:46 · Score: 5, Funny

Well, there goes that slogan.

--
Escher was the first MC and Giger invented the HR department.
1. Re:Raid kills bugs dead! by toddestan · 2015-05-23 09:11 · Score: 1
  
  In Soviet Russia, bug kills raid!
In particular, NO redundancy. Reliability drops. by Ungrounded+Lightning · 2015-05-21 06:05 · Score: 5, Informative

Losing data goes with the territory if you're going to use RAID 0.
In particular, RAID 0 combines disks with no redundancy. It's JUST about capacity and speed, striping the data across several drives on several controllers, so it comes at you faster when you read it and gets shoved out faster when you write it. RAID 0 doesn't even have a parity disk to allow you to recover from failure of one drive or loss of one sector.
That means the failure rate is WORSE than that of an individual disk. If any of the combined disks fails, the total array fails.
(Of course it's still worse if a software bug injects additional failures. B-b But don't assume, because "there's a RAID 0 corruption bug", that there is ANY problem with the similarly-named, but utterly distinct, higher-level RAID configurations which are directed toward reliability, rather than ONLY raw speed and capacity.)

--
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
Re:Happy Corruption Day from The Golden Girls! by Darinbob · 2015-05-21 08:07 · Score: 1

I thought it was "commandante".
Re:Raid0 and ext4 is a bad combination by twistedcubic · 2015-05-22 05:39 · Score: 1

This is not true.
Bleeding edgy by ShreddingSplinters · 2015-05-22 08:25 · Score: 1

Since when did bug reports become viral news?
Re:Linux has become like Windows by vilanye · 2015-05-22 09:25 · Score: 1

Not even a decent troll.
Running opensuse 13.2 on desktop and server, no crashes and with KDE with all the bells and whistles it still comes in under 400 MB of RAM when the boot process is completed.
Sure the kernel sources are getting large, but it supports a ton of hardware on several processor architectures.
Talking about "the file system" and "the desktop" not only shows you are in fact a troll, but displays your total ignorance of Linux.