Slashdot Mirror


What's the Damage? Measuring fsck Under XFS and Ext4 On Big Storage

An anonymous reader writes "Enterprise Storage Forum's long-awaited Linux file system Fsck testing is finally complete. Find out just how bad the Linux file system scaling problem really is."

26 of 196 comments (clear)

  1. fsck speed, want safety by Anonymous Coward · · Score: 3, Insightful

    How fast a full fsck scan is is my last concern. What about how successful they are at recovering the filesystem?

    1. Re:fsck speed, want safety by h4rr4r · · Score: 5, Insightful

      If you need to fsck you should already be restoring from backups onto another machine.

    2. Re:fsck speed, want safety by rickb928 · · Score: 4, Insightful

      More helpful advice from the Linux community. Thank you ever so much, once again right on point, timely, and effective.

      --
      deleting the extra space after periods so i can stay relevant, yeah.
    3. Re:fsck speed, want safety by h4rr4r · · Score: 3, Interesting

      Because sometimes it does work. Relying on any such software is stupid.

      While the FSCK/CHKDSK runs you restore onto another machine. This way if the check finishes first, you can use it until you can switch over to the restored machine. It also can save your ass if you are not smart enough/fortunate enough to have good backups.

    4. Re:fsck speed, want safety by darkpixel2k · · Score: 5, Funny

      when I need to fsck, I just call my girlfriend

      Why? Do you not know how to use the command line?

      --
      There's no place like ::1 (I've completed my transition to IPv6)
    5. Re:fsck speed, want safety by hackstraw · · Score: 5, Interesting

      The largest filesystem I admin is just shy of 1/2 petabyte. And its one in number. Backing up everything on that filesystem is simply not feasible. To put it in perspective 1 stream @ 200 MiB/s would take almost 28 days to backup the whole thing. I would imagine a restore would take about the same order. Telling hundreds of users their files are unavailable for reading or writing for 30 days is not really an option, so I run fsck.

      Backups simply are not really an option past 20+ terabytes of storage, and simply not feasible if the storage is volatile in nature. AFAIK everyone has gone to redundancy over backups at scale.

    6. Re:fsck speed, want safety by chuckymonkey · · Score: 5, Insightful

      You're fairly wrong there, you can actually back that much data up. You just have to be willing to pay for some seriously large tape libraries and they're not cheap. We're in the process of installing a 700TB array with a 1.5PB tape library backup. You just have to do the backups using filesystem snapshots and run them pretty much constantly.

      --
      "Some books contain the machinery required to create and sustain universes."-Tycho
    7. Re:fsck speed, want safety by phorm · · Score: 3, Insightful

      If you're in a scenario where "Backups are not really an option", somebody is doing something wrong...

      How long did it take you to get to 0.5PB? If you use a differential backup/sync, then you should generally only need to copy *NEW* data, and the old stuff will already be there.

    8. Re:fsck speed, want safety by tlhIngan · · Score: 4, Interesting

      The largest filesystem I admin is just shy of 1/2 petabyte. And its one in number. Backing up everything on that filesystem is simply not feasible. To put it in perspective 1 stream @ 200 MiB/s would take almost 28 days to backup the whole thing. I would imagine a restore would take about the same order. Telling hundreds of users their files are unavailable for reading or writing for 30 days is not really an option, so I run fsck.

      Which means You're Doing It Wrong(tm).

      Two words: volume snapshot.

      What it does is give you a view of the filesystem as it exists at that the time the snapshot is taken. The frozen image is mounted in another mountpoint (read-only), while the snapshotted voume is still accessible (read-write). Changes to the volume since the snapshot was taken won't be in the snapshot (obviously).

      Your backup points to that snapshot which won't change and that's copied to tape. Once you're done backing up 30 days later, you delete the snapshot.

      Since your backup takes so long, you'd immediately then make another snapshot and being the backup again.

      If it's a database, the database backup tools work on a database snapshot - it will be correct and consistent as of when the snapshot was taken while the database remains available for reading and writing outside of the snapshot.

      Having to take a system down to back it up is a dead concept on modern OSes as they all tend to have snapshot capability.

    9. Re:fsck speed, want safety by _LORAX_ · · Score: 4, Informative

      Backups simply are not really an option past 20+ terabytes of storage, and simply not feasible if the storage is volatile in nature. AFAIK everyone has gone to redundancy over backups at scale.

      200TB/130TB usable clustered/distributed system with 4x LTO5 drives and we do a full snapshot to tape every week. With data that size you either pay up-front for proper engineering or you pay for the life of the system for poor performance and eventual cleanup of the mess.

    10. Re:fsck speed, want safety by lvxferre · · Score: 5, Funny

      Protip: if 'make love' returns no target, you need to do the job by hand.

      --
      Nerdy news for your nerdy needs? http://www.soylentnews.org Soylent News is people!
    11. Re:fsck speed, want safety by h4rr4r · · Score: 3, Insightful

      Most people are worried more about cost then reliability.

      Most people is often a category that does not do things the best way or the right way.

    12. Re:fsck speed, want safety by chuckymonkey · · Score: 3, Insightful

      I know I'm posting to an AC here, but I want to point something out. "Backups simply are not really an option past 20+ terabytes of storage, and simply not feasible if the storage is volatile in nature." He was claiming that it's not feasible to back up more than 20+ TB of storage when in fact it is. I was pointing out that yes you can, but it's pretty expensive.

      --
      "Some books contain the machinery required to create and sustain universes."-Tycho
  2. fsck xfs does something? by drewstah · · Score: 3, Interesting

    When I had some EBS problems a couple years ago, I figured I would run xfs_check. It seemed to do absolutely nothing, even if there were disks known to be bad in the md array. xfs is nice and fast, but I haven't seen the xfs_check or xfs_repair to do either of the things I'd assume they'd do -- check and repair. I found it easier to delete the volumes and start from scratch, because any compromised xfs filesystem seems to be totally unfixable. Is fsck for xfs new?

    --
    I do stuff Zhrodague
    1. Re:fsck xfs does something? by Sipper · · Score: 3, Informative

      I set up an xfs volume a couple years back. After copying a few files over nfs, it became corrupted. the xfs fsck did something -- it told me that it was so corrupted, it couldn't be fixed.

      I think you mean xfs_repair. On XFS, fsck is a no-op.

      I've never yet seen xfs_repair tell me there was an issue it couldn't fix -- that sounds unusual. However there have been lots of changes to XFS in the Linux kernel in recent years, and occasionally there has been a few nasty bugs, some of which I ran into. Linux-2.6.19 in particular had some nasty XFS filesystem corruption bugs.

  3. Damage? by eggstasy · · Score: 3, Funny

    Honey badger don't give a fsck.

  4. Re:linux is fail by gweihir · · Score: 4, Insightful

    A cranky coward from the shadows is not s reliable source of information.

    I have used AIX and Solaris, and I can say that a lot of stuff is easier on Linux.

    --
    Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
  5. Re:linux is fail by Anonymous Coward · · Score: 5, Funny
    sudo kill yourself

    ;-)

  6. Re:Why bother? by _LORAX_ · · Score: 3, Interesting

    Our BTRFS evaluation resulted in rejecting it for some very serious problems ( what they claim are snapshots are actually clones, panic in low memory situations, no fsck, horrible support tools, developers who are hostile to criticism, pre-release software, ... ). ZFS was nice, but limited to non-distributed systems and still had a non-trivial amount of volume and backend management headaches. Personally I use ZFS for my personal servers at home ( incremental snapshots are the bomb ) but out production systems needed more.

  7. Re:linux is fail by ifrag · · Score: 3, Funny

    I like how you completely ignored Solaris yet still presented the comment as if it was a valid counterargument.

    I also like how GP completely ignored Solaris. I just like the fact it is being ignored.

    --
    Fear is the mind killer.
  8. Re:Why bother? by Guspaz · · Score: 3, Interesting

    ZFS now runs pretty well on Linux too, as a kernel module, thanks to zfsonlinux. If you're running a Debian-based distro, installing it is trivial (one command to add the PPA, one command to install the package).

  9. Re:linux is fail by aix+tom · · Score: 4, Informative

    You see my nick?

    AIX sucks more than Linux.

    Usual process for "weird"* AIX Problems:

    1) weird problem occurs after install. You report problem to IBM.
    2) IBM asks for your software version, see they are the newest ones available, and say they look into it.
    3) You ask several month later if they did find anything. They ask for your software version, they ask you to upgrade and see if the problem goes away.
    4) You upgrade to newest version.
    5) go to 2)

    *There are of course non-weird problems where you get the answer from IBM support in 2-3 days, and from Linux forums in 2-3 minutes.

  10. Re:linux is fail by lvxferre · · Score: 3, Funny

    Why would you replace a zero-ed string with another? At least use /dev/random, bro.

    --
    Nerdy news for your nerdy needs? http://www.soylentnews.org Soylent News is people!
  11. Re:linux is fail by Saxophonist · · Score: 3, Funny

    No, you're thinking of ReiserFS.

  12. Re:linux is fail by jd · · Score: 3, Interesting

    Works best if you use the "Doom as Sys Admin" hack.

    --
    It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
  13. Re:linux is fail by jd · · Score: 5, Interesting

    A lot of stuff is also faster on Linux, particularly on the x86. Solaris x86 is dog slow. AIX ("aches") is an appropriate name for a mainframe OS that never really got the hang of this new-fangled "interactive user" stuff. It's a good mainframe OS, that is what it is designed for, tuned for and intended for, but traditional mainframe batch transactional work isn't the sort of payload that is typically run these days. The high-end users want hard real-time (i.e.: they know to the microsecond - or nanosecond, in some cases - exactly when each process will start and stop) for data collection, data analysis and simulation. The data centers want massive multithreading for gigantic servers with minimal overhead and service guarantees per thread. The typical user wants extremely low latency interactive. None of these are pre-scripted batch jobs.

    Now, if you wanted to develop a data warehouse for, say, technical writings, journalism, etc, where you're compiling a collection of things that can be typeset overnight, that may be doable as a batch job. However, anyone planning on publishing a journal that needs 72 terabytes of storage had best consider the marketplace a little more closely first. A publishing company, say Nature, might conceivably have use for AIX for batch work. I could see the number of submissions, referee responses and article selections per journal being such that a mainframe would be a perfectly valid way to do things. Even then, it might still be sufficiently small that a live transactional database would be more cost-effective.

    Traditionally, batch processing has been a niche market for electrical and gas companies, etc, where the number of customers is staggering. Even then, it has largely been replaced with live transactional systems because customers want things adjusted NOW and not overnight or at the end of the week.

    Mass mailers still use batch processing, but printing is the bottleneck and there is no point in having an expensive OS process everything in a fraction of a second on an expensive mainframe when it takes N actual real-world seconds before a printer becomes available to take the next block of data. You need run no faster than the slowest component because the end produce won't be delivered any faster. You would have to have a gigantic number of printers before the OS became a significant factor and most shops just don't have that kind of printing power.

    --
    It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)