Slashdot Mirror


Apple Introduces New File System AFPS With Tons Of 'Solid' Features (apple.com)

On the sidelines of its Worldwide Developer's Conference, Apple also quietly unveiled a new file system dubbed APFS (Apple File System). Here's how the company describes it: HFS+ and its predecessor HFS are more than 30 years old. These file systems were developed in an era of floppy disks and spinning hard drives, where file sizes were calculated in kilobytes or megabytes. Today, solid-state drives store millions of files, accounting for gigabytes or terabytes of data. There is now also a greater importance placed on keeping sensitive information secure and safe from prying eyes. A new file system is needed to meet the current needs of Apple products, and support new technologies for decades to come.Ars Technica dived into the documentation to find that APFS comes with a range of "solid" features including support for 64-bit inode numbering, and improved granularity of object time-stamping. "APFS supports nanosecond time stamp granularity rather than the 1-second time stamp granularity in HFS+." It also supports copy-on-write metadata scheme which aims to ensure that file system commits and writes to the file system journal stay in sync even if "something happens during the write -- like if the system loses power." The new file system offers an improvement over Apple's previous full-disk encryption File Vault application. It also features Snapshots (that lets you throw off a read-only instant of a file system at any given point in time), and Clones. According to the documentation, APFS can create file or directory clones -- and like a proper next-generation file system, it does so instantly, rather than having to wait for data to be copied. From the report: Also interesting is the concept of "space sharing," where multiple volumes can be created out of the same chunk of underlying physical space. This sounds on first glance a lot like enterprise-style thin provisioning, where you can do things like create four 1TB volumes on a single 1TB disk, and each volume grows as space is added to it. You can add physical storage to keep up with the volume's growth without having to resize the logical volume.As the documentation notes, things are in early stage, so it might take a while before AFPS becomes available to general users.

38 of 295 comments (clear)

  1. If Swift is any guide... by JoeyRox · · Score: 4, Funny

    This new filesystem should become stable in about 2028.

    1. Re:If Swift is any guide... by PopeRatzo · · Score: 3, Funny

      1. Your username screams "fanboi"

      His first choice for username was "TimCooksTastyBottom" but Apple's board of directors didn't think that was an appropriate Slashdot username for their CEO, so they settled on "macs4all"..

      --
      You are welcome on my lawn.
    2. Re:If Swift is any guide... by TheRaven64 · · Score: 2

      The general rule of thumb is 10 years between first release that adventurous users trust and solid enough for serious use on important data. ZFS past that point last year. BTRFS will in another three years (or 7 years, depending on how you count). The only exceptions are things like UFS2 and vFAT that are simple extensions to existing filesystems and embedded filesystems that are very simple.

      --
      I am TheRaven on Soylent News
    3. Re:If Swift is any guide... by peragrin · · Score: 3, Informative

      The question becomes is how long has this new fs been under development at Apple? Apple is well known for designing and testing software in the background and only announce it after it has reached a staple point. So it could be 3-5 years in development already.

      I can see it rolling out to Mac users with the next is and bieng default in just 2-3 years.

      --
      i thought once I was found, but it was only a dream.
    4. Re:If Swift is any guide... by macs4all · · Score: 3, Informative

      This is just a guess, but the reason most people call you a "fanboi" is because of 2 things:

      1. Your username screams "fanboi" 2. Every single one of your posts promotes macs and apple

      Like I said, just a guess, but, it just could be your fault.

      Anyone who thinks that the gentle wish connoted by my Username is cause for the amount of ad hominem abuse I have received is sadly lacking in online etiquette.

      I actually make many posts on Slashdot that have nothing to do with Apple. Depends on the Thread.

    5. Re:If Swift is any guide... by dgatwood · · Score: 4, Interesting

      So it could be 3-5 years in development already.

      I suspect it has been in progress since Apple's decision to pull ZFS, which offered many of these features... and possibly longer. That's way more than 5 years; in fact, next year (the expected release year), it will have been the ten years that the GP says makes a filesystem trustworthy.

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    6. Re:If Swift is any guide... by macs4all · · Score: 4, Insightful

      Gentle wish? Fuck your gentle wish. You're happy using unjustly overpriced SHIT because you're a dumbass. The rest of us who have more knowledge and experience then you'll ever have in your whole life, know better.

      Been designing computer hardware and software since 1976.

      Fluent in dozens (literally) of Assembly-languages from 6502 to ARM7 TDMI, plus C, PHP, HTML and several BASIC variants. Never did like C++ or Java, though...

      Paid Embedded Developer (hardware and software) for nearly 40 years, with a specialty in R&D of industrial Real-Time measurement and control PRODUCTS.

      Currently Develop Windows ERP Applications.

      Certified MS SQL Server Admin.

      The list goes on...

      Yeah. I'm a dumbass alright.

      STFU.

      I like Apple equipment and OSes precisely BECAUSE I got all that "Work ON my computer" shit out of my system 30 frickin' YEARS ago.

      Apple stuff isn't overpriced; because my time (and frustration) is actually WORTH something.

  2. Compression by paulhar · · Score: 3, Insightful

    C'mon, it's 2016. Where is compression?

    1. Re:Compression by Anonymous Coward · · Score: 2, Informative

      because it's 2016 and disk compression isn't necessary for everyday use. You have inordinately cheap disk, and performance far outweighs the need for compression. Sure, you could find lots of value in compression.... and you can get it with file compression utilities. Any compression algorithm that would give anything better than "average" couldn't be stream oriented and would therefore likely kill performance.
      Yes, it could be done. But is it needed? Nope.

    2. Re:Compression by mlts · · Score: 2

      I wouldn't mind deduplication either.

    3. Re:Compression by paulhar · · Score: 2

      I disagree, as ZFS demonstrates >500MB/sec to a compressed filesystem is very achievable, including random I/O access.

      FIle compression utilities don't work well with virtual machines. You can't just start the VM in a zip file...

    4. Re:Compression by macs4all · · Score: 2, Informative

      C'mon, it's 2016. Where is compression?

      Well, it has been part of HFS+ since Snow Leopard (2005). Where have you been?

      So, I would imagine that the new FS will support it as well.

    5. Re: Compression by RLaager · · Score: 2

      I said lz4, not gzip. The difference is massive.

    6. Re:Compression by ArchieBunker · · Score: 5, Insightful

      What kinds of files are people generating today? Pictures and video. What kinds of files are already compressed to begin with? Pictures and video. Compression doesn't make sense unless you have massive amounts of text or database files.

      --
      Only the State obtains its revenue by coercion. - Murray Rothbard
    7. Re: Compression by rl117 · · Score: 3, Insightful
      It depends entirely upon the type of data. See the examples below. Some data gives a little compression; there's a lot of binary data in my homedir which doesn't compress too well. On the other hand, source code, system logs and mail can compress superbly.

      Regarding the "performance penalty", it's generally going to be positive and improve performance. We are talking lz4 here, not gzip/bzip2/xz. It's fast, trading a lower compression ratio for performance. It can compress and decompress blocks in parallel. It can do this much faster than it can read data from disk, so you'll actually improve read and write speeds. And this is on top of ZFS being able to pull data of multiple spindles as the data is distributed over multiple zvols, with redundant copies of data, etc. It's likely not the penalty you think it is. It does multiple rounds of lightweight lz4 compression to reduce the entropy, and it bails out early if poorly compressible.

      % zfs get refcompressratio red/home/rleigh system/usr/ports system/usr/src system/var/log system/var/mail
      NAME PROPERTY VALUE SOURCE
      red/home/rleigh refcompressratio 1.33x -
      system/usr/ports refcompressratio 1.60x -
      system/usr/src refcompressratio 2.16x -
      system/var/log refcompressratio 6.54x -
      system/var/mail refcompressratio 4.99x -

      With compression like this, you no longer need to bother compressing rotated logs. And while the homedir compression is small in comparison, it's gained me an extra 100GiB just for this single dataset, which is not to be sneezed at.

  3. Good Luck by bill_mcgonigle · · Score: 3, Insightful

    It's a hard job. We're into year fifteen of ZFS and it's just starting to gain some features that make administration of it manageable by non-experts. Give it another five before you want to make it your default on a desktop for grandma. BTRFS will be along five years after that.

    If Apple can pull off something similar in a couple years, it will be a major triumph. It's too bad for everybody that Steve got bitchy at Jonathan and the community hasn't had Apple's help as a contributor for the past decade.

    --
    My God, it's Full of Source!
    OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
  4. This has been my biggest gripe about OS X/macOS... by mlts · · Score: 4, Informative

    I'm glad Apple has introduced this. As of now, the snapshot API and others are not present, but now Apple is on parity with everyone else in the industry.

    APFS isn't like ZFS or btrfs, but more like ReFS in the fact that it still requires a logical volume manager. It would be nice if it had RAID, but that is a minor item, compared to just getting rid of HFS+, which just had to be killed.

    Some features I like:

    The ability to encrypt volumes with multiple volume keys. It looks like it will be similar to Oracle's ZFS on Solaris, but the implementation can be completely different.

    Snapshots. Something like zfs send and zfs send -i will be quite useful for backups.
    Copy-on-write capability, which is useful for VMs.

    Of course, it appears that Apple will be documenting and publishing the FS's specs in 2017, which will be even more useful for compatibility.

    All and all, even though there is no RAID 5/RAID-Z, or LVM replacement, this is a heck of a lot better than what OS X/macOS has now.

  5. Re:NIH? by mlts · · Score: 4, Insightful

    Licensing. Apple did flirt with ZFS, but for some reason, and I would guess it was license issues, they decided not to go that route. Using btrfs would bring GPL/BSD licensing issues. So, Apple either had to license something like ReFS from MS, or roll their own.

  6. Re: Swift is stable. by hsmith · · Score: 2

    Lack of backwards compatibility is the issue. You shouldn't have to refactor your entire code base once a year (oh and Xcode doesn't support Swift refactoring). That is the gripe. Move the language forward, don't break it yearly. Swift is a nice language, I enjoy using it, but this is a big issue.

  7. Re:Swift is stable. by mlts · · Score: 4, Funny

    Swift 2 is stable enough that I get occasional calls from recruiters asking for five years of it as a language for dev jobs. So, if it is good enough to transcend time/space, it should be stable enough.

  8. Not Invented Here Syndrome? by mi · · Score: 4, Informative

    I was hopiing Apple would license ZFS

    ZFS is under CDDL and would not even need to be "licensed" in the usual sense — it is free for anybody to take. "Too free" for certain zealots, in fact, which is why it was not part of Linux kernel for a while — until the supposed "license incompatibility" myths got debunked.

    Even Linux now offers ZFS — Apple would've had a much easier time porting it, because MacOS is already FreeBSD-based and the FreeBSD-project had ZFS available "out of the box" for several major releases spanning many years.

    What did Apple find lacking about ZFS, that would justify creating their own, is, indeed, a mystery. Probably, a case of the Not Invented Here Syndrome. Sad...

    --
    In Soviet Washington the swamp drains you.
    1. Re:Not Invented Here Syndrome? by Anonymous Coward · · Score: 4, Informative

      Apple was working on ZFS support from 2007 (there was read only support in snow leopard) to 2010 or so. From what I've heard, they dropped it for legal reasons. Nope, not the CDL, not the Oracle/Sun buyout (Jobs and Ellison were good friends), but the NetApps ZFS lawsuit.

    2. Re:Not Invented Here Syndrome? by MachineShedFred · · Score: 4, Informative

      They actually had ZFS working for 10.6, but scrapped it because they couldn't come to terms with Sun. The package was on MacOS Forge back in the day, and the lead developer of it left Apple shortly afterward and created his own 3rd party implementation.

      This was before ZFS was licensed under CDDL.

      --
      Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
    3. Re:Not Invented Here Syndrome? by TheRaven64 · · Score: 4, Interesting

      It wasn't before ZFS was CDDL, ZFS was CDDL from the start. I've heard two conflicting explanations of why Apple dropped support (after publicly announcing it) from different people in their CoreOS team. One of them involved Apple wanting a proprietary license (CDDL is per-file copyleft, so Apple would have had to release changes that they made to any of the ZFS files) with terms that Sun/Oracle wouldn't grant, though I get the impression that the NetApp lawsuit may have been more of an issue.

      --
      I am TheRaven on Soylent News
    4. Re:Not Invented Here Syndrome? by jedidiah · · Score: 2

      > Apple has money that's vulnerable to baseless patent infringement lawsuits.

      Creating your own separate implementation doesn't stop that.

      --
      A Pirate and a Puritan look the same on a balance sheet.
  9. I've got a crazy idea by cyber-vandal · · Score: 3, Insightful

    How about letting users unplug removable media without having to eject it first like every other OS has had for about a decade.

    1. Re:I've got a crazy idea by Feral+Nerd · · Score: 2

      How about letting users unplug removable media without having to eject it first like no other OS has.

      There is no OS in existence that allows that.

      As a Mac user, I've noticed that Windows 8 and up seem to handle that particular situation just fine.

      Linux and MacOS (AFAIK) use write caching which makes it a good idea to eject USB drives on these operating systems (caveat: it's been about a years since I last checked this, things may have changed). Windows on the other hand does not "handle USB drive yanking just fine", it just disables the write cache for external which reduces the chances of file system corruption but does not eliminate it 100%. Disabling the write cache also slows performance (according to the Windows device properties menu, policies tab). Generally it is always a good idea to eject your drive unless you have a journaled file system on it which should theoretically be able to recover from yanking related corruption. Having said that I'd still recommend disabling write caching on your system if you are a habitual yanker.

    2. Re:I've got a crazy idea by Anonymous Coward · · Score: 5, Informative

      Having occasionally yanked out removable media on OS X without properly ejecting it, you can do so now. But you run the same risks as every other OS and commonly-used filesystem: that things will be corrupted in the process and have to be fixed the next time you insert the drive.

      What are these "other OS" you speak of? Windows? No. It will happily corrupt files depending upon what you are doing with the drive in question at the time you yank it out. Likewise Linux and most of its filesystems. Modern journaled filesystems are likely to be able to put things back into some semblance of order in the aftermath, but if you think it is routine to be able to do this without special setup you are mistaken.

      The only thing I've noticed is that Windows will complain less frequently when you yank out a device, whereas OS X will reliably and correctly warn you that doing so is dangerous and not recommended unless you eject it in software first. In fact, OS X is better at informing you which program has files open on the device when you attempt to eject it, whereas Windows will just vaguely tell you that something is still holding up the process. Oh, and Windows "helpfully" disables write caching to slow down your pluggable devices in an attempt to diminish the likelihood you'll corrupt something. Whether you consider that truly helpful or not is debatable. It's a significant tradeoff.

    3. Re:I've got a crazy idea by enriquevagu · · Score: 2

      The parent is right.

      But not only that. The flash controller could be running a background process, such as offline deduplication or data block movement for static wear levelling. These processes are *not* triggered by reads or writes from the OS, so even when you are not actively writing to the disk, simply removing it without ejecting *might* cause data corruption and data loss.

  10. Re:NIH? by CrashNBrn · · Score: 4, Insightful

    Would you want to be licensing anything from Oracle today?

  11. Re:Bring on OJFS by jittles · · Score: 4, Funny

    I was hoping Apple would license ZFS or even Veritas Volume Manager/Veritas FS from Symantec.

    I thought Veritas was also called Online Journaled File System (OnlineJFS or OJFS). What else is OJFS?

    OJFS? Why do you computer types insist on naming your filesystems after murders?

  12. Re: Swift is stable. by Anonymous Coward · · Score: 4, Insightful

    What a ridiculous argument.

    By that logic, backwards compatibility is never an issue. Why even try to offer it? You can just keep using the old version! Compatibility solved!

  13. Re:Nanosecond granularity? by tepples · · Score: 2

    It means you can guarantee that each file has a unique 64-bit timestamp by simply assigning a sequential nanosecond timestamp if two files get written at once. It also gives an opportunity to work around UNIX's year 2038 problem (2^31 UTC seconds since 1970) and Apple's year 2040 problem (2^32 UTC seconds since 1904), pushing it out to at least 2262 (2^63 UTC nanoseconds since 1970).

  14. Re:Bring on OJFS by Anonymous Coward · · Score: 2, Funny

    Well there's ReiserFS. At least there's precedent.

  15. ZFS is not recommended for non-ECC RAM by perpenso · · Score: 5, Informative

    ZFS is not recommended for non-ECC RAM. RAM errors can get propagated to disk by application read operations, not just writes.
    http://research.cs.wisc.edu/ad...

    1. Re:ZFS is not recommended for non-ECC RAM by Fweeky · · Score: 3, Insightful

      When checksums fail ZFS will assume the problem is on disk and attempt to "repair" the data on disk. This automatic repair is a great feature, when your RAM can be trusted.

      Repair by attempting to correct the data from a redundant location, if one exists, and if its checksum passes. The bit flips required to make such a process actually damage your data seems quite convoluted - it'd have to be multiple errors in different locations happening at just the right times - one in the read before the checksum is checked, one in the data to repair it after the checksum has been verified but before it's written back.

      "By default, access time updates are enabled in ZFS; therefore, a read-only workload will update the access time of any file accessed. Consequently, when the structure containing the access time (znode) goes inactive (or when there is another workload that updates the znode), ZFS writes the block holding the znode to disk and updates and writes all its parental blocks. Therefore, any corruption to these blocks will become permanent after the flush caused by the access time update"

      In-memory filesystem metadata can get damaged and end up in on-disk structures regardless of which one you use, and it's far from the only fs with atime updates. Is ZFS really significantly more vulnerable to this by comparison, or is it just that ZFS won't defend you against it?

      My quick skim of the paper suggests the latter. They don't seem to condemn ZFS for being worse, rather, they show it suffers the same sort of problems they find ext2 suffers from in face of memory errors, while demonstrating it's great at picking up errors from the disk/IO controller/etc.

    2. Re:ZFS is not recommended for non-ECC RAM by Fweeky · · Score: 2

      For you it may be a low risk. For Apple its not. Apple will be shipping millions of machines.

      And these machines are already vulnerable just to single bit errors anywhere both in the IO path and in memory.

      The repair-of-death you describe involves multiple errors in the memory path occurring in a specific order and in relatively specific places, that are already dangerous to existing filesystems.

      The atime update metadata corruption you quote is similarly already a problem with existing filesystems. In fact it's more of a problem for these filesystems because they're overwriting existing metadata, not creating new copies of metadata that can be rolled back in a disaster as ZFS does.

      Even if we take it as true that ZFS is more vulnerable to these specific types of error (by no means demonstrated), that needs to be balanced against all the other errors it's less vulnerable against.

      Stop approaching this from the perspective that ZFS is flawed. Rather approach this from the perspective that ZFS assumes memory can be trusted

      ... so does every other filesystem. I'll quote another bit of that paper you like:

      "In addition to ZFS, we have applied the same fault injection framework used in Section 5 to a simpler filesystem,ext2. Our initial results indicate that ext2 is also vulnerable to memory corruptions. For example, corrupt data can be returned to the user or written to disk. When certain fields of a VFS inode are corrupted, operations on that inode fail or the whole system crashes. If the inode is dirty, the corrupted fields of the VFS inode are propagated to the inode in the page cache and are then written to disk, making the corruptions permanent. Moreover, if the superblock in the page cache is corrupted and flushed to disk, it might result in an unmountable filesystem"

      Intel is more likely to support ECC in lower end CPUs (ex i3) than in mid to higher end CPUS (ex i5, i7)

      i7-class Xeons (E3-XXXX) support ECC and are usually priced basically identically to their i7 cousins. i3's get used in tiny NAS systems like HP Microservers, probably why they come in ECC variants.

      Another difficulty for a consumer oriented company like Apple, making using ECC not really an option for them

      I'm sure Apple are more than capable of pushing for it if they considered it a priority. They have the purchasing power, they have the margins, they have the PR to make people wet themselves over the benefits if they so choose.

  16. Re:Swift is stable. by Darinbob · · Score: 2

    It's HR boilerplate. They know nothing about the technology, but they know that for entry level they want degree plus course work in $X, for junior they want 5 years $X, and for senior they want 10 years $X and $Y. I've read a job description for our group once and showed it to my manager asking what position it was for, and he replied "wait, that's not what I wrote!"