Slashdot Mirror

← Back to Stories (view on slashdot.org)

Dropbox Open Sources DivANS: a Compression Algorithm In Rust Compiled To WASM (dropbox.com)

Posted by EditorDavid on from the cmprssn dept.

Slashdot reader danielrh writes: DivANS is a new compression algorithm developed at Dropbox that can be denser than Brotli, 7zip or zstd at the cost of compression and decompression speed. The code uses some of the new vector intrinsics in Rust and is multithreaded. It has a demo running in the browser.

One of the new ideas is that it has an Intermediate Representation, like a compiler, and that lets developers mashup different compression algorithms and build compression optimizers that run over the IR. The project is looking for community involvement and experimentation.

33 comments

  1. Apache License by infolation · · Score: 3, Informative

    It's not mentioned in the article but DivANS is released under the Apache License.

  2. AC Open Sores On Hispanic From Rusty Pole by Anonymous Coward · · Score: -1

    Frost ladies

  3. Main upside by Anonymous Coward · · Score: 0

    Its main upside appears to be being written in rust. Well done, chaps, for such astounding relevance in your rust movement community contributions.

    1. Re:Main upside by KiloByte · · Score: 1

      Its main upside appears to be being written in rust.

      Yay for a language with API breaks that make its compiler unbuildable with previous minor versions of itself, and such a stellar portability.

      --
      The creatures outside looked from Alt-Right to Antifa; but already it was impossible to say which was which.
    2. Re: Main upside by Anonymous Coward · · Score: 0

      The language, its standard library and its one and only implementation all sucking are all minor problems compared to Rust's totalitarian community. I don't intend this as an insult, but in my opinion most Rust community members suffer from some degree of autism or Asperger's. This means that they need a Code of Conduct to function like something resembling a community, because they have no inherent social skills to guide them otherwise. It also means that they go way overboard about stuff like 'social justice' (which had nothing to do with programming at all), and they even have the Rust Moderation Team that effectively serves to eliminate anybody deemed 'undesirable'. It's a very pathetic community that puts on a fake veneer of civility.

    3. Re: Main upside by phantomfive · · Score: 1

      The language, its standard library and its one and only implementation all sucking are all minor problems compared to Rust's totalitarian community. I don't intend this as an insult, but in my opinion most Rust community members suffer from some degree of autism or Asperger's

      I'd rather have that than have so party clowns or people who are obsessed with popularity. What you need for good quality is heavy focus and deep knowledge, and whether or not they have the social skills of a Kardashian is irrelevant.

      --
      "First they came for the slanderers and i said nothing."
    4. Re: Main upside by Anonymous Coward · · Score: 0

      It's not assburgers - they just enjoy persecuting people.

    5. Re: Main upside by Anonymous Coward · · Score: 0

      Ya rust doesnâ(TM)t have that either.

    6. Re:Main upside by nagora · · Score: 1

      I have no mod points and I must scream - yes. I am sick of new languages that are just toys for their development team to ticker with ad infinitum. A bit more thought before v0.0.1 and a lot less breaking v7.74.91 code with v7.74.91-r1 and I might bother my arse to look at these things.

      --
      "Encyclopedia" is to "Wikipedia" what "Library" is to "Some people at a bus stop"
    7. Re: Main upside by Anonymous Coward · · Score: 0

      There is no such thing as Aspergers. It was a bullshit disease and it got axed as a valid diagnosis.

      Fuck assburgers.

  4. Dead project by Anonymous Coward · · Score: -1

    They're using spaces...

    1. Re:Dead project by Anonymous Coward · · Score: -1

      but they're compressed!

    2. Re: Dead project by Anonymous Coward · · Score: -1, Flamebait

      The Kike King Trump is going to be dead after decades in Ft Leavenworth once Mueller has tried him for treason, where rabid trans nlggers will sodomize him for eternity.

    3. Re: Dead project by Anonymous Coward · · Score: -1

      Time to ban this spammer. You're as bad as APK and the idiot who spams about creimer. You trolls are like an incredibly unfunny version of The Three Stooges.

    4. Re: Dead project by Anonymous Coward · · Score: -1

      Hey! Leave APK out of this. His HOSTS file engine is amazing! I haven't even needed to run CleanMyPc since finding this tool. I've only used the Windows version, but I saw recently he's now supporting this "Linux" OS. If that's good enough for APK, I think I might have to take a look at it.

    5. Re: Dead project by Anonymous Coward · · Score: -1

      Hey! Leave creimer out of this. He is a reformed IT closet cleaner who ran out of content to put on YouTube. His videos got 800 views in a couple days, then went stale. I tried to give him the funds to pay for more bot views, but the good, god fearing, honest, hard working man that is creimer, turned it down.

      He made $50 in 3 months off of affiliate goat c tees. The man is clearly a visionary. Visionaries such as creimer are rarely understood.

    6. Re: Dead project by ArchieBunker · · Score: 0, Offtopic

      I remember when the trolls were funny and creative. Now they are sad autists who were shunned from 4chan.

      --
      Only the State obtains its revenue by coercion. - Murray Rothbard
    7. Re: Dead project by Anonymous Coward · · Score: 0

      He would have to spam the same technically illiterate wall of text for years and amass a sock puppet army to be able to compare to the troll APK.

      numbnuts

    8. Re: Dead project by Anonymous Coward · · Score: 0

      You're unidentifiable anonymous stalker of apk he calls jealous lil' jowie. You're 3 stooges in 1. Anonymous, your real account + your sockpuppet account. Probably many sockpuppet alternate accounts for you though. Apk sure got your goat didn't he when he pointed out truth you can't take that you are a nerdowell he also calls you.

    9. Re: Dead project by Reverend+Green · · Score: 1

      Nah. Most of them are straight up astroturfers and disinformation operators.

    10. Re: Dead project by Anonymous Coward · · Score: 0

      #derpderpderp russian trolls under every rock

  5. YOU FAIL IT! by Anonymous Coward · · Score: -1

    YOU FAIL IT!

  6. Thats all well and good but... by Hentai007 · · Score: 2, Funny

    What's the Weissman score?

  7. ultimate compression- been there, done that by swell · · Score: 1, Funny

    I like Dropbox and I'm sure they have a nice algorithm, but . . .

    Does nobody remember the ultimate compression algorithm from 1995 that could scrunch any amount of data to less than 1024 bytes. The DataFiles/16 program got quite a lot of publicity for WEB Technologies.

    As I recall there were some inconveniences; for instance for really serious compression one had to run the software multiple times- compress, then compress the resulting file, then compress that resulting file. Nevertheless that was a lot of compression! There were minor technical glitches. For example, the decompressed file was quite unlike the original.

    --
    ...omphaloskepsis often...
    1. Re:ultimate compression- been there, done that by Anonymous Coward · · Score: 0

      Yep, for every new generation of programmers there is always a subset that is unable to grasp that for some files to be smaller when compressed, others have to be bigger when using the same algorithm.
      The exception is of course lossy compression, where instead, multiple different files will compress and decompress into identical sequences.

  8. one man's huge ... by epine · · Score: 1

    Even a 1% improvement in compression efficiency can make a huge difference.

    Hard Drive Cost Per Gigabyte — July 2017

    Looks like we're on track for $20/TB, if you purchase in bulk.

    Let's monetize a "huge difference" at $1000 (which I regard as the smallest available value for a "huge difference").

    Thus, your 1% extra compression needs to save 50 TB to make a "huge difference" of one large.

    Correct me if I'm wrong, but I'm thinking your dataset needs to be on the order of 5 PB for a 1% compression improvement to shave 50 TB.

    5 PB works out to 200,000 single-layer Blu-ray disks.

    Nice home library. (I think we can already safely assume it's not mostly drama, unless you're Pacman-ratting a good half of the entire IMDB movie list, behind the darknet spider from hell.)

    1. Re:one man's huge ... by SoftwareArtist · · Score: 1

      Here are the prices for Amazon cloud storage. Depending on the type of storage, it ranges from $0.025 to $0.125 per GB per month. Yes, that's a lot more than buying a hard drive, but a huge stack of hard drives is pretty useless for storing lots of data. This gives immediate availability to all your data, backups, etc.

      Let's say a company has 1 PB of data they need to store. Depending on the type of storage they need, that will cost between $25,000 and $125,000 per month. A 1% reduction in that cost could save them over $1000 per month, which is definitely meaningful.

      --
      "I'm too busy to research this and form an educated opinion, but I do have time to tell everyone my uninformed opinion."
    2. Re:one man's huge ... by Anonymous Coward · · Score: 1

      Correct me if I'm wrong

      You are not wrong, but you are missing the entire point of this.

      If it was about disk storage space then Dropbox would be fine with just compressing it locally. There would be no need whatsoever to compile to WASM.
      The point of having a WASM compressor is that you can compress the files on the client side without them having to install any programs for it.
      The saving isn't in disk space, it is in bandwidth.

      While compressing in the browser might be inefficient there is still an extra bonus for Dropbox here since the workload is offloaded to the user rather than being done by their servers.

      Assuming they already compressed the data they will have exactly the same disk usage as before but with less bandwidth and less CPU usage.

    3. Re:one man's huge ... by Kjella · · Score: 1

      Let's say a company has 1 PB of data they need to store. Depending on the type of storage they need, that will cost between $25,000 and $125,000 per month. A 1% reduction in that cost could save them over $1000 per month, which is definitely meaningful.

      Except 1PB is a lot of data. Walmart for example have 40 PB in their data cloud, so they could save ~$40,000 on a $500,000,000,000 business. CERN has 200 PB so that'd save ~$200,000 compared to the $9,000,000,000 budget of the LHC. It's a rounding error and I think if you're working with that kind of data you've already worked on much more specific ways to compress it that won't leave much value in a general compression algorithm. Like Google working on a new video compression algorithm for YouTube makes sense. But there's little point in making yet another compression format for 1%, unless it's transparent like in a LTO tape drive or something.

      --
      Live today, because you never know what tomorrow brings
  9. no CoC! by Anonymous Coward · · Score: 0

    And no petty-authoritarian Progressive "code of conduct"! So us deplorable straight men are free to contribute. Good call, Dropbox, good call.

  10. Itâ(TM)s not storage stupid by Anonymous Coward · · Score: 0

    Storage costs are negligible. Itâ(TM)s the cost and speed of transmission.

  11. No such thing as better universal compression by goombah99 · · Score: 1

    The more a compression algorithm can compress one file, there has to be another file that it actually makes larger. Entropy is a bitch.

    --
    Some drink at the fountain of knowledge. Others just gargle.
    1. Re: No such thing as better universal compression by Anonymous Coward · · Score: 0

      Generally speaking, compressing the already compressed file fits your definition