Slashdot Mirror

← Back to Stories (view on slashdot.org)

Dropbox Open Sources DivANS: a Compression Algorithm In Rust Compiled To WASM (dropbox.com)

Posted by EditorDavid on from the cmprssn dept.

Slashdot reader danielrh writes: DivANS is a new compression algorithm developed at Dropbox that can be denser than Brotli, 7zip or zstd at the cost of compression and decompression speed. The code uses some of the new vector intrinsics in Rust and is multithreaded. It has a demo running in the browser.

One of the new ideas is that it has an Intermediate Representation, like a compiler, and that lets developers mashup different compression algorithms and build compression optimizers that run over the IR. The project is looking for community involvement and experimentation.

33 comments

  1. Apache License by infolation · · Score: 3, Informative

    It's not mentioned in the article but DivANS is released under the Apache License.

  2. Main upside by Anonymous Coward · · Score: 0

    Its main upside appears to be being written in rust. Well done, chaps, for such astounding relevance in your rust movement community contributions.

    1. Re:Main upside by KiloByte · · Score: 1

      Its main upside appears to be being written in rust.

      Yay for a language with API breaks that make its compiler unbuildable with previous minor versions of itself, and such a stellar portability.

      --
      The creatures outside looked from Alt-Right to Antifa; but already it was impossible to say which was which.
    2. Re: Main upside by Anonymous Coward · · Score: 0

      The language, its standard library and its one and only implementation all sucking are all minor problems compared to Rust's totalitarian community. I don't intend this as an insult, but in my opinion most Rust community members suffer from some degree of autism or Asperger's. This means that they need a Code of Conduct to function like something resembling a community, because they have no inherent social skills to guide them otherwise. It also means that they go way overboard about stuff like 'social justice' (which had nothing to do with programming at all), and they even have the Rust Moderation Team that effectively serves to eliminate anybody deemed 'undesirable'. It's a very pathetic community that puts on a fake veneer of civility.

    3. Re: Main upside by phantomfive · · Score: 1

      The language, its standard library and its one and only implementation all sucking are all minor problems compared to Rust's totalitarian community. I don't intend this as an insult, but in my opinion most Rust community members suffer from some degree of autism or Asperger's

      I'd rather have that than have so party clowns or people who are obsessed with popularity. What you need for good quality is heavy focus and deep knowledge, and whether or not they have the social skills of a Kardashian is irrelevant.

      --
      "First they came for the slanderers and i said nothing."
    4. Re: Main upside by Anonymous Coward · · Score: 0

      It's not assburgers - they just enjoy persecuting people.

    5. Re: Main upside by Anonymous Coward · · Score: 0

      Ya rust doesnâ(TM)t have that either.

    6. Re:Main upside by nagora · · Score: 1

      I have no mod points and I must scream - yes. I am sick of new languages that are just toys for their development team to ticker with ad infinitum. A bit more thought before v0.0.1 and a lot less breaking v7.74.91 code with v7.74.91-r1 and I might bother my arse to look at these things.

      --
      "Encyclopedia" is to "Wikipedia" what "Library" is to "Some people at a bus stop"
    7. Re: Main upside by Anonymous Coward · · Score: 0

      There is no such thing as Aspergers. It was a bullshit disease and it got axed as a valid diagnosis.

      Fuck assburgers.

  3. Thats all well and good but... by Hentai007 · · Score: 2, Funny

    What's the Weissman score?

  4. Re: Dead project by ArchieBunker · · Score: 0, Offtopic

    I remember when the trolls were funny and creative. Now they are sad autists who were shunned from 4chan.

    --
    Only the State obtains its revenue by coercion. - Murray Rothbard
  5. ultimate compression- been there, done that by swell · · Score: 1, Funny

    I like Dropbox and I'm sure they have a nice algorithm, but . . .

    Does nobody remember the ultimate compression algorithm from 1995 that could scrunch any amount of data to less than 1024 bytes. The DataFiles/16 program got quite a lot of publicity for WEB Technologies.

    As I recall there were some inconveniences; for instance for really serious compression one had to run the software multiple times- compress, then compress the resulting file, then compress that resulting file. Nevertheless that was a lot of compression! There were minor technical glitches. For example, the decompressed file was quite unlike the original.

    --
    ...omphaloskepsis often...
    1. Re:ultimate compression- been there, done that by Anonymous Coward · · Score: 0

      Yep, for every new generation of programmers there is always a subset that is unable to grasp that for some files to be smaller when compressed, others have to be bigger when using the same algorithm.
      The exception is of course lossy compression, where instead, multiple different files will compress and decompress into identical sequences.

  6. Re: Dead project by Anonymous Coward · · Score: 0

    He would have to spam the same technically illiterate wall of text for years and amass a sock puppet army to be able to compare to the troll APK.

    numbnuts

  7. one man's huge ... by epine · · Score: 1

    Even a 1% improvement in compression efficiency can make a huge difference.

    Hard Drive Cost Per Gigabyte — July 2017

    Looks like we're on track for $20/TB, if you purchase in bulk.

    Let's monetize a "huge difference" at $1000 (which I regard as the smallest available value for a "huge difference").

    Thus, your 1% extra compression needs to save 50 TB to make a "huge difference" of one large.

    Correct me if I'm wrong, but I'm thinking your dataset needs to be on the order of 5 PB for a 1% compression improvement to shave 50 TB.

    5 PB works out to 200,000 single-layer Blu-ray disks.

    Nice home library. (I think we can already safely assume it's not mostly drama, unless you're Pacman-ratting a good half of the entire IMDB movie list, behind the darknet spider from hell.)

    1. Re:one man's huge ... by SoftwareArtist · · Score: 1

      Here are the prices for Amazon cloud storage. Depending on the type of storage, it ranges from $0.025 to $0.125 per GB per month. Yes, that's a lot more than buying a hard drive, but a huge stack of hard drives is pretty useless for storing lots of data. This gives immediate availability to all your data, backups, etc.

      Let's say a company has 1 PB of data they need to store. Depending on the type of storage they need, that will cost between $25,000 and $125,000 per month. A 1% reduction in that cost could save them over $1000 per month, which is definitely meaningful.

      --
      "I'm too busy to research this and form an educated opinion, but I do have time to tell everyone my uninformed opinion."
    2. Re:one man's huge ... by Anonymous Coward · · Score: 1

      Correct me if I'm wrong

      You are not wrong, but you are missing the entire point of this.

      If it was about disk storage space then Dropbox would be fine with just compressing it locally. There would be no need whatsoever to compile to WASM.
      The point of having a WASM compressor is that you can compress the files on the client side without them having to install any programs for it.
      The saving isn't in disk space, it is in bandwidth.

      While compressing in the browser might be inefficient there is still an extra bonus for Dropbox here since the workload is offloaded to the user rather than being done by their servers.

      Assuming they already compressed the data they will have exactly the same disk usage as before but with less bandwidth and less CPU usage.

    3. Re:one man's huge ... by Kjella · · Score: 1

      Let's say a company has 1 PB of data they need to store. Depending on the type of storage they need, that will cost between $25,000 and $125,000 per month. A 1% reduction in that cost could save them over $1000 per month, which is definitely meaningful.

      Except 1PB is a lot of data. Walmart for example have 40 PB in their data cloud, so they could save ~$40,000 on a $500,000,000,000 business. CERN has 200 PB so that'd save ~$200,000 compared to the $9,000,000,000 budget of the LHC. It's a rounding error and I think if you're working with that kind of data you've already worked on much more specific ways to compress it that won't leave much value in a general compression algorithm. Like Google working on a new video compression algorithm for YouTube makes sense. But there's little point in making yet another compression format for 1%, unless it's transparent like in a LTO tape drive or something.

      --
      Live today, because you never know what tomorrow brings
  8. Re: Dead project by Anonymous Coward · · Score: 0

    You're unidentifiable anonymous stalker of apk he calls jealous lil' jowie. You're 3 stooges in 1. Anonymous, your real account + your sockpuppet account. Probably many sockpuppet alternate accounts for you though. Apk sure got your goat didn't he when he pointed out truth you can't take that you are a nerdowell he also calls you.

  9. no CoC! by Anonymous Coward · · Score: 0

    And no petty-authoritarian Progressive "code of conduct"! So us deplorable straight men are free to contribute. Good call, Dropbox, good call.

  10. Re: Dead project by Reverend+Green · · Score: 1

    Nah. Most of them are straight up astroturfers and disinformation operators.

  11. Itâ(TM)s not storage stupid by Anonymous Coward · · Score: 0

    Storage costs are negligible. Itâ(TM)s the cost and speed of transmission.

  12. No such thing as better universal compression by goombah99 · · Score: 1

    The more a compression algorithm can compress one file, there has to be another file that it actually makes larger. Entropy is a bitch.

    --
    Some drink at the fountain of knowledge. Others just gargle.
    1. Re: No such thing as better universal compression by Anonymous Coward · · Score: 0

      Generally speaking, compressing the already compressed file fits your definition

  13. Re: Dead project by Anonymous Coward · · Score: 0

    #derpderpderp russian trolls under every rock