Slashdot Mirror


Google Further Shrinks the Size of Android App Updates (engadget.com)

Google says it has found and implemented a new way to make app updates on Android smaller. From a report on Engadget: They're introducing a new approach to app updates that promises to radically shrink the size of updates with "file-by-file" patching. The resulting patches tend to be about 65 percent smaller than the app itself, and are sometimes over 90 percent smaller. In the right circumstances, that could make the difference between updating while you're on cellular versus waiting until you find WiFi. The technique revolves around spotting changes in the uncompressed files (that is, when they're not squeezed into a typical app package). Google first decompresses the old and new app versions to determine the changes between files and create a patch. After that, updating is just a matter of unpacking the app on your device, applying changes and compressing it again.

50 comments

  1. Compress smarter by JoeMerchant · · Score: 2

    Get smaller patch sizes by compressing the things that people actually change... didn't git do something like this?

    1. Re:Compress smarter by PatientZero · · Score: 1

      This sounds an awful lot like middle-out compression. I smell a lawsuit!

      --
      Freedom to fear. Freedom from thought. Freedom to kill.
      I guess the War on Terror really is about freedom!
  2. What a breakthrough! by jeffb+(2.718) · · Score: 5, Insightful

    Now, if only there were a way to shrink the update further by transferring only the parts of files that have changed -- in other words, the information that's actually different...

    1. Re:What a breakthrough! by JoeMerchant · · Score: 2

      diff, rsync, git, et.al. are NIH, can't use them without compromising security of the user lock-in.

    2. Re:What a breakthrough! by CastrTroy · · Score: 1

      I imagine that a lot of the stuff that's changed such as images and sounds doesn't really work well with diff anyway.

      --

      Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
    3. Re:What a breakthrough! by Anonymous Coward · · Score: 0

      Doesn't Windows do this?

      And hasn't Microsoft figured out a long time ago that this is actually a pretty bad idea since, if you have version 1 of something, and version 8 is the most recent, you end up having to download patch 1 to 2, 2 to 3, 3 to 4, etc, and apply them all successively (hoping the process doesn't fail in the meantime), making the update process take forever?

      This is the primary motivation behind their new "let's bundle everything together into a mega patch" effort, after all.

      This works great for git and stuff, but not so great for whole applications.

    4. Re:What a breakthrough! by Anonymous Coward · · Score: 0

      Doesn't Windows do this?

      And hasn't Microsoft figured out a long time ago that this is actually a pretty bad idea since, if you have version 1 of something, and version 8 is the most recent, you end up having to download patch 1 to 2, 2 to 3, 3 to 4, etc, and apply them all successively (hoping the process doesn't fail in the meantime), making the update process take forever?

      This is the primary motivation behind their new "let's bundle everything together into a mega patch" effort, after all.

      This works great for git and stuff, but not so great for whole applications.

      Baloney.... RPM based systems, such as Fedora and RHEL have been doing this for years. I am sure that all of the other package based Linux systems do it as well.

      The update downloads the delta patches and then applies them.

    5. Re:What a breakthrough! by unrtst · · Score: 1

      I imagine that a lot of the stuff that's changed such as images and sounds doesn't really work well with diff anyway.

      See the rsync protocol, which can be used by things other than rsync. For example, BackupPC uses it to transfer only the parts of files that have changed, which includes binary files.

    6. Re:What a breakthrough! by Anonymous Coward · · Score: 0

      There's a binary diff that's used by the firefox updater, debdelta and similar.

    7. Re:What a breakthrough! by unrtst · · Score: 1

      ... a pretty bad idea since, if you have version 1 of something, and version 8 is the most recent, you end up having to download patch 1 to 2, 2 to 3, 3 to 4, etc, and apply them all successively ...

      So the problem is obvious then, and easy to resolve.
      * calculate the patchset from version 1 to 8; from 2 to 8; from 3 to 8; ... etc ...; from 7 to 8.
      * provide the patchset based on their current version and target version

      Alternatively
      * provide the patchset between versions, but if the user is further out of date, just provide the whole new apk.
      * TFS claimed a savings of only 65%, so sending more than two patches would use more than sending the whole apk.
      * Patches between distant versions would likely have worse results than 65% savings too.
      * You could even make it conditional, since you have the sizes when being requested.

    8. Re:What a breakthrough! by SuperKendall · · Score: 1

      I would argue that you don't need to worry about diff for images and sounds, because an image/sound is either wholly replaced or not changing release to release. Those would simply not be sent for most updates if not changed. The remaining content could benefit from diffs.

      --
      "There is more worth loving than we have strength to love." - Brian Jay Stanley
    9. Re:What a breakthrough! by Nahor · · Score: 2

      They do have their own binary diff format. They use it to update Chrome:
      https://www.chromium.org/devel...

    10. Re:What a breakthrough! by Anonymous Coward · · Score: 0

      They do have their own binary diff format. They use it to update Chrome:
      https://www.chromium.org/devel...

      Google Chrome is based on the open source Chromium. Chromium has the binary diff format, and Google benefits from it. So do I, as I'm a Chromium user on all laptops and desktops here.

    11. Re:What a breakthrough! by JoeMerchant · · Score: 1

      That's because Windows users are unruly, blocking updates, etc.

      How many Android users just let the updates roll in hourly?

    12. Re:What a breakthrough! by tepples · · Score: 1

      But do the major repositories that distribute Chromium (Debian, Ubuntu, Fedora, CentOS, etc.) use this binary diff technology? Or do they still resend the entire deb/rpm?

    13. Re:What a breakthrough! by tepples · · Score: 1

      Lossless compression, such as PKZIP for Microsoft Office and LibreOffice documents, FLAC for audio, and PNG for images, makes it less trivial in the general case to find "only the parts of files that have changed". A protocol like rsync would need to have specific support for each archive container and lossless codec in order to identify said "parts". This story is about Google Play Store gaining such support for APK files, which use the PKZIP container.

    14. Re:What a breakthrough! by Anonymous Coward · · Score: 0

      Now, if only there were a way to shrink the update further by transferring only the parts of files that have changed -- in other words, the information that's actually different...

      The large files in APKs tend to be media files which don't usefully diff anyway. I know the engineers who designed this and they looked at rdiff, et al and found it increased the complexity and processing requiremetns without doing much for download sizes.

    15. Re:What a breakthrough! by unrtst · · Score: 1

      They are unzipping the apk first. It's right in the summary. So the GP I replied to was referring to the files within the apk, and said, "a lot of the stuff that's changed such as images and sounds doesn't really work well with diff anyway". This has nothing to do with the lossless compression you mentioned (which, though I doubt they're doing this, could be diff'd very well with a slightly smarter diff, as they are lossless, so they can be decompressed first, then diff'd, and the diff can be sent and applied).
      That said, jpg's and mp3's within the apk would be difficult to diff if the original changed or the processor changed. However, if the original image or sound did not change, and it was processed (wav to mp3, png to jpg, whatever) by the same thing that did it the last time, then there will be no change on the file. If it was a raw file of some sort, and part of it changed, it should diff fairly cleanly.

    16. Re:What a breakthrough! by Hognoxious · · Score: 1

      Another way would be to install fewer shitty apps in the first place. I reckon half of of my phone's storage is taken up by stuff that I don't use. I don't even know what some of it even is.

      --
      Confucius say, "Find worm in apple - bad. Find half a worm - worse."
    17. Re:What a breakthrough! by mlyle · · Score: 1

      His comment was "only the parts of [image, audio] files that changed. It seems you are agreeing with him.

    18. Re:What a breakthrough! by Anonymous Coward · · Score: 0

      Or do they still resend the entire deb/rpm?

      No. RPM deltas have been implemented for a long time in CentOS and Fedora.

  3. So, the next step is ... by Anonymous Coward · · Score: 0

    rsync?

    Not only file by file but only changes within a file?

  4. Wow by kamapuaa · · Score: 0

    People still have limited data plans, where this really matters to the end-user?

    Why?

    --
    Slashdot: providing anti-social weirdos a soapbox, since 1997.
    1. Re:Wow by godrik · · Score: 4, Insightful

      Provided some simple apps can be north of 100MB, I very much welcome that you can update without redownloading everything. That is particularly true for apps that insist on being at the latest version to run.

    2. Re:Wow by Anonymous Coward · · Score: 0

      Why do you care? Unlimited data plans have been very cheap for several years now.

    3. Re:Wow by SQLGuru · · Score: 2

      Because even unlimited plans are limited......pick one and there's a cap (usually right around 22GB for some reason) where the speed will drop. If you hit that cap often enough, you'll be dropped from the carrier. It's in the fine print.

      And they aren't "cheap" unless you're a single person making decent money or with few other bills to pay. Family plans save money vs a bunch of individual plans, but they still aren't cheap.

    4. Re:Wow by Anonymous Coward · · Score: 1

      Yes, you stupid idiot. It turns out that phone service all over the world is not the same. If you actually crawled out of the basement and walked your fat ass outside and traveled a little bit you might know that.

    5. Re:Wow by Calydor · · Score: 1

      *) Offer valid only in some countries.

      --
      -=This sig has nothing to do with my comment. Move along now=-
    6. Re:Wow by Anonymous Coward · · Score: 0

      Because even unlimited plans are limited......pick one and there's a cap (usually right around 22GB for some reason) where the speed will drop. If you hit that cap often enough, you'll be dropped from the carrier. It's in the fine print.

      And they aren't "cheap" unless you're a single person making decent money or with few other bills to pay. Family plans save money vs a bunch of individual plans, but they still aren't cheap.

      They're cheap here in Finland. We have unlimited plans starting at eur20/month for 20Mbps (or lower prices if you want lower speed) which restricts you to the bandwidth every second of the month. Of course, you can get a 3G/4G contract with a cap if you want it, but most people don't...

    7. Re:Wow by Anonymous Coward · · Score: 0

      Wow that is a lot of anger about cell phone plans.

      Anyway, Slashdot is a US oriented site.

    8. Re:Wow by Anonymous Coward · · Score: 0

      Google Fi... gets pricey if you treat it as "unlimited" data plan... (yeah, it's unlimited, at $10 per gigabyte...).

      (and yes, those youtube and ad videos pretty quickly can push monthly usage towards a few gigabytes--consider that each and every web-page you view costs you about 10-50 cents to visit).

    9. Re:Wow by tepples · · Score: 2

      Unlimited data plans have been very cheap for several years now.

      Not in Slashdot's home country.

      In some more remote parts of the U.S., the only option for high-speed home Internet is satellite, and satellite doesn't have unmetered data during waking hours. In other remote parts, such as East Buchanan, Iowa, DSL is harshly metered as well.

      Cellular carrier T-Mobile has introduced an unmetered plan, whose name reminds one of synthetic motor oil. But at $70 per month for the first of four lines on a plan, it's more expensive than metered plans, and it puts the subscriber at the back of QoS after using 25 GB in a month. Furthermore, in some parts of the U.S., Verizon is the only cellular carrier with adequate coverage, and Verizon has refused to offer unmetered plans.

    10. Re:Wow by Anonymous Coward · · Score: 0

      Most anger problems (such as yours) are actually mis-directed anger. You lash out angrily at others needlessly, in your private life and on the internet, most likely some other thing beyond your control is actually the root to your problem. Google "Why am I so angry", and click some of the links that come up. Psychology Today explains it well. Have a nice day! :)

  5. Google research scientists have done it again! by Anonymous Coward · · Score: 1

    Always amazed out the fact innovation coming out of that place

    1. Re:Google research scientists have done it again! by Anonymous Coward · · Score: 0

      lol

  6. Will this make unity apps suck less? by Anonymous Coward · · Score: 0

    I thought I read something awhile ago that indicated that unity apps package all of their resources in one blob. My guess is that means no difference to those apps?

    1. Re: Will this make unity apps suck less? by Anonymous Coward · · Score: 0

      The blob is in the eye of the beholder

  7. Bloat by Anonymous Coward · · Score: 0

    Back when I made apps I compressed with kzip and that produces smaller apk files, this will undo all that effort.

  8. Oh, sure, this willwork by rickb928 · · Score: 1

    Especially the updating over Cellular vs, WiFi thing.

    I regularly get updates (yeah, some of us do) that try to force going over WiFi, despite my having set updates over both Cellular (Mobile) and WiFi and expecting these to happen. Whether the carrier, Google Play Store, or the app dev do this, they try to force these over WiFi, size not being an obvious factor.

    So I have to manually trigger these.

    Morons. What fun it is to have WiFi enabled on my phone during lunch, when I walk by 6 different 'open' hotspots, each of which wants me to accept some T&C whatever, and each latches onto my phone and prevents it from even receiving SMS until I do something. Yes. T-Mobile. Probably due to WiFi Calling, I bet.

    It's not all kittens and glitter out there. Stuff sometimes doesn't really work like ti was expected to, and I should not complain. Much. I get paid to deal with these sorts of problems, and if I were not, or they did not happen, I would have to work for a living.

    It's not hard work to point out the obvious, just annoying.

    --
    deleting the extra space after periods so i can stay relevant, yeah.
  9. All well and good... by QuietLagoon · · Score: 0

    But if the OEMs don't bother patching, who cares how small the patches are? Google needs to address the elephant in the room - how can google, ummmm, encourage its OEMs to patch Android systems?

  10. Quick get a patent by Anonymous Coward · · Score: 1

    What a novel concept nobody has ever implemented before. Certainly worthy of patent protection.

  11. Google discovers deltarpm by mrsam · · Score: 1

    How old is deltarpm? I'm sure that deb packages must have something analogous. The funny thing is there has been some chatter on fedora-devel about removing delta rpms. Supposedly it's no longer considered useful anymore. Don't recall what's the status on that.

  12. Was Harder Than It Seems by Anonymous Coward · · Score: 0

    I tried doing this with some software I released back in dial-up days. On the surface it sounds great, just download a tiny binary diff and apply it to the program. In practice it was a nightmare. First, I didn't read the fine print closely enough. While the diff tool provided very small diff files, it wasn't deterministic. When you applied the patch you didn't always end up with a 100% clone of the new file. I don't have a clue how that's useful, what data format can handle random corruption? So I switched to a different diff tool. This one's diff files were larger but they always worked. Now you need to generate diff files for every release version of your software. Users don't like downloading and applying update 1, update 2, update 3, etc... But they also don't like having to skim through a ton of updates trying to matching their specific version to the patch they need. Users would grab the wrong patch then complain the software sucked.

    In the end it was easier to tell them to just download the latest version (100MB instead of a 15MB diff) and install it on top of the old one. Nowadays I guess it's easier because the proper patch would be downloaded automatically, but things weren't always that simple. Now get off my lawn. I think I see a flock of pink flamingos heading this way. Time to play with the lawn darts.

  13. What about the signatures? by mveloso · · Score: 1

    What happens to the app signatures when you do this?

    1. Re:What about the signatures? by Anonymous Coward · · Score: 0

      Output file is identical to the original "new" file. Signatures are preserved

  14. It doesn't usually go so great... by RhettLivingston · · Score: 1

    for tech companies that reduce the need for a commodity that their major distributors are over-profiting from.

  15. Most people don't live in Finland by tepples · · Score: 1

    How many American citizens can Finland's immigration absorb?

  16. Zip-aware Rsync by Jezral · · Score: 1

    So, they're slowly moving towards a version of rsync that's aware of zip files. Hell, zip files are compressed per-file, so it doesn't even need to be aware of the compression.

    And Microsoft is also looking at differential updates. Seems everyone is busy reinventing rsync or zsync.

  17. command line example by Anonymous Coward · · Score: 0

    patch -Pnes mydickapp