Slashdot Mirror

← Back to Stories (view on slashdot.org)

APT Speed For Incremental Updates Gets a Massive Performance Boost

Posted by timothy on from the just-the-bit-I-missed dept.

jones_supa writes: Developer Julian Andres Klode has this week made some improvements to significantly increase the speed of incremental updates with Debian GNU/Linux's APT update system. His optimizations have yielded the apt-get program to suddenly yield 10x performance when compared to the old code. These improvements also make APT with PDiff now faster than the default, non-incremental behavior. Beyond the improvements that landed this week, Julian is still exploring other areas for improving APT update performance. More details via his blog post.

23 of 162 comments (clear)

  1. Re:Now only... by kthreadd · · Score: 2

    I'm missing something here. With all the flap about systemd, why the rush of all the distros to adopt it?

    There has hardly been a "rush" to speak of. Fedora was first and switched in 2011, that's almost five years ago and we still have distros that haven't switched yet. This is one of the absolute slowest tech migration ever in the Linux community.

    I'm on Mint, but even that is slated to go to systemd at the next major release. Binary logs, etc.? No thanks.

    If you by "binary logs" mean regular text logs with a bit of metadata attached so that you can actually find stuff then yes, that's a good thing.

  2. Bad code is everywhere by mveloso · · Score: 3, Funny

    Wow, reading one byte at a time unbuffered? Who does that in real life? It's been well-known for like 30 years that buffered reading is an order of magnitude faster than byte-at-a-time - which matches the above result. The standard C library does buffered reads, unless you turn them off explicitly.

    Did someone really turn that off explicitly? Why?

    Jesus, someone should check the XML parsers. Maybe the same guy wrote an XML parser and it's doing byte reads.

    1. Re:Bad code is everywhere by guruevi · · Score: 5, Insightful

      a) Back in the day we did because memory was expensive and these things were to run on 386'es and other platforms that might not have the room for a sizable buffer and memory/bus/CPU bus were all equally fast. You only need a buffer if your machine is busy doing other things)

      b) It might be a benefit on certain platforms but in certain situations it feels (without looking at the rest of the code) like the code might introduce a buffer overflow issue (he explicitly removes the zero-buffer option if the file read returns a null pointer as it's buffer).

      c) Ask the original developer or do a blame-search for that code before 'fixing' things.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    2. Re:Bad code is everywhere by jaklode · · Score: 5, Informative

      It's not using standard I/O function, but pure syscalls, which are obviously unbuffered. And the same code paths are also used for other stuff that maps files. Performance critical code implemented a buffer on top of that, and the ReadLine() function experiencing the main issue was only added as a convenience function and not used for anything critical until a few months ago (and we forgot that it was not optimized). Anyway, we implement buffering for ReadLine() now. I'll try to make it generic for both reads (all reads) and writes, but so far have not succeeded, probably because some code depends on unbuffered reads or writes.

  3. Re:Now only... by gmack · · Score: 4, Interesting

    Mainly because that flap involved very few of the actual technical people and people got upset long after the decision was already made. It also seems to be the case that some trolls went off on a misinformation campaign complete with fake bug reports. Quite frankly, I was terrified of systemd from what I was reading here, but then I actually read up on the subject and realized most of what people have been saying about it is false.

    Now that I've actually tried it, my desktops boot faster and I have had a much easier time customizing the boot sequence of some of the servers I maintain.

  4. Many thanks by NotInHere · · Score: 3, Informative

    Really looking forward to have apt-get speeds that can be compared with pacman. Julian Andres Klode, if you read this, please continue the great work!

    1. Re:Many thanks by jaklode · · Score: 5, Funny

      I'm reading everything :)

  5. Re:Now only... by kthreadd · · Score: 3, Interesting

    Of course. It's not just plain text appended to a set of files so you would have to use journalctl to access them. Nothing wrong with that. Now this gives you a number of benefits. Since journald actually knows about the data that it stores it can make intelligent decisions. Everyone who has ever had to clean up a filled up /var/log partition knows how broken logrotate is. What if the journal could actually know about this and rotate based on size? It also makes it possible to actually search for things without having to take different date and log formats into account, and actually know that it came from the correct program. I know, crazy.

  6. Re:Yum by kthreadd · · Score: 2

    Someone did, the result is called dnf and has replaced yum in Fedora.

  7. Re:Now only... by wjcofkc · · Score: 4, Insightful

    What's wrong with binary logs?

    Text is a terrible format for efficient storage of and access to structured data
    Access to binary logs is O(1) instead of O(n)
    journalctl outputs a pixel-perfect copy of what /var/log/messages was
    You can query more effectively and precisely than with awk, sed and grep
    You can still use awk, sed and grep if you want
    You can run syslogd in parallel and have your text file as well
    The binary format is well documented
    Traditional logs are binary as well as soon as they are rotated and compressed

    For fucks sake already, can we not have a single Linux related discussion that has nothing to do with systemd without it spiraling into a systemd flame fest? Systemd is not the devil. All I read here from detractors are people who are regurgitating bullshit they overheard while riding the bandwagon they blindly jumped on without actually having a single clue what they are talking about. Talk about the blind leading the blind. Meanwhile anyone with a clue who tries to chime in with a voice of reason is simply drowned out. Does using the word binary in sentence where you also refer to logs make you feel like some kind of super hacker? Sometimes I really think that's what all this never ending bullshit is about.

    --
    Brought to you by Carl's Junior.
  8. XML is broken by design performance-wise... by ffkom · · Score: 3, Informative
    ... it's not even a prefix-code "per document", so only using buffered reads when parsing XML wouldn't even allow you to avoid reading into the data following the end of the XML document you actually want to parse.

    If something is XML based and time criticial, I wouldn't bother to optimize the XML parsing, but rather exchange XML for a non braindead format to start with.

    1. Re:XML is broken by design performance-wise... by hey! · · Score: 2

      There's nothing braindead about XML, it's just not the right tool for many jobs it is used for. It's over-engineering a solution to a simple problem because everyone else is doing it that way that's braindead.

      When XML became hot around the end of the 90s, people did what they always do with a hot technology; they used it whether or not it made sense just to have it on their resume. It never made sense for things like an over-the-network data serialization format; or as a configuration file format where some kind of key/value file format would do; but it was all over the place in those kinds of roles. What it's really good at is encoding complex documents re-using multiple heterogeneous standards (e.g. a text document with SVG diagrams, MathML expressions, and Dublin Core metadata). It's a comprehensive and practical tool to use on those kinds of problems.

      --
      Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
  9. How the fuck did this slowness even happen?! by Anonymous Coward · · Score: 4, Interesting

    Holy fuck, I just read the blog article and this is what it said:

    The reason for this is that our I/O is unbuffered, and we were reading one byte at a time in order to read lines.

    Writes are still unbuffered, and account for about 75% to 80% of our runtime.

    How the fuck did that all happen to begin with?! Who the fuck wrote the code like that initially?! How fucking long has this been the case?!

    I understand that bugs will happen. I really do. But this is like a total breakdown in process. Why the fuck weren't these problems detected sooner, like when the code was committed and reviewed? The code was reviewed, right?!

    We aren't talking about some minor software project here. This is apt, for fuck's sake! This is one of the core pieces of Debian's (and Ubuntu's, and many other distros') basic infrastructure. This is the kind of shit that has to be done properly, yet it clearly wasn't in this case.

    The Debian project needs to address how this shit even happened to begin with. This is fucking unbelievable. The entire Debian community deserves a full explanation as to how this debacle happened.

    I thank God every day that I moved all of my servers from Debian to OpenBSD after the systemd decision. I know that the OpenBSD devs take code reviews and code quality extremely seriously, not just with the core OS itself, but even with software written by others. The OpenBSD project will even create and maintain their own custom forks of third party software if the original developers can't get their shit together!

    1. Re:How the fuck did this slowness even happen?! by zopper · · Score: 3, Interesting

      IMO, this originates in early days of APT. And because it worked, nobody wanted to touch it later when the speed become an issue, because nobody wanted to risk breaking it, because, as you said, it is a critical part of the infrastructure.

      And now, though, when you explained us how BSD is better, go back and pretend there are no such old lines hacked together long time ago, with nobody dusting them in years. (By the way, how long there was, for example, HeartBleed in BSD? Until discovered and fixed also on every other platform? Wow, they do really great job with the reviews!)

    2. Re:How the fuck did this slowness even happen?! by jaklode · · Score: 5, Informative

      Gosh it was slow code. Not so much bad code. There are 3 people working on APT in their free time. Instead of complaining, do something about it. You can join #debian-apt on OFTC, view the commits live, and check them. What matters more is that there are almost no functional regressions thanks to our test suite. Checking for performance regressions is an entirely different beast altogether.

    3. Re:How the fuck did this slowness even happen?! by Kjella · · Score: 2

      The reason is that while apt is a critical piece of infrastructure, incremental apt really isn't. The use case is that you're supposed to download a smaller diff, apply the diff and install the updated version either faster or cheaper if you're on a metered line. If you don't have a compatible earlier version, you have to use the traditional package anyway. So you can always fall back to that, it's only nice-to-have. The people on broadband don't care, the ones on really slow lines probably want their updates offline by some other channel. But for those who do care, usually around 56k dial-up speed or so it has mostly been solved by making modular packages that depend on a main package so you only update the parts that are needed. Incremental packages would be even more fine grained, but is of relatively little importance, which is why few people care and code probably hasn't been reviewed much for performance.

      --
      Live today, because you never know what tomorrow brings
    4. Re:How the fuck did this slowness even happen?! by jaklode · · Score: 2

      Buffered reading my seem obvious too you, but the code in question had some users outside the class that worked directly on the file descriptor, and thus will not work correctly with buffered reading. Also, it's used in other code that implements its own buffering or does not need one, because it simply copies page-size blocks from one file to another. As another example, I have not figured out how to do go completely buffered without messing everything up yet: test suite fails all over the place if I switch from syscalls read,write,etc. to fread(), fwrite(), etc. Write buffering is especially weird: Callers now need to really make sure to actually flush the contents to the disk (or close the file), otherwise things will fail. As we cannot expect all callers to do that, we probably need to turn this into an optional flag.

  10. Does DNF still use the same RPM handling code? by ffkom · · Score: 2

    If yes, I would wonder how much speed could it gain over yum... after all, whenever I used yum to update some OS, the vast majority of time was spent with the stuff "rpm" would do, not with the database maintainance yum adds.

  11. Re:only two enhancements remaining by jaklode · · Score: 5, Informative

    Well, we do download in parallel if you use httpredir.debian.org and httpredir.debian.org returns different mirrors for different packages (which it does not do all the time, but reasonably often). I don't like installing in parallel, or downloading and installing at the same time, as they just make the error handling harder, for modest speedup.

  12. Re:Now only... by Kjella · · Score: 3, Interesting

    What's wrong with binary logs?

    Nothing, if they're well-designed and ACID compliant, which journald is not:

    It's rare that I choose to defend systemd, but ACID compliance does not mean you never have to deal with a corrupt database. It is a software technique to make sure transactions complete in an atomic, consistent, isolated and durable way but it still presumes a "perfect" system and if a bit flip happens in memory or on disk outside the programming logic then ACID will fail. That is why you have ECC, RAID1/5/6 and ZFS, but even they fail and sometimes you have genuinely "impossible" results like you've added $2+$2 to the account and the result is $5 (bit flip from 0x0100 to 0x0101). If you're using plain text and UTF-8 this can happen there as well, there are combinations that are simply illegal to use. You expect the parser to ignore the "impossible" and carry on, apparently that's what journald is doing too.

    --
    Live today, because you never know what tomorrow brings
  13. Re:Now only... by hey! · · Score: 2

    I have no idea why people insist on boot times. You are aware that a machine spends little time booting and a hell of a lot more running things?

    Sure, except when you do reboot it might be important to do it quickly some of the time.

    It's absolutely true that it's more important to optimize the common case over the uncommon case, but that doesn't mean the uncommon case is necessarily insignificant.

    --
    Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
  14. Re:Now only... by driblio · · Score: 2

    Do try harder.

    http://www.freedesktop.org/wiki/Software/systemd/journal-files/

  15. Re:Is there journalctl for other operating systems by thegarbz · · Score: 2

    Is this a requirement for you? Set journald to output classic plain text logs as well. You get all the benefits of a nice utility to sort through your logs and the ASCII text files which you find so critical to your use-case.